1.1 million comments from Hacker News: small data full-text / analytics test
In this test we use the data collection of 1.1M Hacker News curated comments with numeric fields from https://zenodo.org/record/45901. In the modern world 1 million of documents can be considered a very small data set which, however, can be typical for many applications: blogs and news sites, online stores, job, automotive and real estate sites and so on. It’s typical for such applications to have:
- not very long textual data in one or multiple fields
- and a number of attributes
The source of the data collection is https://zenodo.org/record/45901.
The record structure is:
So far we have made this test available for 4 databases:
- MySQL - “world’s most popular” open source OLTP database,
- Clickhouse - a powerful OLAP database,
- Elasticsearch - general purpose “search and analytics engine”,
- Manticore Search - “database for search”, Elasticsearch alternative.
We’ve tried to make as little changes to database default settings as possible to not give either of them an unfair advantage:
- MySQL: no tuning
CREATE TABLE ..., FULLTEXT(story_text,story_author,comment_text,comment_author))and standard mysql docker image .
- Clickhouse: no tuning
CREATE TABLE ... ENGINE = MergeTree() ORDER BY id SETTINGS index_granularity = 8192and standard clickhouse-server docker image .
- Elasticsearch: also no tuning
bootstrap.memory_lock=truesince as said on https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_disable_swapping it needs to be done for performance. The docker image is also from the vendor.
- Manticore Search
min_infix_len = 2since in Elasticsearch by default you can do infix full-text search and it would be unfair advantage if Manticore was running w/o infixes. Unfortunately it’s not possible in Clickhouse at all, so it’s given the handicap.
- and as for the others we use their own docker image + their columnar library (but it’s not used in this test).
We’ve also configured the databases to not use any internal caches:
SYSTEM DROP MARK CACHE,
SYSTEM DROP UNCOMPRESSED CACHE,
SYSTEM DROP COMPILED EXPRESSION CACHEafter each query .
"index.queries.cache.enabled": falsein its configuration
/_cache/clear?request=true&query=true&fielddata=trueafter each query .
- For Manticore Search in its configuration file:
qcache_max_bytes = 0
docstore_cache_size = 0
- Operating system:
- we do
echo 3 > /proc/sys/vm/drop_caches; syncbefore each new query
- we do
The query set consists of both full-text and analytical (filtering, sorting, grouping, aggregating) queries:
You can find all the results on the results page by selecting “Test: hn_small”.
Remember that the only high quality metric is “Fast avg” since it guarantees low coefficient of variation and high queries count conducted for each query. The other 2 (“Fastest” and “Slowest”) are provided with no guarantee since:
- Slowest - is a single attempt result, in most cases the very first coldest query. Even though we purge OS cache before each cold query it can’t be considered stable. So it can be used for informational purposes and is greyed out in the below summary.
- Fastest - just the very fastest result, it should be in most cases similar to the “Fast avg” metric, but can be more volatile from run to run.
Remember the tests including the results are 100% transparent as well as everything in this project, so:
- you can use the test framework to learn how they were made
- and find raw test results in the results directory.
Unlike other less transparent and less objective benchmarks we are not making any conclusions, we are just leaving screenshots of the results here:
4 competitors at once
MySQL vs Clickhouse
MySQL vs Elasitcsearch
MySQL vs Manticore Search
Clickhouse vs Elasticsearch
Manticore Search vs Elasticsearch
Manticore Search vs Clickhouse
The author of this test and the test framework is a member of Manticore Search core team and the test was initially made to compare Manticore Search with Elasticsearch, but as shown above and can be verified in the open source code and by running the same test yourself Manticore Search wasn’t given any unfair advantage, so the test can be considered unprejudiced. However, if something is missing or wrong (i.e. non-objective) in the test feel free to make a pull request or an issue on Github . Your take is appreciated! Thank you for spending your time reading this!