Contents

110 million comments from Hacker News: medium data full-text / analytics test

Intro

In this test we use the data collection of 1.1M Hacker News curated comments with numeric fields from https://zenodo.org/record/45901 multiplied by 100. 110 million documents can be considered a medium size data set in the modern world. You you can meet similar size datasets on big blogs and news sites, big online stores, classifieds and so on. It’s typical for such applications to have:

  • not very long textual data in one or multiple fields
  • and a number of attributes

Data collection

The source of the data collection is https://zenodo.org/record/45901.

The record structure is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
"properties": {
   "story_id": {"type": "integer"},
   "story_text": {"type": "text"},
   "story_author": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
   "comment_id": {"type": "integer"},
   "comment_text": {"type": "text"},
   "comment_author": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
   "comment_ranking": {"type": "integer"},
   "author_comment_count": {"type": "integer"},
   "story_comment_count": {"type": "integer"}
}

Databases

So far we have made this test available for 3 databases:

We’ve tried to make as little changes to database default settings as possible to not give either of them an unfair advantage:

  • Clickhouse: no tuning , just CREATE TABLE ... ENGINE = MergeTree() ORDER BY id and standard clickhouse-server docker image.
  • Elasticsearch: as we saw in another test sharding can help Elasticsearch signficantly, so given 100+ M documents is not the smallest dataset we decided it would be more fair to:
  • Manticore Search is also used in a form of their own docker image + the columnar library they provide . The following updates have been made to their defaults:
    • min_infix_len = 2 since in Elasticsearch by default you can do infix full-text search and it would be not fair to let Manticore run in lighter mode (w/o infixes). Unfortunately it’s not possible in Clickhouse at all, so it’s given the handicap.
    • we tested Manticore in two modes:
      • row-wise storage which is a default one, therefore is worth testing
      • columnar storage: the data collection is of medium size, so provided Elasticsearch and Clickhouse internally use column-oriented structures it seems fair to compare them with Manticore’s columnar storage too.

We’ve also configured the databases to not use any internal caches:

  • Clickhouse:
    • SYSTEM DROP MARK CACHE, SYSTEM DROP UNCOMPRESSED CACHE, SYSTEM DROP COMPILED EXPRESSION CACHE after each query .
  • Elasticsearch:
    • "index.queries.cache.enabled": false in its configuration
    • /_cache/clear?request=true&query=true&fielddata=true after each query .
  • For Manticore Search in its configuration file:
    • qcache_max_bytes = 0
    • docstore_cache_size = 0
  • Operating system:
    • we do echo 3 > /proc/sys/vm/drop_caches; sync before each new query

Queries

The query set consists of both full-text and analytical (filtering, sorting, grouping, aggregating) queries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
[
"select count(*) from hn",
"select count(*) from hn where comment_ranking=100",
"select count(*) from hn where comment_ranking=500",
"select count(*) from hn where comment_ranking > 300 and comment_ranking < 500",
"select story_author, count(*) from hn group by story_author order by count(*) desc limit 20",
"select story_author, avg(comment_ranking) avg from hn group by story_author order by avg desc limit 20",
"select comment_ranking, count(*) from hn group by comment_ranking order by count(*) desc limit 20",
"select comment_ranking, avg(author_comment_count) avg from hn group by comment_ranking order by avg desc, comment_ranking desc limit 20",
"select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn group by comment_ranking order by avg desc, comment_ranking desc limit 20",
"select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn where comment_ranking < 10 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count) avg from hn where match('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse": "select comment_ranking, avg(author_comment_count) avg from hn where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch": "select comment_ranking, avg(author_comment_count) avg from hn where query('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql": "select comment_ranking, avg(author_comment_count) avg from hn where match(story_text,story_author,comment_text,comment_author) against ('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20"
},
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count) avg from hn where match('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse":"select comment_ranking, avg(author_comment_count) avg from hn where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch":"select comment_ranking, avg(author_comment_count) avg from hn where query('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql":"select comment_ranking, avg(author_comment_count) avg from hn where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20"
},
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn where match('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn where query('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20"
},
{
    "manticoresearch": "select * from hn where match('abc') limit 20",
    "clickhouse": "select * from hn where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) limit 20",
    "elasticsearch": "select * from hn where query('abc') limit 20",
    "mysql": "select * from hn where match(story_text,story_author,comment_text,comment_author) against ('google') limit 20"
},
{
    "manticoresearch": "select * from hn where match('abc -google') limit 20",
    "clickhouse": "select * from hn where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) and not (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) limit 20",
    "elasticsearch": "select * from hn where query('abc !google') limit 20",
    "mysql": "select * from hn where match(story_text,story_author,comment_text,comment_author) against ('abc -google') limit 20"
},
{
    "manticoresearch": "select * from hn where match('\"elon musk\"') limit 20",
    "clickhouse": "select * from hn where (match(story_text, '(?i)\\Welon\\Wmusk\\W') or match(story_author,'(?i)\\Welon\\Wmusk\\W') or match(comment_text, '(?i)\\Welon\\Wmusk\\W') or match(comment_author, '(?i)\\Welon\\Wmusk\\W')) limit 20",
    "elasticsearch": "select * from hn where query('\\\"elon musk\\\"') limit 20",
    "mysql": "select * from hn where match(story_text,story_author,comment_text,comment_author) against ('\"elon musk\"') limit 20"
},
{
    "manticoresearch": "select * from hn where match('abc') order by comment_ranking asc limit 20",
    "clickhouse": "select * from hn where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) order by comment_ranking asc limit 20",
    "elasticsearch": "select * from hn where query('abc') order by comment_ranking asc limit 20",
    "mysql": "select * from hn where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc limit 20"
},
{
    "manticoresearch": "select * from hn where match('abc') order by comment_ranking asc, story_id desc limit 20",
    "clickhouse": "select * from hn where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) order by comment_ranking asc, story_id desc limit 20",
    "elasticsearch": "select * from hn where query('abc') order by comment_ranking asc, story_id desc limit 20",
    "mysql": "select * from hn where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc, story_id desc limit 20"
},
{
    "manticoresearch": "select count(*) from hn where match('google') and comment_ranking > 200",
    "clickhouse": "select count(*) from hn where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200",
    "elasticsearch": "select count(*) from hn where query('google') and comment_ranking > 200",
    "mysql": "select count(*) from hn where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200"
},
{
    "manticoresearch": "select story_id from hn where match('me') order by comment_ranking asc limit 20",
    "clickhouse": "select story_id from hn where (match(story_text, '(?i)\\Wme\\W') or match(story_author,'(?i)\\Wme\\W') or match(comment_text, '(?i)\\Wme\\W') or match(comment_author, '(?i)\\Wme\\W')) order by comment_ranking asc limit 20",
    "elasticsearch": "select story_id from hn where query('me') order by comment_ranking asc limit 20",
    "mysql": "select story_id from hn where match(story_text,story_author,comment_text,comment_author) against ('me') order by comment_ranking asc limit 20"
},
{
    "manticoresearch": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn where match('abc') limit 20",
    "clickhouse": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) limit 20",
    "elasticsearch": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn where query('abc') limit 20",
    "mysql": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn where match(story_text,story_author,comment_text,comment_author) against ('abc') limit 20"
},
"select * from hn order by comment_ranking asc limit 20",
"select * from hn order by comment_ranking desc limit 20",
"select * from hn order by comment_ranking asc, story_id asc limit 20",
"select comment_ranking from hn order by comment_ranking asc limit 20",
"select comment_ranking, story_text from hn order by comment_ranking asc limit 20",
"select count(*) from hn where comment_ranking in (100,200)",
"select story_id from hn order by comment_ranking asc, author_comment_count asc, story_comment_count asc, comment_id asc limit 20"
]

Results

You can find all the results on the results page by selecting “Test: hn”.

Remember that the only high quality metric is “Fast avg” since it guarantees low coefficient of variation and high queries count conducted for each query. The other 2 (“Fastest” and “Slowest”) are provided with no guarantee since:

  • Slowest - is a single attempt result, in most cases the very first coldest query. Even though we purge OS cache before each cold query it can’t be considered stable. So it can be used for informational purposes and is greyed out in the below summary.
  • Fastest - just the very fastest result, it should be in most cases similar to the “Fast avg” metric, but can be more volatile from run to run.

Remember the tests including the results are 100% transparent as well as everything in this project, so:

Unlike other less transparent and less objective benchmarks we are not making any conclusions, we are just leaving screenshots of the results here:

4 competitors at once

/test-hn/msc_msr_es_ch.png

Clickhouse vs Elasticsearch

/test-hn/ch_es.png

Manticore Search (columnar storage) vs Elasticsearch

/test-hn/msc_es.png

Manticore Search (columnar storage) vs Clickhouse

/test-hn/msc_ch.png

Manticore Search row-wise storage vs columnar storage

/test-hn/msc_msr.png

What about MySQL?

As you can see on the screenshots MySQL has been also tested, but we don’t compare it with the others here since it was heavily tuned - keys were added based on the queries.

Disclaimer

The author of this test and the test framework is a member of Manticore Search core team and the test was initially made to compare Manticore Search with Elasticsearch, but as shown above and can be verified in the open source code and by running the same test yourself Manticore Search wasn’t given any unfair advantage, so the test can be considered unprejudiced. However, if something is missing or wrong (i.e. non-objective) in the test feel free to make a pull request or an issue on Github . Your take is appreciated! Thank you for spending your time reading this!