đź“ŠBenchmark results About Testing principles Test framework Blog

Contents

1.1 million comments from Hacker News: small data full-text / analytics test

Sergey Nikolaev
   3418 words   17 minutes 
/test-hn-small/featured-image.jpg

Intro

In this test we use the data collection of 1.1M Hacker News curated comments with numeric fields from https://zenodo.org/record/45901. In the modern world 1 million of documents can be considered a very small data set which, however, can be typical for many applications: blogs and news sites, online stores, job, automotive and real estate sites and so on. It’s typical for such applications to have:

  • not very long textual data in one or multiple fields
  • and a number of attributes

Data collection

The source of the data collection is https://zenodo.org/record/45901.

The record structure is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

"properties": {
   "story_id": {"type": "integer"},
   "story_text": {"type": "text"},
   "story_author": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
   "comment_id": {"type": "integer"},
   "comment_text": {"type": "text"},
   "comment_author": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
   "comment_ranking": {"type": "integer"},
   "author_comment_count": {"type": "integer"},
   "story_comment_count": {"type": "integer"}
}

Databases

So far we have made this test available for 4 databases:

  • MySQL - “world’s most popular” open source OLTP database,
  • Percona Server for MySQL - free, fully compatible, enhanced and open source drop-in replacement for any MySQL database,
  • ClickHouse - a powerful OLAP database,
  • Elasticsearch - general purpose “search and analytics engine”,
  • Manticore Search - “database for search”, Elasticsearch alternative.

In this test we make as little changes to database default settings as possible to not give either of them an unfair advantage. Testing at max tuning is no less important, but it’s a subject for another benchmark. Here we want to understand what latency a regular non-experienced user can get after just installing a database and running it with its default settings. But to make it fair to compare one with another we still had to change a few settings:

  • MySQL and Percona Server for MySQL: no tuning , just CREATE TABLE ..., FULLTEXT(story_text,story_author,comment_text,comment_author)) and standard mysql docker image .
  • ClickHouse: no tuning , just CREATE TABLE ... ENGINE = MergeTree() ORDER BY id SETTINGS index_granularity = 8192 and standard clickhouse-server docker image .
  • Elasticsearch: also no tuning except for bootstrap.memory_lock=true since as said on https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_disable_swapping it needs to be done for performance. The docker image is also from the vendor.
  • Manticore Search :
    • min_infix_len = 2 since in Elasticsearch by default you can do infix full-text search and it would be unfair advantage if Manticore was running w/o infixes (that increase response time).
    • secondary_indexes = 1 which enables secondary indexes while filtering (when loading data that’s built anyway). Since Elasticsearch uses secondary indexes by default and it’s fairly easy to enable the same in Manticore it makes sense to do it. Unfortunately in ClickHouse, MySQL and Percona Server user would have to make quite an effort to do the same, hence it’s not done, since it would then be considered a heavy tuning which would then require further tuning of the other databases which would make things too complicated and unfair.
    • and as for the others we use their official docker image + their columnar library (but it’s not used in this test).

About caches

We’ve also configured the databases to not use any internal caches. Why this is important:

  1. In this benchmark, we conduct an accurate latency measurement to find out what response time users can expect if they run one of the tested queries at a random moment, not after running the same query many times consequently.
  2. Any cache is a shortcut to low latency. As written in Wikipedia “cache stores data so that future requests for that data can be served faster”. But caches are different, they can be divided into 2 main groups:

    • đź‘Ś those that just cache raw data stored on disk. For example many databases use mmap() to map the data stored on disk to memory, access it easily and let the operating system take care about the rest (reading it from disk when there’s free memory, removing it from memory when it’s needed for something more important etc). This is ok in terms of performance testing, because we let each database leverage the benefit of using the OS page cache (or its internal similar cache that just reads data from disk) That’s exactly what we do in this benchmark.

    • âť— those that are used to save results of previous calculations. And it’s fine in many cases, but in terms of this benchmark letting database enable such a cache is a bad idea, because:

      • it breaks proper measuring: instead of measuring calculation time you start measuring how long it takes to find a value by a key in memory. It’s not something we want to do in this test (but it’s interesting in general and we’ll perhaps do it in the future and publish some article “Benchmark of caches”).
      • even if they save not a full result of a particular query, but results of its sub-calculations it’s not good, because it breaks the idea of the test - “what response time users can expect if they run one of the tested queries at a random moment”.
      • some databases have such a cache (it’s usually called “query cache”), others don’t so if we don’t disable database internal caches we’ll give an unfair advantage to those having that.

      So we do everything to make sure none of the database does this kind of caching.

What exactly we do to achieve that:

  • Clickhouse:
    • SYSTEM DROP MARK CACHE, SYSTEM DROP UNCOMPRESSED CACHE, SYSTEM DROP COMPILED EXPRESSION CACHE before testing each new query (not each attempt of the same query).
  • Elasticsearch:
    • "index.queries.cache.enabled": false in its configuration
    • /_cache/clear?request=true&query=true&fielddata=true before testing each new query (not each attempt of the same query).
  • Manticore Search (in configuration file):
    • qcache_max_bytes = 0
    • docstore_cache_size = 0
  • Operating system:
    • we do echo 3 > /proc/sys/vm/drop_caches; sync before each NEW query (NOT each attempt). I.e. for each new query we:
      • stop database
      • drop OS cache
      • start it back
      • make the very first cold query and measure its time
      • and make tens more attempts (up to 100 or until the coefficient of variation is low enough to consider the test results high quality)

Queries

The query set consists of both full-text and analytical (filtering, sorting, grouping, aggregating) queries:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306

[
{
    "default": "select count(*) from hn_small",
    "meilisearch": [
        "/indexes/hn_small/stats",
        []
    ]
},
{
    "default": "select count(*) from hn_small where comment_ranking=100",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "filter": "comment_ranking = 100",
            "limit": 0
        }
    ]
},
{
    "default": "select count(*) from hn_small where comment_ranking=500",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "filter": "comment_ranking = 500",
            "limit": 0
        }
    ]
},
{
    "default": "select count(*) from hn_small where comment_ranking > 300 and comment_ranking < 500",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "filter": "comment_ranking > 300 AND comment_ranking < 500",
            "limit": 0
        }
    ]
},
{
    "default": "select story_author, count(*) from hn_small group by story_author order by count(*) desc limit 20",
    "meilisearch": "-"
},
{
    "default": "select story_author, avg(comment_ranking) avg from hn_small group by story_author order by avg desc limit 20",
    "meilisearch": "-"
},
{
    "default": "select comment_ranking, count(*) from hn_small group by comment_ranking order by count(*) desc limit 20",
    "meilisearch": "-"
},
{
    "default": "select comment_ranking, avg(author_comment_count) avg from hn_small group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "default": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "default": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where comment_ranking < 10 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count) avg from hn_small where match('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse": "select comment_ranking, avg(author_comment_count) avg from hn_small where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch": "select comment_ranking, avg(author_comment_count) avg from hn_small where query('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql": "select comment_ranking, avg(author_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql_percona": "select comment_ranking, avg(author_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "postgres": "select comment_ranking, avg(author_comment_count) avg from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'google') group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count) avg from hn_small where match('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse":"select comment_ranking, avg(author_comment_count) avg from hn_small where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch":"select comment_ranking, avg(author_comment_count) avg from hn_small where query('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql":"select comment_ranking, avg(author_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql_percona":"select comment_ranking, avg(author_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "postgres": "select comment_ranking, avg(author_comment_count) avg from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "manticoresearch": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where match('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "clickhouse": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "elasticsearch": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where query('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "mysql_percona": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "postgres": "select comment_ranking, avg(author_comment_count+story_comment_count) avg from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'google') and comment_ranking > 200 group by comment_ranking order by avg desc, comment_ranking desc limit 20",
    "meilisearch": "-"
},
{
    "manticoresearch": "select * from hn_small where match('abc') limit 20",
    "clickhouse": "select * from hn_small where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) limit 20",
    "elasticsearch": "select * from hn_small where query('abc') limit 20",
    "mysql": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') limit 20",
    "mysql_percona": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') limit 20",
    "postgres": "select * from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'google') limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "abc",
            "limit": 20
        }
    ]
},
{
    "manticoresearch": "select * from hn_small where match('abc -google') limit 20",
    "clickhouse": "select * from hn_small where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) and not (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) limit 20",
    "elasticsearch": "select * from hn_small where query('abc !google') limit 20",
    "mysql": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc -google' IN BOOLEAN MODE) limit 20",
    "mysql_percona": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc -google' IN BOOLEAN MODE) limit 20",
    "postgres": "select * from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'abc & !google') limit 20",
    "meilisearch": "-"
},
{
    "manticoresearch": "select * from hn_small where match('\"elon musk\"') limit 20",
    "clickhouse": "select * from hn_small where (match(story_text, '(?i)\\Welon\\Wmusk\\W') or match(story_author,'(?i)\\Welon\\Wmusk\\W') or match(comment_text, '(?i)\\Welon\\Wmusk\\W') or match(comment_author, '(?i)\\Welon\\Wmusk\\W')) limit 20",
    "elasticsearch": "select * from hn_small where query('\\\"elon musk\\\"') limit 20",
    "mysql": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('\"elon musk\"') limit 20",
    "mysql_percona": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('\"elon musk\"') limit 20",
    "postgres": "select * from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ phraseto_tsquery('english', 'elon musk') limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "elon musk",
            "limit": 20
        }
    ]
},
{
    "manticoresearch": "select * from hn_small where match('abc') order by comment_ranking asc limit 20",
    "clickhouse": "select * from hn_small where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) order by comment_ranking asc limit 20",
    "elasticsearch": "select * from hn_small where query('abc') order by comment_ranking asc limit 20",
    "mysql": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc limit 20",
    "mysql_percona": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc limit 20",
    "postgres": "select * from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'abc') order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "abc",
            "sort": [
                "comment_ranking:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "manticoresearch": "select * from hn_small where match('abc') order by comment_ranking asc, story_id desc limit 20",
    "clickhouse": "select * from hn_small where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) order by comment_ranking asc, story_id desc limit 20",
    "elasticsearch": "select * from hn_small where query('abc') order by comment_ranking asc, story_id desc limit 20",
    "mysql": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc, story_id desc limit 20",
    "mysql_percona": "select * from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') order by comment_ranking asc, story_id desc limit 20",
    "postgres": "select * from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'abc') order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "abc",
            "sort": [
                "comment_ranking:asc",
                "story_id:desc"
            ],
            "limit": 20
        }
    ]
},
{
    "manticoresearch": "select count(*) from hn_small where match('google') and comment_ranking > 200",
    "clickhouse": "select count(*) from hn_small where (match(story_text, '(?i)\\Wgoogle\\W') or match(story_author,'(?i)\\Wgoogle\\W') or match(comment_text, '(?i)\\Wgoogle\\W') or match(comment_author, '(?i)\\Wgoogle\\W')) and comment_ranking > 200",
    "elasticsearch": "select count(*) from hn_small where query('google') and comment_ranking > 200",
    "mysql": "select count(*) from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200",
    "mysql_percona": "select count(*) from hn_small where match(story_text,story_author,comment_text,comment_author) against ('google') and comment_ranking > 200",
    "postgres": "select count(*) from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'google') and comment_ranking > 200",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "google",
            "filter": "comment_ranking > 200",
            "limit": 0
        }
    ]
},
{
    "manticoresearch": "select story_id from hn_small where match('me') order by comment_ranking asc limit 20",
    "clickhouse": "select story_id from hn_small where (match(story_text, '(?i)\\Wme\\W') or match(story_author,'(?i)\\Wme\\W') or match(comment_text, '(?i)\\Wme\\W') or match(comment_author, '(?i)\\Wme\\W')) order by comment_ranking asc limit 20",
    "elasticsearch": "select story_id from hn_small where query('me') order by comment_ranking asc limit 20",
    "mysql": "select story_id from hn_small where match(story_text,story_author,comment_text,comment_author) against ('me') order by comment_ranking asc limit 20",
    "mysql_percona": "select story_id from hn_small where match(story_text,story_author,comment_text,comment_author) against ('me') order by comment_ranking asc limit 20",
    "postgres": "select story_id from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'me') order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "me",
            "attributesToRetrieve": ["story_id"],
            "sort": [
                "comment_ranking:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "manticoresearch": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small where match('abc') limit 20",
    "clickhouse": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small where (match(story_text, '(?i)\\Wabc\\W') or match(story_author,'(?i)\\Wabc\\W') or match(comment_text, '(?i)\\Wabc\\W') or match(comment_author, '(?i)\\Wabc\\W')) limit 20",
    "elasticsearch": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small where query('abc') limit 20",
    "mysql": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') limit 20",
    "mysql_percona": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small where match(story_text,story_author,comment_text,comment_author) against ('abc') limit 20",
    "postgres": "select story_id, comment_id, comment_ranking, author_comment_count, story_comment_count, story_author, comment_author from hn_small WHERE story_text_ts || story_author_ts || comment_text_ts || comment_author_ts @@ to_tsquery('english', 'abc') limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "q": "abc",
            "attributesToRetrieve": ["story_id", "comment_id", "comment_ranking", "author_comment_count", "story_comment_count", "story_author", "comment_author"],
            "limit": 20
        }
    ]
},
{
    "default": "select * from hn_small order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "sort": [
                "comment_ranking:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "default": "select * from hn_small order by comment_ranking desc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "sort": [
                "comment_ranking:desc"
            ],
            "limit": 20
        }
    ]
},
{
    "default": "select * from hn_small order by comment_ranking asc, story_id asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "sort": [
                "comment_ranking:asc",
                "story_id:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "default": "select comment_ranking from hn_small order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "attributesToRetrieve": ["comment_ranking"],
            "sort": [
                "comment_ranking:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "default": "select comment_ranking, story_text from hn_small order by comment_ranking asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "attributesToRetrieve": ["comment_ranking", "story_text"],
            "sort": [
                "comment_ranking:asc"
            ],
            "limit": 20
        }
    ]
},
{
    "default": "select count(*) from hn_small where comment_ranking in (100,200)",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "filter": "comment_ranking = 100 OR comment_ranking = 200",
            "limit": 0
        }
    ]
},
{
    "default": "select story_id from hn_small order by comment_ranking asc, author_comment_count asc, story_comment_count asc, comment_id asc limit 20",
    "meilisearch": [
        "/indexes/hn_small/search",
        {
            "attributesToRetrieve": ["story_id"],
            "sort": [
                "comment_ranking:asc",
                "author_comment_count:asc",
                "comment_id:asc"
            ],
            "limit": 20
        }
    ]
}
]

Results

You can find all the results on the results page by selecting “Test: hn_small”.

Remember that the only high quality metric is “Fast avg” since it guarantees low coefficient of variation and high queries count conducted for each query. The other 2 (“Fastest” and “Slowest”) are provided with no guarantee since:

  • Slowest - is a single attempt result, in most cases the very first coldest query. Even though we purge OS cache before each cold query it can’t be considered stable. So it can be used for informational purposes only (even though many benchmark authors publish such results without any disclaimer).
  • Fastest - just the very fastest result, it should be in most cases similar to the “Fast avg” metric, but can be more volatile from run to run.

Remember the tests including the results are 100% transparent as well as everything in this project, so:

Unlike other less transparent and less objective benchmarks we are not making any conclusions, we are just leaving screenshots of the results here:

All competitors at once

/test-hn-small/5.png

MySQL vs ClickHouse

/test-hn-small/my_ch.png

MySQL vs Elasticsearch

/test-hn-small/my_es.png

/test-hn-small/my_ms.png

MySQL vs Percona Server

/test-hn-small/my_pe.png

ClickHouse vs Elasticsearch

/test-hn-small/ch_es.png

Manticore Search vs Elasticsearch

/test-hn-small/ms_es.png

Manticore Search vs ClickHouse

/test-hn-small/ms_ch.png

Disclaimer

The author of this test and the test framework is a member of Manticore Search core team and the test was initially made to compare Manticore Search with Elasticsearch, but as shown above and can be verified in the open source code and by running the same test yourself Manticore Search wasn’t given any unfair advantage, so the test can be considered unprejudiced. However, if something is missing or wrong (i.e. non-objective) in the test feel free to make a pull request or an issue on Github . Your take is appreciated! Thank you for spending your time reading this!

Back | Home