10 million Nginx logs

Intro
In this article we’ll observe another test added to https://db-benchmarks.com/ - 10+ million standard HTTP logs collected by Nginx on ecommerce website zanbil.ir.
Data collection
We found the data collection on https://www.kaggle.com/datasets/eliasdabbas/web-server-access-logs and found it very interesting to make a test with since the dataset represents a very standard nginx http access log. Here’s an example:
|
|
Probably most of web sites in the Internet have a similar log. Many website admins and devops want to be able to process logs like this to do filtering and analytics.
After parsing by the framework there are 11 fields in the log:
- 7 string fields
- 4 integer fields
The whole list of fields and their data types is:
|
|
We preliminarily convert the raw log to CSV so it’s easier to load to different databases and search engines.
Databases
So far we have made this test available for 3 databases:
- Clickhouse - a powerful OLAP database,
- Elasticsearch - general purpose “search and analytics engine”,
- Manticore Search - “database for search”, Elasticsearch alternative.
We’ve tried to make as little changes to database default settings as possible to not give either of them an unfair advantage:
- Clickhouse: no tuning
, just
CREATE TABLE ... ENGINE = MergeTree() ORDER BY id
and standard clickhouse-server docker image. - Elasticsearch: we test in 2 modes:
- with no tuning at all which is probably what most users do
- with number of shards equal to the number of CPU cores on the server, so Elasticsearch can utilize the CPUs more efficiently for lower response time, since as said in Elasticsearch official guide “Each shard runs the search on a single CPU thread”. The dataset size is only 3.5 GB, so it’s not clear if it’s required or not, but that’s why we are testing it.
bootstrap.memory_lock=true
since as said on https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_disable_swapping it needs to be done for performance.
- Manticore Search - no tuning
, just a plain index built up from the CSV.
- we test Manticore’s default row-wise storage
- and columnar storage since Elasticsearch and Clickhouse don’t provide row-oriented stores and it may be more fair to compare with Manticore running in this mode.
We’ve also configured the databases to not use any internal caches:
- Clickhouse:
SYSTEM DROP MARK CACHE
,SYSTEM DROP UNCOMPRESSED CACHE
,SYSTEM DROP COMPILED EXPRESSION CACHE
after each query .
- Elasticsearch:
"index.queries.cache.enabled": false
in its configuration/_cache/clear?request=true&query=true&fielddata=true
after each query .
- For Manticore Search in its configuration file:
qcache_max_bytes = 0
docstore_cache_size = 0
- Operating system:
- we do
echo 3 > /proc/sys/vm/drop_caches; sync
before each new query
- we do
Queries
The queries are mostly analytical that do filtering, sorting and grouping, but we’ve also included one full-text query which searches in the request URL:
|
|
Results
You can find all the results on the results page by selecting “Test: logs10m”.
Remember that the only high quality metric is “Fast avg” since it guarantees low coefficient of variation and high queries count conducted for each query. The other 2 (“Fastest” and “Slowest”) are provided with no guarantee since:
- Slowest - is a single attempt result, in most cases the very first coldest query. Even though we purge OS cache before each cold query it can’t be considered stable. So it can be used for informational purposes and is greyed out in the below summary.
- Fastest - just the very fastest result, it should be in most cases similar to the “Fast avg” metric, but can be more volatile from run to run.
Remember the tests including the results are 100% transparent as well as everything in this project, so:
- you can use the test framework to learn how they were made
- and find raw test results in the results directory.
Unlike other less transparent and less objective benchmarks we are not making any conclusions, we are just leaving screenshots of the results here:
3 competitors with no tuning at once
Unfortunately Elasticsearch timed out for 2 queries, hence they were excluded from the final score calculation.
Elasticsearch with no tuning vs Manticore Search (default row-wise storage)
Unfortunately Elasticsearch timed out for 2 queries, hence they were excluded from the final score calculation.
Elasticsearch with no tuning vs tuned
Unfortunately Elasticsearch timed out for 2 queries, hence they were excluded from the final score calculation.
Elasticsearch tuned vs Manticore Search (default row-wise storage)
Unfortunately Elasticsearch timed out for 2 queries, hence they were excluded from the final score calculation.
Elasticsearch tuned vs Manticore Search (columnar storage)
Unfortunately Elasticsearch timed out for 2 queries, hence they were excluded from the final score calculation.
Clickhouse vs Manticore Search (columnar storage)
Manticore Search row-wise vs columnar
Disclaimer
The author of this test and the test framework is a member of Manticore Search core team and the test was initially made to compare Manticore Search with Elasticsearch, but as shown above and can be verified in the open source code and by running the same test yourself Manticore Search wasn’t given any unfair advantage, so the test can be considered unprejudiced. However, if something is missing or wrong (i.e. non-objective) in the test feel free to make a pull request or an issue on Github . Your take is appreciated! Thank you for spending your time reading this!