About db-benchmarks open source test framework
The test framework which is used on the backend of https://db-benchmarks.com is fully Open Source (AGPLv3 license) and can be found at https://github.com/db-benchmarks/db-benchmarks . Here’s what it does:
- Automates data loading to the databases/search engines included in the repository.
- Can run a database/search engine in Docker with a particular CPU/RAM constraint.
- While testing:
- Purges OS cache automatically
- Automates purging database caches before each cold run
- Restarts the database before each cold run
- Looks after your CPU temperature to avoid throttling
- Looks after the coefficient of variation while making queries and can stop as soon as:
- The CV is low enough
- And the number of queries made is sufficient
- After starting a database/search engine, lets it do its warm-up stage (pre-read needed data from disk), stops waiting as soon as:
- There’s no IO for a few seconds
- And it can connect to the database/search engine
- After stopping a database/search engine waits until it fully stops
- Can accept different timeouts: start, warm-up, initial connection, getting info about the database/search engine, query
- Can emulate one physical core which allows benchmarking algorithmic capabilities of databases more objectively (
- Can accept all the values as command line arguments as well as environment variables for easier integration with CI systems
--testsaves test results to file
--savesaves test results from files to a remote database (neither of those that have been tested)
- Tracks a lot of things while testing:
- Server info: CPU, memory, running processes, filesystem, hostname
- Current repository info to make sure there’s no local changes
- Performance metrics: each query response time in microseconds, aggregated stats:
- Coefficient of variation of all queries
- Coefficient of variation of 80% fastest queries
- Cold query’s response time
- Avg(response times)
- Avg(80% fastest queries’ response times)
- Slowest query’s response time
- Database/search engine info:
select * limit 1to make sure the data collections are similar in different databases
- internal database/search engine data structures status (chunks, shards, segments, partitions, parts, etc.)
- Makes it easy to limit CPU/RAM consumption inside or outside the test (using environment variables
- Allows to start each database/search engine easily the same way it’s started by the framework for manual testing and preparation of test queries.
Before you deploy the test framework, make sure you have the following:
- Linux server fully dedicated to testing
- Fresh CPU thermal paste to make sure your CPUs don’t throttle down
sensorsto control CPU temperature to prevent throttling
- git clone from the repository:
git clone email@example.com:db-benchmarks/db-benchmarks.git cd db-benchmarks
.envwith the default value of the memory (in megabytes) the test framework can use for secondary tasks (data loading, getting info about databases)
First you need to prepare a test:
Go to a particular test’s directory (all tests must be in directory
./tests), for example “hn_small”:
Run the init script:
- download the data collection from the Internet
- build the tables/indices
../../test (it’s in the project root’s folder) to see the options:
And run the test:
Now you have test results in
./results/ (in the root of the repository), for example:
Save to db to visualize
You can now upload the results to the database for further visualization. The visualization tool, which is used on https://db-benchmarks.com/, is also open source and can be found at https://github.com/db-benchmarks/ui.
Here’s how you can save the results:
Make pull request
We are eager to see your test results. If you believe they should be added to https://db-benchmarks.com, please make a pull request of your results to this repository.
Please keep the following in mind:
- Your results should be located in the directory
- If it’s a new test/engine, any other changes should be included in the same pull request.
- It is important that we, and anyone else, should be able to reproduce your test and hopefully get similar results.
We will then:
- Review your results to ensure they follow the testing principles.
- Reproduce your test on our hardware to ensure they are comparable with other tests.
- Discuss any arising questions with you.
- And, if everything checks out, we will merge your pull request.
How to start a particular database / search engine with a particular dataset
- start Elasticsearch to test “logs10m” with the following settings:
suffix=_tuned: maps ./tests/logs10m/es/data/idx_tuned as a data directory
mem=32768limits RAM to 32GB, if not specified the default will be used from file
cpuset="0,1": Elasticsearch’s container will be running only on CPU cores 0 and 1 (which may be the first whole physical CPU)
To stop - just