Testing principles

Our belief is that a fair database benchmark should follow some key principles:

✅ Test different databases on exactly same hardware

Otherwise you at least can’t appeal to little percents differences in test results.

✅ Test with full OS cache purged before each test

Otherwise you can’t test cold queries.

✅ Database which is being tested should have all it’s internal caches disabled

Otherwise you’ll measure cache performance.

✅ Best if you measure a cold run too. It’s especially important for analytical queries where cold queries may happen often

Otherwise you completely hide how the database can handle I/O.

✅ Nothing else should be running during testing

Otherwise your test results may be very unstable.

✅ You need to restart database before each query

Otherwise previous queries can still impact current query’s response time, despite clearing internal caches.

✅ You need to wait until the database warms up completely after it’s started

Otherwise you can at least end up competing with db’s warmup process for I/O which can spoil your test results severely.

✅ Best if you provide a coefficient of variation, so everyone understands how stable your resutls are and make sure yourself it’s low enough

Coefficient of variation is a very good metric which shows how stable your test results are. If it’s higher than N% you can’t say one database is N% faster than another.

✅ Best if you test on a fixed CPU frequency

Otherwise if you are using “on-demand” cpu governor (which is normally a default) it can easily turn your 500ms response time into a 1000+ ms.

✅ Best if you test on SSD/NVME rather than HDD

Otherwise depending on where your files are located on HDD you can get up to 2x lower/higher I/O performance (we tested), which can make at least your cold queries results wrong.