NYC taxi rides is probably the most commonly used benchmark in the area of data analytics.
It started with Todd W. Schneider deciding to prepare the collection first in 2015 to analyze 1.1 billion NYC Taxi and Uber Trips. Then Mark Litwintschik continued by testing lots of databases and search engines using the data collection.
In this test we use the data collection of 1.1M Hacker News curated comments with numeric fields from https://zenodo.org/record/45901 multiplied by 100. 110 million documents can be considered a medium size data set in the modern world. You you can meet similar size datasets on big blogs and news sites, big online stores, classifieds and so on. It’s typical for such applications to have:
In this test we use the data collection of 1.1M Hacker News curated comments with numeric fields from https://zenodo.org/record/45901. In the modern world 1 million of documents can be considered a very small data set which, however, can be typical for many applications: blogs and news sites, online stores, job, automotive and real estate sites and so on. It’s typical for such applications to have:
In this article we’ll observe another test added to https://db-benchmarks.com/ - 10+ million standard HTTP logs collected by Nginx on ecommerce website zanbil.ir.
At https://db-benchmarks.com/ we test different open source databases and search engines and develop an open source platform so you can do it too. In this article I would like to share 10 most important principles that we’ve formulated for ourselves that help us make high quality benchmarks.