DB, Quality Assurance

Benchmarking NoSQL with YCSB (Yahoo! Cloud Serving Benchmark)

Quick Notes on NoSQL

  • NoSQL is an approach to databases that represents a shift away from traditional relational database management systems (RDBMS).
  • It is referred to as “non SQL”, “non-relational” or “not only SQL“.
  • NoSQL databases do not rely on structures (tables, columns, rows etc.) and use more flexible data models.
  • NoSQL is particularly useful for storing unstructured data, which is growing far more rapidly than structured data and does not fit the relational schemas of RDBMS.
  • Common types of unstructured data include: user and session data; chat, messaging, and log data; time series data such as IoT and device data; and large objects such as video and images.

What is Performance Testing?

  • Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device.
  • This process can involve quantitative tests done in a lab and measuring the response, transactions per second, throughput etc.
  • Some of the testing techniques involved in performance testing include Load Testing, Stress Testing, Soak Testing, Spike Testing etc.

Tool Used

  • The tool used for Performance Testing is YCSB (Yahoo! Cloud Serving Benchmark).
  • YCSB is a popular Java open-source specification and program suite developed at Yahoo!
  • It is often used to compare relative performance of NoSQL database management systems.
  • Pre-requisites for installing this tool: Python, Java
    • Supported DB’s: HBase,Hypertable,Cassandra,Couchbase,DynamoDB,Voldemort,MongoDB,OrientDB,Infinispan,Redis,GemFire,DynamoDB,Tarantool,Memcached…and many others

YCSB tool includes a set of core Workloads to benchmark the target databases.

Testing Environment & Setup

  • YCSB Tool (Version-0.12.0)
  • Python (Version-2.7.10)
  • Windows-8.1/CentOS-7 (Tested on both)
  • MongoDB (Version-3.2), Storage Engine: WiredTiger
  • Processor: Intel(R) Core(TM) i5 CPU @2.20GHz
  • RAM: 8.00 GB
  • System Type: 64-bit OS, x64-based processor
  • Workload Setup:
    • Record Count=10 million
    • Threads=16
    • Get Status=10 sec intervals
    • Run Time=30 mins

Running Steps

  • Set up the database system to test.
  • Choose the appropriate DB interface layer.
  • Choose the appropriate workload.
  • Choose the appropriate runtime parameters (number of client threads, target throughput, etc.).
  • Load the data.
  • Execute the workload.

Test Bed:

Test Observations:

Pros and Cons of the Tool:

  • Advantages of the Tool:
    • Being Opensource it has high ROI.
    • Easy installation and configuration.
    • Inbuilt workloads and support for implementing new workloads.
    • Support for multiple and big data.


  • Disadvantages:
    • Not much user friendly reports
    • Results are to be gathered from information in logs.


  • https://github.com/brianfrankcooper/YCSB/wiki
  • http://benchmark-docs.readthedocs.io/en/latest/benchmarks/mongodb-ycsb-ct720-jul2015.1.html#
  • https://www.mongodb.com/scale/mongodb-benchmark

About The Author

Leave a Reply