TPC-C is a well-known OLTP benchmark that emulates a real-world scenario where a number of terminal operators execute five different types of transactions against a database. The business scenario is an order processing system, the five transactions are: New-order, Payment, Delivery, Order-status and Warehouse Stock-level. The number of warehouses that each terminal is connected to defines the scaling factor of the database.
TPC-C is friendly to database partitioning or “sharding”. However, 1% of new order transactions and 15% of payment transactions are sent to a warehouse that the terminal is not connected to. If the database is horizontally partitioned some transactions span across multiple shards, making it impossible to run multiple independent database instances, each servicing unique ranges of warehouses.
NuoDB does not require explicit application sharding or partitioning to scale and our DBT-2 results were achieved with a single database and no application level partitioning. NuoDB can easily cope with the situation presented by TPC-C where transactions are directed to a different warehouse. It was an interesting challenge and a very good test for the NuoDB architecture. We could achieve near linear scale-out, without having to use complicated, expensive to provision and maintain client-server DBMS configurations, nor the near complete rewrite of application code to run the benchmark on one of the NoSQL document or key-value stores.
The results presented here are initial benchmarking, we will be doing much more over the coming months. Keep reading to get a preview of our plans!
The detailed TPC-C description, specification can be found at http://www.tpc.org/tpcc/
DBT-2 is an implementation of TPC-C specification, originally created for MySQL and later extended to support PostgreSQL, Drizzle and SQLite. It is written in C and some integration at the source code level is required to add support for a new database. Luckily, DBT-2 has a generic implementation that only needed some minor adjustments to work with NuoDB. Having done so, it was easy for us to add code to interface with our C++ driver to connect/disconnect, execute a query and iterate through a result set. Our implementation uses SEQUENCES to generate a new order IDs rather than a counter defined on the “district” table.
The NuoDB benchmark with all the minor changes can be found athttps://github.com/nuodb/dbt2.
NuoDB multi-node test
We ran DBT-2 with multiple machines to demonstrate how a single NuoDB database scales as more NuoDB Transaction Engines are run. To do this, we allocated 21 commodity server machines in our lab. One machine was used exclusively for a Storage Manager and it was equipped with an SSD. Each of the remaining 20 machines were running one Transaction Engine and one DBT-2 client (each DBT-2 client had 20 connections).
The test was conducted using 5, 10, and 20 Transaction Engines with scale factor of 30, 60 and 120 warehouses. Every client was assigned a distinct subset of warehouses based on the number of nodes. Cardinality of the subset was calculated as W/H, where W is the number of warehouses and H is the number of Transaction Engines. The duration of the test was set for 10 minutes. Performance was measured in New Order Transactions Per Minute (NOTPM)
6 to 21 Rack-Mounted SuperMicro MicroCloud Units, consisting of:
- Intel Xeon E3-1270V2 Quad-Core 8 hyper-threaded Server Processor
- 4x8GB Kingston DDR3-1333MHz ECC memory
- Crucial 256GB M4 SSD
- Dual-port 10 gigabit LAN cards
- 24 port 10 gigabit LAN switch (Single port)
Hardware cost is approximately $25,000 for a 21-machine configuration.
- 5-20 Transaction Engines, configured with –mem 24G
- 1 Storage Manager
- 1 DBT-2 Client per Transaction Engine
|Transaction Engines||Total # of|
One of the things that is commonly measured is cost per NOTPM. We already noted that our Hardware cost is about $25,000 and if we factor in NuoDB software cost which our sales team has indicated to us for 21 servers (20 Transaction Engines and 1 Storage Manager) is $39,150 we can calculate cost/NOTPM.
Cost/NOTPM = ($25,000+$39,150)/348,871 = .183878
Our experience with YCSB on the Google Compute Engine was documented in Tommy Reilly’s blog post at: http://goo.gl/SUfAL. YCSB is a simple Key-Value pair benchmark designed for testing more limited NOSQL products – which we compare quite favorably against. In comparison to YCSB, TPCC includes much more complex workloads, just the New Order transaction is approximately 10 individual SQL operations and the entire set of TPCC operations are run simultaneously. Keep that in mind when comparing contrasting results between different types of benchmarks.
Summary and next steps
We have demonstrated how NuoDB performs using the DBT-2 benchmark. The result show significantly improved linear scalability as more Transaction Engines are added to manage additional warehouses.
While we are pleased with our initial DBT-2 results, we have found opportunities for further optimization of the system. We’re actively working on those optimizations and will produce revised results in the coming weeks. We’re also planning to expand our testing using DBT-2, to include the following:
- Larger number of Warehouses
- Larger number of Transaction Engines
- Comparative results on different cloud platforms such as Amazon Web Services (AWS) and Google Compute Engine (GCE)
- “Chaos Monkey” testing to show resilience and good performance in the face of partial system failures
- Mixed workload: NOTPM along with dedicated Analytic queries across all warehouses in the system