Linux-Based Benchmark Pushes In-Memory Database Boundaries for Scalability, Performance

In a new benchmark report, McObject's eXtremeDB-64, a 64-bit in-memory embedded database, pushes past the known boundaries of in-memory database system (IMDS) scalability and performance, demonstrating IMDSs' breakthrough responsiveness as well as their ability to grow to a size that is well within the range of very large on-disk databases.

The benchmark results, obtained on a 160-core SGI® Altix 4700 server running SUSE Linux Enterprise Server version 9 from Novell at Louisiana Immersive Technologies Enterprise (LITE), at the Research Park of the University of Louisiana at Lafayette, has important implications for organizations that are augmenting their IT infrastructure with new high-performance systems for time-sensitive, data-intensive tasks. These activities range from business intelligence to geospatial analysis, pharmaceuticals research and telecommunications routing.

The benchmark results indicate that IMDS technology meets the scalability requirements of this new category of application, and delivers dramatically faster query, join and subquery performance than is possible using on-disk databases. It also shows IMDSs can leverage the multi-core architectures becoming common in companies, university research labs, and government, with nearly linear scalability performance from a single processor core up to 160 cores.

In-memory database performance has been largely uncharted, and as a result, McObject's test was able to replace certain predictions and beliefs about IMDSs with facts. One myth predicted a dramatic drop in IMDSs' data Ingest (provisioning) efficiency as the database reached maximum size. If true, the drawback could have limited IMDSs' practicality for applications requiring very large databases (VLDBs), because of the time required to load the database with data.

However, in McObject's test, eXtremeDB-64's per-row insert time increased only moderately as the database grew to one Terabyte and beyond, with average insert time increasing from 6.9 microseconds per row for the first quartile of data to 8.3 microseconds per row for the last quartile.

McObject has published its benchmark report on for free download. In-Memory Database Systems (IMDSs) Beyond the Terabyte Size Boundary includes test results as well as query source code (using both SQL and the native eXtremeDB interface) and database schema from the benchmark application.

Benchmark report findings include:

For a simple SELECT against the 1.17 Terabyte, 15.54 billion row database, eXtremeDB-64 processed 87.78 million query transactions per second using its native application programming interface (API) and 28.14 million transactions per second using a SQL ODBC API. To put these results in perspective, consider that the lingua franca for discussing query performance is transactions per minute;

In more complex JOIN operations, the benchmark report documents performance of 11.13 million operations per second with the native API, and 4.36 million operations per second using SQL ODBC;

When provisioning the in-memory database with data, ingest performance between the first and last quartiles decreased by just 20 percent - much less than the precipitous drop-off that is often predicted for the later stages of populating a very large database (VLDB);

Backup and restore functions for VLDB required just a fraction of the time needed for initial provisioning, suggesting the persistence needs of time-sensitive, data-intensive applications can be met without undue latency;

Nearly linear scalability from a one processor up to 160 processor cores is possible, with some minor and unavoidable drop-off in scalability occurring for operations that involve a higher proportion of system "housekeeping."

"McObject's new benchmark report, In-Memory Database Systems (IMDSs) Beyond the Terabyte Size Boundary, provides further support for IMDSs' value in real-time applications requiring rapid analysis of very large databases. This test-based evidence, coupled with the growing role IMDSs are already playing in the enterprise, laboratories and government agencies, paints a very bright future for this technology," Steve Graves, McObject Co-founder and CEO, said.

"The test proved that eXtremeDB-64 can support an arbitrarily large number of concurrent processes/threads and deliver consistent performance. Performance levels were maintained as database size increased to uncharted territory for IMDSs. The performance observed was in line with results obtained with much smaller databases, given a processor clock speed of 1.6 Ghz," Graves said.

"The McObject eXtremeDB in-memory database system has proven to be an excellent high-performing solution when coupled with the unparalleled scalability of the SGI Altix server platform for real-time in-memory database operations," said Ravi Pendekanti, Senior Director of Solutions Marketing, SGI.