Specialized databases and programming languages power high-performance trading apps.

Investment banks and hedge funds alike are deploying high-end technology for storing and manipulating vast stores of data.

While the sell side has traditionally been the leader in terms of trading technology adoption the buy side is rapidly catching up, propelled in part by the SEC’s market access rule.

The SEC recently adopted Rule 15c3-5, which applies to broker-dealers with market access to an exchange or alternative trading system, as well as to broker-dealers that provide customers or other persons with access to trading securities directly on an exchange or alternative trading system.

Under the rule, the broker-dealer is responsible for having risk management
controls and supervisory procedures reasonably designed to ensure compliance with all laws, rules and regulations.

For a hedge fund, compliance with the market access rule involves a delicate balancing act whereby pre-trade risk checks need to exhaustive without compromising speed.

“Compliance is all about risk management,” Simon Garland, chief strategist at Kx Systems, told Markets Media.  “If a high-frequency trading firm is slowed down to a great degree by pre-trade risk checks, then it’s not high frequency anymore.”

The solution lies in speeding up the process by which data is stored and retrieved, which in the case of a large-scale trading shop can involve trillions of records.

This requires specialized technology.

“With the kinds of data that bulge-bracket firms deal with, there needs to be a programming language and database,” said Garland. “If you’ve got a trillion records stored in a traditional database, and then need to scoop out a billion records and ship it to a Java-based app, that entails an enormous amount of latency.”

On the other hand, if the data can be stored and processed in one location, business latency is reduced, which gives firms an edge when reacting to market data.

Kx Systems’ kdb+ is a high performance, column-oriented database. With a single data format for real-time and historical data, kdb+ provides a unified database that eliminates latency across multiple data management systems.

Kdb+ includes q, a general-purpose programming language that can access data directly, avoiding the performance degradation of first reading in data, then sending the data to an external routine.

“A columnar database with an array-processing language is a natural fit for capital markets apps, as opposed to shrink-wrapped systems,” said Garland. “Global institutions are using kdb+ as the central database to capture, store and analyze massive quantities of time sensitive data.”

The Securities Technology Analysis Center (STAC) recently ran benchmark testing on time series data using Kx’s kdb+, in conjunction with HP’s Intel-based servers.

The benchmarks included calculating the NBBO (National Best Bid and Offer) for a specific day, calculating volume curves over 20 days and a theoretical P&L, among others.

The benchmarks showed significant improvements compared to the last set of benchmarks published by STAC in April 2011. For example, the NBBO calculation saw a 50% improvement on the previous test results.