By Terry Flanagan

Real-Time Data Management

Traditional Big Data technologies too storage-centric, experts say.

New technologies are likely to yield benefit in harnessing Big Data in capital markets.

Such systems leverage massively parallel processing architectures, in-memory processing and appliance technologies for predictive analytics on structured information, Hadoop appliances for unstructured information and purpose-built appliances for simulations.

“Traditional Big Data solutions are storage-centric processing models with relatively long computational or response cycles,” Richard Tibbetts, chief technology officer at software provider StreamBase Systems, told Markets Media.

“These big data storage and analysis technologies have a place in capital markets, but we are seeing slow adoption because trading firms need so much more,” Tibbetts said.

Many trading firms are already sophisticated in their use of market data, which will be difficult for Hadoop et al to match. At the same time, tick databases don’t extend naturally to unstructured data or massively parallel map reduce jobs.

“Until some storage and historical analytics technology bridges the gap the leading firms will use a combination of tick stores and Hadoop for historical data storage and analysis,” said Tibbetts.

In order to make the shift to real time analytics, organizations will need a combination of real-time data management technology and the right analytics. Getting the right analytics requires the ability to derive the best parameters for those real time decisions from a combination of historical data, statistical analysis, and domain expertise.

“Look for firms to adopt a variety of approaches to getting analytics off the desktop and into real-time data flows,” said Tibbetts. “The best solutions will empower quantitative analysts and domain experts to participate in operationalizing their ideas, reducing time to market and assuring the best analytics are driving real time decisions.”

The goal of the analysis of large amounts of data in capital markets is to drive real-time decisions about working orders in the market, whereas in other industries it’s more likely to be monthly or weekly decisions about website provisioning, design and management, certainly not real-time decisions that are measured in milliseconds.
Front desk trading activities have long been measured by milliseconds, but middle and back offices still struggle to maintain a real-time view of positions, markets and operations.

StreamBase, a provider of complex-event processing (CEP) technology, has launched StreamBase LiveView, which enables firms to address the challenges of managing high-speed order flow and reacting in real-time.

Related articles

  1. Regulatory reporting is an important part of MiFID II.

  2. Data extraction and integration is the second stage of a digitization process.

  3. Shedding Light on 'Dark' Data

    Financial Instrument Global Identifier enables consistency through trade lifecycle and across institutions.

  4. Spending on ESG data has an annual growth rate of 20%.

  5. Increased electronification has created useable and accessible real-time and historic trade data.