Effective market surveillance in an era of high-frequency trading involves several key challenges, including both the quantity and diversity of the data. A firm might need to represent the same cash equity in several different formats when it trades on different exchanges and orders flow through different parts of the business.

Consolidating these different message formats makes oversight possible in real time. Real-time systems enable fast response times, including through circuit breakers, which can stop a certain algorithm from trading.

“We think that innovation should be welcomed, and that speed is good,” said Stefan Hendrickx, CEO of Ancoa, in a blog posting. “But we do agree that regulators need to step up their game and move to as much real-time market surveillance as possible. Regulators need to look beyond orders at the full order book – so they need a system that can handle peak volumes of millions of messages per seconds and billions of messages per day.”

Ancoa provides high-end surveillance software for financial markets, enabling regulators, exchanges, brokers and trading desks to protect their reputation and keep on top of regulatory requirements in a rapidly changing landscape dominated by new rules, market fragmentation and increasing volumes.

Ancoa has completed a joint research effort with Intel Labs whereby Ancoa was able to improvement its surveillance applications by increased utilization of multi-core technology.

Ancoa and Intel conducted the analysis at the Intel data center using the latest Xeon processor E5 product family, using Intel’s performance analysis tools.

Based on this performance analysis, Ancoa selected message consolidation as the ideal candidate for optimization. Message consolidation is where transactional messages are normalized so they can be treated uniformly regardless of their urging, asset class or type.

The performance gains from consolidation were impressive. For example, the throughput increased from 30,000 transactions per second a single core to 140,000 TPS for the multicore version.

“Intel is delighted to have worked with the Ancoa team to optimize the performance of their technology in a Big Data environment like market surveillance, where the demands of regulators, exchanges and other market players for a high performance solution are very acutely felt,” said Gordon Hughes, global sales director of financial service alliances at Intel. “Ancoa has created a great solution for real-time message consolidation.”

Thanks to its work with Intel, Ancoa managed to refine its linearly scalable platform to be capable of handling 3.5 billion messages per trading session per multicore server, up from 1 billion messages per trading session before the optimization.

“The outcome of our work with Intel promises a real impact on the markets in years to come in the form of scalable market surveillance and monitoring,” said Hendrickx. “We estimate that the arms race of faster trading and more quote messages is nowhere near the end. The Ancoa platform is capable of handling the torrent – the onslaught – of data that is coming at peaks times. Using Intel’s technology, we took the scalability and performance to the next level.”