With the advent of new regulations, the issue of ‘big data’ is becoming increasingly intractable, but new solutions—in the form of cloud-based technology—may ease the burden for capital markets firms.

Planning ahead for storage requirements is a challenge, as volume and diversity of data is often affected by increased activity in the markets which is hard to foresee, along with the seasonality of specific market activities.

The main concern is that existing data technologies and infrastructures could soon reach their limits of scalability and performance.

“One of the key things about data processing is that it’s distributed,” said Elliott Noma, managing director at Garrett Asset Management, a systematic trading firm. “In the past, trading applications would require a single SQL database, but today databases need to reside across multiple servers, and it needs to be synchronized and updated in real time.”

With the recent introduction of the need to record and store voice calls and SMS on mobile phones for trading activities in the U.K., many technologists are raising the issue of big data as additional voice recording requirements contribute to the growing repository of data being held by their firms to meet with regulatory requirements.

“With big data, technologists share the concern that they appear to be facing a rapidly increasing need to hold more data,” said Sebastien Jaouen, head of global sales trading, community services, Orange Business Services – Trading Solutions, a provider of electronic trading infrastructure.

This is not only a challenge in respect of volume, but also diversity, according to Jaouen. “Trading communication now takes many forms; voice, instant messaging and SMS as well as factoring in social networks, e-mail and other systems related data—i.e. electronic payments or transaction messaging,” he said.

“The expectation is that regulators will push for this broader range of data types to be held for far longer periods than before, with current averages in the range of three to six months moving closer to three to five years, as seen in the Dodd Frank and MiFID II proposals,” said Jaouen.

Cloud technology offers a solution to this quandary.

“In today’s cloud-based ‘technology as a service’ world, we are seeing institutions seriously considering specialist service suppliers to provide flexible data storage solutions that are fully scalable,” said Jaouen. “Although this was not a strategy that was immediately accepted for critical applications, due to uncertainty on the risks in security and reliability, the general concept of outsourcing key infrastructure—i.e., data storage, along with highly proprietary and confidential data—is now gaining acceptance.”

The larger the firm the quicker the issue of big data could become a reality. However, timing could be an issue here as these solutions are still relatively new and need to allay concerns, especially with the cautious and risk adverse solution teams within firms.

According to a recent research note by Tabb Group, “cloud computing’s limitless real-time scalability has been at odds with the operational realities of today’s capital markets institutions”.

“Voice challenges around recording requirements for compliance go beyond the previous areas of expertise of the telecommunication specialists, and clearly demonstrates the impact technology convergence already has in this sector, where lines are far more blurred between technologies deployed and managed across the trade floors,” said Jaouen.