E-mail, IM, audio and video, and social media are being mined for market cues.
While Big Data solutions are still in their infancy, market practitioners are taking the challenge of handling large volumes of unstructured data seriously.
Trading decisions are made on information, which increasingly involves storing and analyzing unstructured data, as well as market data. In 2012, the world will create about 1.5 zettabytes of information, most of it unstructured data. One zettabyte is equal to one million terabytes, or 1,000,000,000,000,000,000,000 individual bytes.
“In a Twitter/Google world, traders, risk managers, portfolio managers, and hedge fund managers are accustomed to getting instant access to the data they need,” said Bob Wolfert, managing director at First Derivatives. “Internal data management professionals are now held to a higher standard of responsiveness.”
Analytics entail using unstructured data to make predictions about future events, which simply can’t be accomplished using traditional data warehousing techniques.
Firms are attempting to synthesize traditional database technology for structured data with emerging Big Data technologies for unstructured data, such as Hadoop.
“Big Data is about finding value in a sea of unstructured data,” said Neil McGovern, senior director of strategy and financial services at Sybase. “That means performing big-time analysis on lots and lots of data to learn precious lessons for your business.”
Trading strategies that include unstructured data, such as Twitter-based trading strategies, are increasingly important.
In 2012, we expect to see strong growth in the use of unstructured data in alpha discovery and trading strategies,” McGovern said.
Capital markets firms have used advanced technology for years to store decades of market data for their quants, according to McGovern. More than one third of respondents in a Sybase survey indicated that they required more than a decade of market data for analysis and back-testing.
The challenge, in essence, is to “tease out a clear thread from the veritable sea of information to focus, direct and ultimately give meaning to what has been amassed,” said Louis Lovas, director of solutions at OneMarketData, a provider of tick data management and analytics.
“Big Data is about big analytics,” said Lovas. “Finding useful information in oceans of data is an increasingly complex problem. Big Data issues are increasingly rising to the top of the agenda for capital markets firms.”
The collection and management of a multiplicity and diversity of market data across assets and continents is not an easy task.
“There’s a growing realization that the enterprise data management problem is not only getting bigger but can no longer be pushed back because of cost constraints,” said Chris Pickles, head of industry initiatives at BT Global Banking & Financial Markets.
“Regulators and legislators have started to recognize that there’s a lot of data floating around financial institutions which is not being managed as information,” Pickles said.
In some sectors, such as energy, which revolve heavily around e-mail and instant messaging communications, automating the import of unstructured data can provide valuable insights into critical market and trading information.
Transforming unstructured instant-message trading chats into structured trade data can be used to drive real-time portfolio valuation, rapidly create trades, and better manage risk, noted Michael Hinton, chief customer officer at Allegro Development, a provider of energy trading and risk management technology.
A component of the Allegro platform, Trade Connect, automatically imports intraday pricing data from IMs and other sources to provide visibility into all real-time streaming bids and offers in energy markets.
Trade Connect provides an interface to Pivot 360, an application provided by Pivot Inc. that’s designed to enable more efficient OTC trading via instant messaging.
Hinton said: “Through this integration, IMs are streamed directly into the Allegro platform, transforming these one-to-one communications into high-value enterprise data.”