Big Data is synonymous with real-time risk calculations.

Wall Street is examining Big Data through the prism of risk management, whereby the analysis of large amounts of data in capital markets is used to drive real-time decisions.

Front desk trading activities have long been measured by milliseconds, but middle and back offices still struggle to maintain a real-time view of positions, markets and operations.

“Big Data is not a replacement for traditional relational database technology; rather, it’s more of a niche,” said Eric Newcomer, chef architect of investment banking IT at Credit Suisse. “The opportunity for Big Data lies in combining data from multiple sources to provide better analytics, such as calculation of real-time risk.”

Such systems will leverage massively parallel processing architectures, in-memory processing and appliance technologies for predictive analytics on structured information, Hadoop appliances for unstructured information and purpose-built appliances for simulations.

“Big Data is a major focus, and one of our major priorities for the first half of 2012 is to determeine the most optimal storage solution,” said Domhnall McCormack, chief technology officer of investment banking and chief operating officer of IT at UBS.

Thomson Reuters is using Hadoop quite extensively, both as a distributed store and map/reduce as a functional framework to analyze data, said Rich Brown, global head of quantitative and event-driven trading solutions at Thomson Reuters.

“In areas where the ability to query is the bigger need, we use map/reduce to load data in a parallel fashion to Cassandra clusters,” Brown said. “We are also able to use binary mem-table in Cassandra as a distributed cache to keep frequently accessed data in memory.”

Big Data technologies “give us the ability to massively parallel process data using functional programming constructs, store huge datasets in both distributed-memory and direct attached storage and a declarative interface that is not limited by SQL’s reliance on relational algebra,” he said.

Capitals markets rely on both structured and unstructured that needs to be collected, analyzed and stored. Management of these types of data has been a challenge for capital market firms in general.

“Firms are fairly well-equipped to handle structured data such as pricing data, trade analytics, and reference data,” Brown said. “As these data sources become much larger, however, we believe firms will look to outsource the storage and management of their non-proprietary data, and vendors like Thomson Reuters are well positioned to leverage its scale and data expertise to serve this growing client need.”

As the markets continue to change and traditional market data becomes more ubiquitous, firms will continue to look for new data sources and ways to successfully apply that data to their models in order to differentiate their strategies, generate alpha, and better manage risk.

“Working with unstructured data presents many challenges such as understanding what is being said, how it is being said, by whom, in what context, against what backdrop, and under what time considerations,” said Brown. “Those that can analyze this data successfully will have a great opportunity find new ways to improve their models.”

Thomson Reuters helps firms analyze unstructured data with a series of text analysis products which measure aspects of the articles such as how much it is about a particular company (relevance), whether and to what extent it is positive or negative (sentiment), how unique it is (novelty) and with what it is repetitive, what topic it is about (M&A, earnings results, corporate actions, etc.).

“It does this for premium real-time financial news as well as tens of thousands of internet news sources millions of social media sources,” said Brown.

When firms can look at unstructured data through the lens of structured metrics, it makes it far easier to understand the relationships between these unstructured data sources and the market reactions that investors are trying to model. Thomson Reuters provides these capabilities in both hosted and client deployed environments with overnight, intraday, and real-time updates.

“The interaction of news with social media – which one leads or lags – and in which circumstances will be key to understanding if one is likely to see continued momentum or a reversal, for example,” Brown said.