Big Data Moves to the Cloud

Terry Flanagan

New computing paradigm offers limitless potential.

The cloud computing paradigm is being applied to capital markets, changing the economics for storing and processing Big Data.

The goal is to lift the entire high-cost infrastructure into the cloud, where hardware and software resources, apps, and market data can be distributed, shared, and leveraged.

Cloud computing“We are seeing universal trends in terms of technological advancements that are allowing structuring and analysis of large volumes of data in real-time,” said Rutger ter Hoeven, financial segment marketing manager at Interxion.

“Solutions such as BigTable, Cassandra, and SimpleDB are playing a key role in this, especially as these solutions are typically highly scalable as they can be easily spread over a cluster of machines in the cloud,” ter Hoeven said. “The increasing use of supercomputers and the usage of software solutions such as Hadoop are also driving this development.”

On-demand business analytics employ a SaaS [software as a service] model to provide massive data volume support and distributed computing to solve problems faster in the cloud.

“This model has several big advantages, such as the ability to grow and shrink resource usage on an as-needed basis and removing the need to incur the costs and complexities of building and maintaining complex infrastructure in-house, said Matt Blakely, business intelligence technology practice lead at SWI, a software consultancy.

”As data volumes become extreme and must be combined with internal and cloud based sources, the demand and growth for BI platforms which utilize SaaS and the cloud will see a big increase,” Blakely said.

Xignite, a global market data cloud, is providing bond data on-demand, which will allow financial services professionals to adapt strategies to the volatile market on-the-fly.

“Firms will be able to access a wider universe of OTC and exchange-traded bond data without having to commit to a specific universe beforehand,” said Rajesh Razdan, director of product marketing at Xignite.

Xignite offers more than 50 financial cloud APIs and fulfills more than five billion service requests per month for real-time, historical and reference data covering global equities, commodities, currencies, options, fixed income, mutual funds, derivatives and OTC instruments.

“Investment managers, traders, risk and compliance staff all desire access to a large data universe, but their firms struggle to keep up with on-going demands of petabyte-era constant capacity expansion of IT infrastructure to process, store and disseminate this data to various applications and processes,” said Razdan.

Technology and cloud service providers are maturing at a rapid clip.

“The scalability of the cloud is its most compelling aspect,” said Steve Melanaski, chief technology officer for capital markets at Hewlett-Packard. “As volumes of unstructured data such as social media content expand, you don’t have to worry about additional hardware.”

Autonomy, an HP Company, announced this week that Autonomy’s private cloud now manages more than 50 petabytes of web content, video, email and multimedia data on 6,500 servers in 14 data centers around the world.

Fifty petabytes is equal to 665 years of HD-TV video, or 1 billion four-drawer file cabinets filled with text.
Powered by Autonomy’s Intelligent Data Operating Layer (IDOL), the private cloud automatically recognizes concepts and patterns in billions of structured and unstructured data files it ingests and indexes every day.

Autonomy provides a range of cloud-based solutions that leverage Autonomy IDOL’s ability to understand in real time massive volumes of unstructured data inside and outside an organization.

To be sure, cloud computing is a deep dive, requiring investment in skills and technology and the creation of horizontal services, which thereafter can be leveraged in a cloud environment.

“For capital markets, private clouds enable firms to scale out applications on a massive scale, similar to Amazon or Google,” said Roji Oommen, senior director, business development, financial services at Savvis.
“The biggest challenge is migrating legacy applications,” Oommen said. “You can’t just take your existing application stack and move it seamlessly to the cloud. The overwhelming majority of legacy applications are not cloud-friendly.”

Xignite’s bond data cloud offerings include bond security master product for U.S. bonds, price data for Finra TRACE reportable and exchange traded bonds, security master detail for European and Asian corporate bonds, and real-time price data for Finra TRACE reportable and exchange-traded bonds.

Key to Xignite’s offering is the ‘on-demand’ aspect of the platform. Applications can request data on an as-needed basis model, whereby only a subset of data that’s required by the application or process at a particular time can be requested and immediately made available through web services.

“For firms that do need large datasets the platform provides mass data delivery directly into cloud storage drop boxes such as Amazon S3, thus enabling firms to take advantage of cloud computing in a holistic manner,” Razdan said.

Various exchanges are building their cloud data offerings directly on the Xignite platform, thus removing latencies that are associated with tradition data redistribution architectures.

Xignite partners with various exchanges, data vendors and get certain data directly from sources. “Xignite adds value by enabling users to request data piece meal and creating custom composites from multiple data services,” said Razdan.

Related articles

  1. From The Markets

    FIA Launches ETD Tracker

    Online tool displays data on the trading of exchange-traded derivatives.

  2. Under the new agreement, TNS can deliver TASE market data globally.

  3. The first phase of the rewrite was originally due to be implemented in May 2022.

  4. The new benchmarks, with Uniswap launched this year, capture 40% of value in DeFi protocols on Ethereum.

  5. Saphyre’s AI platform allows data to be entered once for simultaneous access.