01.10.2013
By Terry Flanagan

Big Data Gets Bigger

With organizations increasingly mining massive sets of unstructured data, extracting usable information is becoming more difficult.

‘Big data’ analytics, pioneered by Google and Yahoo, is being deployed within the capital markets, enabling application and business teams, and IT administrators, to incorporate big data into their operations.

Software provider Continuum Analytics, for example, has built tools around the Python open-source programming language that can be used to build big data apps in finance.

“Python is becoming increasingly popular in finance for data management, integration tools and advanced analytics,” said Peter Wang, co-founder and president at Continuum Analytics. “We are developing the foundational tools which make it dramatically easier to transform, analyze and visualize large and streaming datasets with Python.”

Continuum Analytics’ Anaconda tool is an enterprise-grade Python toolset which includes of all the popular open-source Python libraries. It also sells proprietary libraries and tools for Python, including optimized database and file access libraries, and a numerical Python compiler, called Numba, that translates Python into low-level machine code.

“We have a powerful web app interface for Python analytics that runs as a public cloud application that can also be deployed by businesses internally on their own servers,” said Wang.

The big data market is projected to increase to $53 billion in 2016, up from $5.1 billion in 2012, according to market research firm Wikibon.

The market, as defined by Wikibon, includes Hadoop software and related hardware; next-generation data warehouses and related hardware; analytic platforms and applications; business intelligence, data mining and data visualization platforms; and data integration platforms.

“In 2013, we’ll see a continued focus on banks finding ways to leverage data to create a more interactive customer experience across channels,” said Mike Panzarella, director of the financial services practice at Perficient, a technology and management consulting firm, in a blog.

Financial institutions must become more data-centric to be able to transform their business models and support both new and existing products and services.

“Data governance and master data management will be fundamental to an organization’s success in the years to come,” said Panzarella. “These two technology best practices are intertwined and critical to the success of our next two banking technology trends.”

For over a decade, Python has been used in scientific computing and highly quantitative domains such as finance, oil and gas, physics and signal processing. At the same time, Python has been used to build massively scalable web applications like YouTube, and has powered much of Google’s internal infrastructure.

“At Continuum, we are developing the next generation of tools to make Python as powerful and successful for big data and business data analytics as it has been for science, engineering and scalable computing,” Wang said.

In addition to premium libraries and tools, Anaconda includes an advanced machine learning library from a partner start-up.

It also includes a “package management” tool that allows users to easily switch between multiple versions of a Python library and multiple versions of the Python interpreter, and update and downgrade across packages with a single command, Wang said.

Related articles

  1. Algorithms have become more prevalent in the spot FX market.

  2. Congress Unlikely to Act on HFT

    QB’s Algo Suite for futures market trade execution is also being co-located to HKEX.

  3. Breaking data silos is key to deploying automation beyond 'nuisance' orders.

  4. They can be used on quantum hardware expected to be available in 5 to 10 years.

  5. Regulation and Liquidity Top Concerns in Fixed income

    Streaming blocks change the basis of matching and price discovery so institutions can find new liquidity.