The International Securities Exchange, a U.S. options exchange, is tackling the issues presented by ‘big data’ in the capital markets with its premium hosted database (PhD), a managed historical tick database.

PhD was launched earlier this year, offering the full Opra (Options Price Reporting Authority) feed, including quotes and trades from all U.S. options exchanges, U.S. equities level one data, pre-computed implied volatilities and Greeks, and corporate action histories.

ISE, along with the nine other U.S. options exchanges, is required to send top of book data to Opra, which disseminates it as a consolidated feed to the industry.

Processing that data in raw firm takes a tremendous amount of CPUs and networking capacity.

“The challenge is that firms couldn’t use the data because they lacked the infrastructure to query the data and to generate derived calculations,” said Jeff Soule, head of market data at ISE.

“With PhD, we capture all the level one equities data, all the Opra data, and calculate on a real-time basis implied volatilities and Greeks on the full Opra feed,” he said. “So instead of having to make those calculations on their own, they’re already pre-calculated in a single container that they can readily access.”

The database can be applied to a variety of functions, including full tick or time interval back testing, validating algorithms, pre- and post-trade analysis, and charting.

The challenges of big data are taxing the resources of market makers and broker-dealers, who provide the bulk of liquidity in listed securities.

“Opra tick data is extremely voluminous—over 100 gigabytes a day is produced,” said Soule. “A year’s worth of data could be 25 terabytes, which is too much data for firms to manage on their own.”

The growth in trading volumes in options is directly correlated to the spike in message volumes.

“Even today with light volumes, we’re looking at four billion messages a day, peaking at 10 billion,” Soule said. “There were five options exchanges when ISE started; now there are 10.”

The number of quotes per trade can exceed 5,000 under normal conditions, but can spike dramatically during periods of high volatility. “When market volatility rose during the financial crisis, quoting traffic exploded because of sudden price movements in the underling instrument,” said Soule.

Big data, which has come to encapsulate exploding volumes of data brought on by changes in market structure, technology and regulations, has supplanted latency as the gating factor in market data.

“The marketplace has evolved so significantly in the last 10 years that the issues that used to be pain points with delivering real-time market data, such as bandwidth and latency, have evaporated due to the ability to take a cross connect at a co-location facility,” said Soule.

Exchanges are repackaging the information they produce into value-added products of structured data, including pricing and tick data, trade analytics, reference data and other commercially-sourced information.

TMX Group, Canada’s top exchange operator, for example, provides equity and derivative data feeds, historical reference and corporate actions data products, which are delivered according to pre-defined data/functional specifications.

Current trends are resulting in more fragmentation, requiring more connectivity and partners. “As fragmentation grows, firms are on an unsustainable path to go it alone—building, maintaining, upgrading, maintaining—on a proprietary solution,” said Todd Albright, senior vice-president of sales and marketing at Activ Financial, a market data vendor.

“They need partners who can provide an integrated solution around the entire trade lifecycle: data, co-location, connectivity, order routing etc.,” Albright said.

PhD was designed as a redundant, hosted infrastructure, which relieves firms of the burden of building and hosting the infrastructure internally.

“We manage over 200 terabytes of data going back to 2005,” Soule at ISE said. “The data is available either predefined queries or custom queries using APIs. We can generate and end of day file and drop copy it to them via FTP.”