As regulators aggressively pursue development of a Consolidated Audit Trail System for transparency across all U.S. markets, the ever-growing size of data to analyze is a significant challenge to broker dealers of all sizes and particularly for compliance and surveillance teams charged with monitoring high-frequency trading activity.

“Despite the industry’s historical reluctance to leverage third party systems, broker-dealer compliance officers are keenly aware of today’s increased regulatory scrutiny and are becoming more amenable to cloud-based solutions in order to reduce risk in the quickest time possible,” said financial management software company Firm58 managing director Sam Mele.

The rise of predictive analytics technologies to separate normal behavior from market manipulation is summoning innovations in execution surveillance, as broker dealers are honing their ability to monitor trading activity to adapt to and adhere to the more stringent regulations.

Firms cannot sit idly by, taking a chance on fines, as these can have significant material impact.

The Financial Industry Regulatory Authority recently levied a record $10 million fine against Newedge for failure to supervise, sending the message that compliance negligence in overlooking manipulative tactics such as “marking the close” and “spoofing” cause considerable systemic risk to the marketplace.

The number of regulatory cases is growing. In all of 2012, 134 broker dealer enforcement actions were recorded; as of July 15, the Securities Exchange Commission had brought charges against 157 entities and individuals, Firm58 wrote in a blog this week.

“Combining historical structured data and real-time, non-structured data to look at execution, orders, fills, and cancels to see if any combination of this is affecting the market is an enormous undertaking,” that will spur compliance and surveillance technology innovations in the future, said Mele.

For now, broker dealer compliance teams, with resource constraints as they are a cost center to the firm’s bottom line, need “a fighting chance to adhere to SEC’s failure to supervise rules,” he continued. With a focus on preventing issues from occurring, they must be able to show they are monitoring structured data on all trades, gleaning information to resolve the issues, and creating an audit trail on actions taken.

“I don’t think a lot of firms have a detailed enough compliance plan, said Mele, whose company recently surveyed a round table of compliance officers to identify trends in their use of licensed tools to monitor trading activity.

The study showed 58% are more worried about an SEC audit this year than a year ago, though more than a third or the respondents still attempt to monitor trading activity manually. While nearly 40% are using third party software now, more than 70% of these respondents spent less than five hours a week using the software. The other 60% of compliance officers surveyed claimed they were more likely to purchase third-party tools within the next year, motivated by stiff penalties and the potential for the firm to suffer irreparable reputational setbacks.

Finra, SEC and the Commodities Futures Trading Commission ask firms to detail how they enforce compliance – what tools are employed to scan potentially millions of transactions a day to monitor and analyze order and transaction data for insider trading and money laundering and execution details that could signal manipulation.

“You better be able to answer with specifics; ‘we use this product, I can show you what we’ve done,’” said Mele. In the future, compliance officers will be forced to show this is being done in unstructured data as well, he predicted.

HFT traders seeking an edge in predicting and capturing alpha are utilizing unstructured data from news feeds, blogs, and social media, some of which has not been factual. “There are no regulations around what you can tweet, but it affects the market. Technology will go there in the future,” and the next level of innovation will center on where the news is coming from, Mele told Markets Media.

There will be checks and balances for leveraging of inaccurate tweets or news stories generated via social media that are inaccurate, he said.