When Things Go Wrong


Trading technology has advanced massively over the past decade, with highly sophisticated connectivity and routing compressing execution speeds into microseconds. But the evolution has brought with it an uncomfortable persistence of glitches and breakdowns, and a growing concern that a higher-speed and more interconnected market can be more conducive to contagion.

From the infamous ‘fat finger’ trades that have been around since markets went electronic to the ‘flash crash’ of May 2010 to more recent snafus at trading venues Bats, Nasdaq, and Knight Capital, market participants are on high alert for when – not if – the next problem will occur.

“Nobody is 100% prepared for anything,” said a senior executive at one institutional market maker, who spoke on condition of anonymity. “Problems can happen to anyone, anywhere.”

One implication of advanced trading technology, perhaps paradoxically, is that the same advances that have streamlined, quickened and lowered the cost of securities trading have also made another tech need more pressing – the need to avert glitches as much as possible and manage them when they occur.

At an industry conference in September, Knight Capital Chief Executive Tom Joyce said the firm’s Aug. 1 trading breakdown proved that operational risk is the primary concern, not regulatory risk as had previously been assumed.

Rich Repetto, Analyst, Sandler O’Neill

Rich Repetto, Analyst, Sandler O’Neill

“That’s the elevation,” Sandler O’Neill analyst Rich Repetto said Sept. 13 at the FIA/OIC Equity Options conference in New York. “What the (U.S. Securities and Exchange Commission) is going to do in regard to controlling technology risk has all of a sudden come to the forefront.”

Concerns extend from the trading desk all the way up to the board and management levels, Repetto noted. “They have to ask if they are exposed, and are their customers exposed,” he said. “I don’t think anyone is safe, or has a free pass.”

The Bats initial public offering on its own exchange in March and the Facebook IPO on Nasdaq in May were noteworthy but not especially relevant for the institutional market maker, but the Knight snafu hit home, according to the executive.

Hitting Home

“We are a proprietary market-making firm that uses technology to access exchanges and provide liquidity, just like Knight is, so that incident came closest to us,” said this executive. “It was a failure of the processes and the systems of a company that accesses exchanges, just like we do.”

Cognizant that Knight’s $440 million loss would have wiped out most trading shops, this source said his firm has responded by checking and double-checking its own systems and processes. “We always have had a focus on risk controls internally, but this brought it to light again,” said the executive. “We went back to see if there was anything we missed, if there was anything new to be learned from the Knight experience. Maybe there is something we weren’t looking for that we should be looking for.”

“The main thing we did was increase awareness internally,” the executive said in early October. “We have not implemented anything new, at least not yet. But we did do a thorough review — highlighting for all staff what the risk controls are, why we have them, how do we use them, etc. And when problems do occur, what is the sequence of events in response.”

This market-making firm is also emphasizing the need for employees on the trading desk to report any irregularities to the appropriate higher-up, rather than unilaterally decide that something is not significant and not say anything. “Don’t judge yourself what’s important or not, report everything,” the executive said.

The executive said his market-making firm can do everything it can to ward off malfunctions or disruptions, but ultimately, the most they can do is minimize, rather than eliminate, the risk of a problem.

Internal safeguards emphasized since Knight’s near-death experience “probably make us better prepared, as there is more awareness,” the executive said. “But you don’t know where and how the next problem is going to happen — all you can do is manage and mitigate risk.”

One aspect of trading technology that has come under heightened scrutiny is not so much any particular technology system itself, but rather the transition periods between technologies. Knight’s trading snafu that happened in early August was blamed on a botched software upgrade, and near-continuous enhancements becoming the norm in technology, transitions are increasingly worrisome.

“We have modified our technology-release-management practices recently and I have been pushing back harder on external firms and initiatives, simply to try and slow the process of change down a bit,” said a Chicago-based proprietary trader.

Specific measures being implemented or considered at this trader’s firm include introducing more granular unit testing, creating more detailed and realistic exchange simulators, introducing automated build/test/release scripts, releasing code only a specific day of the week, and embedding realistic simulated trading directly into production systems to allow ongoing testing of code already in production.

“For better or worse, the nature of the industry is that we have to release software at a pretty fast clip simply to keep up,” the trader said. “The challenge is to make the process of creating and releasing software as robust as possible given that you have to do so frequently.”

One dynamic, rather than static, aspect of technology that needs to be considered ex post facto is the sequence of events of a given situation. For example, while a software glitch touched off the Knight disruption, the fact that the problem continued for more than a half hour reflected that a ‘kill switch’ either was lacking or failed, said Jamil Nazarali, head of execution services at Citadel.

As of late September, NYSE Euronext, Nasdaq, Bats and Direct Edge, the four biggest U.S. stock exchanges, said they were prepared to establish kill switches that would trip if a member’s trading exceeded a specified limit in a given trading period.

Indirect Cost 

Even if traders and investors are not directly affected by trading glitches – as was the case for many with the Bats and Facebook IPOs — they can be hurt if other market participants are scared off because that could reduce liquidity. Retail investors in particular have been known to steer clear of the market in the wake of market disruptions, especially very well-publicized instances such as Facebook.

“It is worrisome,” said John Colville, chief investment officer of the $300 million City of Sacramento pension plan. “The very fast pace is getting us away from fundamentals, which is the value-add of investing. When people are trying to make a tick here and a tick there, it makes investors skeptical about the system.”

“How can a small pension manager compete against a black box? You feel like an underdog,” Colville continued. “It is taking away the wealth-creation aspect of investing. The bull market over the past 18 months has been almost entirely institutional — retail investors think they don’t have a chance.”

As an indication of how serious the issue of market technology is being perceived, the SEC recently convened a roundtable of market participants to discuss the topic.

Importantly, SEC Chairman Mary Schapiro made the distinction between market structure, which encompasses multiple execution venues, dark pools, and the presence of high-frequency trading, and market infrastructure, which she described as “the technology that undergirds trading activity.”

The Oct. 2 roundtable focused on market infrastructure, which Schapiro noted tends to get less attention than market structure. “These matters of infrastructure are essential to any holistic approach to improving how our markets operate,” Schapiro said.

On Oct. 3, shares of Kraft Foods inexplicably soared on the Nasdaq shortly after the market opened in another trading glitch. Several dozen errant trades were cancelled after the incident that was characterized as a broker error, but as is typical for recent disruptions, no definitive explanation for the problem was provided.

Occurring just one day after the SEC roundtable, the Kraft disruption – while comparatively small-scale — lent support to the premise that snafus will inevitably happen, and efforts on the part of market participants and regulators are best geared more towards management and containment rather than elimination. In American football parlance, the idea would be for a defense to concede some very short gains while guarding against big plays.

For buy-side investment managers, technology problems aren’t so much an issue of the disruptions themselves, as direct losses have been small. The Bats IPO was pulled, so investors didn’t lose money; Knight absorbed all of its $440 million trading loss, so the only hits were to Knight shareholders; and stock plunges during the 2010 flash crash lasted only very briefly and many errant trades were cancelled. Some investors did lose money directly in Nasdaq’s fumbled Facebook IPO, but the exchange has offered a compensation plan that will at least partly offset losses.

Gene Needles, chief executive, American Beacon Advisors

Gene Needles, chief executive, American Beacon Advisors

The worry for the buy-side is more that the headline risk of the well-publicized disruptions props up investors’ wariness levels. “We are very concerned,” said Gene Needles, chief executive of American Beacon Advisors. “These events, at least in part, have caused an erosion of investor confidence. There is concern if the game is rigged or not.”

Buy-side managers can do their part to help fix the problem by supporting the Securities and Exchange Commission and other regulatory organizations, Needles said.

John Montgomery, chief investment officer of Bridgeway Capital management, said that trading and investing firms need to make technological integrity, reliability and stability a priority at all times, not just in the wake of a problem.

“We have to be vigilant and continually keep after it even if we think we’re at the top” with regard to tech systems, Montgomery said. “Look at Enron. They didn’t think they were off track, but they were.”

Related articles

  1. Assessing Bond Liquidity
    Daily Email Feature

    Low Touch, High Liquidity

    Janus Henderson traders use a broad spectrum of electronic tools to optimize the search for liquidity.

  2. Florida CFO said ESG standards are being pushed by BlackRock for ideological reasons.

  3. Outlook 2016: Stephen Grainger, SWIFT

    The new regime requires a new investment playbook involving more frequent portfolio changes.

  4. Bats-Direct Edge Complete Merger

    DWS will hold a stake of 30% in the new company.

  5. More than 220 investors representing $30 trillion in AUM have signed up to 'Advance.'