A delicate part of a machine suggests intricacy and quality of workmanship. But the innate delicateness also presents the drawback of fragility, which must be mitigated for the part to be useful.

That is the tightrope of sorts that technologists at trading-technology firms walk. Optimizing the countless parts that perform the granular work that helps clients trade faster and more reliably, while also ensuring those parts can withstand the storms of churning, high-volume markets.

Fragility of hardware, software and networks can be found in many places: the interconnections of complex systems; the web of intricate ‘handshakes’ and nuanced protocols; myriad formats; and the unpredictability of glitches and failures. Meanwhile, resilience manifests itself in battle-tested underlying protocols such as FIX; systems of checks, balances, monitoring and fallbacks; and preventative early communication.

“The major element that speaks to the fragility of systems is that there are a plethora of interconnected, dependent components,” said Suresh Thesayi, chief technology officer at Lime Brokerage, a Wedbush company. “Each of those components in and of themselves has the potential for failure. However, in the aggregate, this probability of failure is magnified.”

Continued Thesayi, “to ensure our systems do not collapse, resulting from the massive amount of data processed, procedures need to be continually monitored, both in an automated and manual fashion, to create a reliable and scalable platform that safeguards against inherent performance deficiencies.”

Quant Need

Bulwarking fragility with resilience is important for all participants in today’s high-speed electronic markets, as a systems breakdown on a high-volatility down day can have consequences for even the staidest buy-and-hold investor. But getting technology right is ultra-critical for the quantitative, systematic firms that make money via a high volume of lightning-fast, machine-generated transactions.

Domeyard LP, a hedge fund focused on high-frequency trading, has built its own software and assembled its own co-located hardware. To ensure resiliency, the firm aims to keep its technology systems simple.

“A simpler system has less room for software errors and makes it easier for the operator to identify the source of the disruption and to rectify the issue quickly,” said Domeyard Partner Christina Qi. “We also try to keep our system as modular as possible. Often the disruption is a sign of a deeper issue that needs to be resolved but that requires a good amount of time to fix. At times like this, it is advantageous to have a system that can run smoothly while the issue is ‘quarantined’ and perhaps certain components of the trading system are taken down.”

For HFT firms and their ilk, technology is what can shave critical nanoseconds off trade executions and market-data latency. More broadly, it is necessary to have a best-in-class tech ecosystem, spanning network infrastructure, trading technology, and back-office systems, all working together.

Systematic traders are victims of their own success, in a way. As firms made money and more players piled into the space over the past decade, trades got more crowded. This forced participants to get more creative in their search for profitable strategies — which has increased vulnerabilities.

“Trading models frequently become more profitable as they become more complex,” said Tucker Balcher, a professor at Georgia Institute of Technology and co-founder of machine learning and quantitative research firm Lucena. “But indeed the more complex the model, the more points of potential failure there are. If any part of the chain breaks, the whole strategy breaks.”

On any given trading day, market indices might be up or down a half a percentage point amid modest volatility, with 6 billion to 7 billion shares changing hands in U.S. equities. But beneath the surface of such ordinariness, technology systems are still tested.

“Consistent day-to-day reliability is paramount,” said Patrick Menard, director of product management at Lime Brokerage. “Any number of adverse conditions may occur, with the possibility of many associated repercussions for our clients. If we can’t establish a solution quickly enough, the client’s experience could be diminished.”

At the same time, it’s the outlier days, when trading is chaotic and volume and volatility spikes, that are the true litmus test for technology systems. June 24, 2016, the day after the Brexit vote, and August 24, 2015, when slowing Chinese growth reverberated in U.S. markets, are recent examples.

Qi of Domeyard noted that a sophisticated trading system has many moving pieces and an enormous amount of data being logged to maintain an audit trail and facilitate debugging — any component could be responsible for a disruption.

“On a high volume or volatility day, the system’s logs can take longer to process and there could be a more urgent demand to fix the issue intraday, which exacerbates the difficulty,” she said. “We see that only the most mature and well-designed trading systems really have an automated procedure that can handle these stress tests, and even established companies with dozens of developers still rely on ad hoc, manual intervention.”

FBO Clients

Ultimately, balancing the tension of fragility and resilience of technological systems is for the benefit of clients, who need their trades executed quickly and reliably. End-user clients can be compared to racecar drivers in this regard — they may not know exactly what’s under the hood or how each part works, but they need the high performance that comes from the engine.

“The markets have never been more complex, with the need to process exponentially increasing volumes of data, meet evolving regulatory requirements and continually account for changing trading venue requirements contributing to the challenge,” said Peter Maragos, CEO of Dash Financial Technologies. “The buy side rightfully expects their counterparties to have systems and procedures in place that prevent downtime and reduce risk, but implementing the myriad changes necessary to deal with this complexity is inherently laden with risk.”

Added Maragos, “balancing the requirements of both exceptional reliability and adaptability is a technology challenge that we and many others in our industry put a tremendous amount of focus on.”