Most financial-market participants will be thankful to turn the page on 2012. It has been a difficult year on many fronts, and year-end is a time for optimism, even if that same optimism has proven unfounded for the past few years.

The big banks that comprise the sell side are no exception. Trading volumes are down; the regulatory pendulum is moving toward tightness; and the overall outlook for economic growth is muted at best. At a time when fewer traders are trading and more investors are on the sideline, there is a bottom-line challenge for Goldman Sachs, Morgan Stanley and their peers on Wall Street.

The challenge is “optimizing their resources and capital in order to realize an adequate return servicing the buy side,” said Madison Gulley, executive vice president and director of global trading for Franklin Templeton Investments, which manages more than $727 billion. “Can the broker continue to intermediate the trading and research process and add recognizable value in this electronic and information-saturated world?”

The issue is one of quality over quantity, Gulley told Markets Media. Banks can offer trading and research globally and across multiple asset classes, but the breadth of offerings isn’t necessarily helpful for an investor who wants best execution on his next transaction. “Can the sell side scale their business yet offer a bespoke, value added, non-saturated product which enhances the buy side’s performance?” Gulley asked rhetorically. “This is a real challenge, and I am not sure how it is reconciled.”

One concern emanating out of Washington is the Dodd-Frank Wall Street Reform and Consumer Protection Act, a sweeping set of new rules that is expected to be a drag on financial-market firms. For big banks, specific negative effects of Dodd-Frank may come from the Volcker Rule, which is meant to rein in banks’ trading for their own accounts, and an initiative to push derivatives trading out of the realm of private negotiations and onto lit exchanges.

The sell side’s main challenges heading into 2013 are “adapting to Dodd Frank, mitigating Knight-like scenarios, (and) navigating more market structure changes,” said Mike Buek, principal in the Vanguard Equity Investment Group.

A wider adoption of effective ‘kill switch’ mechanisms can be a remedy for trading glitches in that it would contain problems quickly. Buek noted that specific market-structure developments that project as meaningful for sell-side firms include changes in order types, continued evolution of dark pools, and regulations such as trade-at and limit up-limit down.

The long lead time for Dodd-Frank, details of which are still being negotiated almost two and a half years after being signed into law, may present a challenge. Sell-side firms “will have to scramble quickly to prepare before it goes live,” Buek said. In the absence of significant further delays, “some firms will not be ready,” he said.

Another trend within regulation is the creeping need for areas outside the front office to maintain compliance.

“We are seeing more and more interest from customers to extend their insight and management out from the front office into middle-office and management systems,” said Kevin Covington, chief executive of software vendor ITRS Group. “A lot of systems that were managed in isolation are now considered part of the production envelope, and real-time insight is needed. The focus used to be predominantly front office, now it’s more and more on the entire process.”

Sell-side banks will also have to contend with rising data volumes and volatility that will make planning, forecasting and managing risk difficult, according to Simon Garland, chief strategist at Kx Systems. More certain will be the never-ending competition among service providers.

“There will always be unmet needs, or better, perceptions of needs,” Garland said. “Systems could always be faster. Everyone wants to be faster than the competition.”