With the possibility of another three exchanges trading options, market participants are questioning if there is enough order flow to go around.

Nasdaq OMX is planning on starting up a new options trading venue aimed at the individual investor. The new platform would be the third to operate under the Nasdaq umbrella, adding to its existing options exchanges, Nasdaq Options Market and Nasdaq OMX PHLX.

There are diminishing returns at some point,” said one electronic trader. “Options used to only trade at one exchange, then the SEC mandated competition and there have been reduced costs across the board. Now we have nine, and some say we don’t need that many. Fragmentation has occurred.”

“From our perspective, connecting to a new exchange is a loss leader. You hope new ones introduce something unique that will help grow the entire pie, but it’s to be determined at the moment.”

The International Securities Exchange is also considering starting up a new exchange for the trading of options, according to chief executive Gary Katz. In bucking the industry-wide shift away from traditional trading floors and toward all-electronic models, the new venue could potentially have a physical floor. However, any floor that the ISE would create wouldn’t look like the traditional open-outcry trading pits. Rather, it would be something completely new and innovative, and different to anything currently on the market.

In addition to the new venue from the ISE, an all-new electronic exchange is also in the works, to be based out of Miami. Miami International Holdings is planning on launching a new options exchange during the third quarter of this year. The New Jersey-based company has also said that it plans to launch an equities exchange as well. Both platforms are to be located in Miami and would be built from the ground up. They would focus on listing securities and stocks from companies based in Central and South America.

If each of the new venues comes to fruition, that would bring the number of options trading venues in the U.S. to 12, up from the current nine.

Broker-dealers and other market participants believe there is room for new competition entering the exchange and trading venue space if they bring something new to the table. However, it may now be close to saturation point.

“It’s starting to become a problem, with having too many exchanges and too many places to route and add support to,” said Eugene Choe, managing director at Credit Suisse.

“Liquidity gets thinner and thinner, and complexity continues to grow. The pie has gotten smaller over the last few years. Infrastructure costs continue to stay the same. At some point, you need to say enough is enough, let’s deal with what we have here. If there is something wrong with the market or if there is a lack of liquidity, which I don’t think it does, then let’s deal with it then.”

“When it comes to a new exchange, there’s always room for a new idea,” said one trading venue executive. “There’s always room in any industry for a competitor with a new idea, whether an ATS [alternative trading system] with unique function or an exchange with new technology.”

“We are planning the launch of a new options platform in 2012 that would complement the hybrid floor-based Nasdaq OMX PHLX and the fully electronic Nasdaq Options Market,” said Tom Wittman, head of Nasdaq OMX’s U.S. options business, in a statement last week. “It is too early to tell the exact pricing or functionalities, but our aim is always to offer innovative market structure that improves the quality of the marketplace.”

Nasdaq’s two options venues currently handle just over a quarter of daily options trading volume in the U.S. Its new venue is expected to provide rebates to retail traders and charge fees to market makers. Pending regulatory approval, it can launch as early as the summer.

Competition in the equities space has also been high in recent years, with 13 trading venues currently in the U.S. While much of the trading flow is spread between NYSE, Nasdaq, Bats Global Markets and Direct Edge, each of those companies operates two or three venues apiece.