* Suit alleges 3 ex-employees violated non-compete clauses

* Teza had hired former Goldman Sachs computer programmer

* Teza spokesman says firm turned over computers to FBI (Adds details from the complaint, court details)

By Svea Herbst-Bayliss

BOSTON, July 9 (Reuters) - Citadel Investment Group, one of the world's most successful hedge fund firms, sued a former top executive in its highly successful quantitative trading unit and two others for setting up their own firm.

Chicago-based Citadel, founded by 40-year-old billionaire Kenneth Griffin, said in a lawsuit filed on Thursday that Mikhail Malyshev, 40, and two other former employees had violated their non-compete clauses by starting their own firm, Teza Technologies LLC. The lawsuit was filed in the the circuit court of Cook County, Illinois.

Teza Technologies made headlines this week when it was identified as the firm that had hired Sergey Aleynikov, a former Goldman Sachs Group Inc (GS.N: Quote, Profile, Research) computer programmer whom federal prosecutors had accused of stealing trade secrets from the Wall Street investment bank.

Malyshev, a Russian emigre with a doctorate in astrophysics from Princeton, left Citadel's quantitative trading unit in February after the funds he helped run returned about 40 percent last year. Their performance stood out at a time when most hedge funds lost money and Citadel's flagship portfolios tumbled 50 percent.

Citadel, which manages $11 billion and one of whose flagship hedge funds returned an average 20 percent per year between 1998 and 2006, said it zealously guards the secrecy of its own computer codes. The hedge fund firm said it spent hundreds of millions of dollars to develop strategies, software and hardware, or what is sometimes referred to as the "secret sauce" of the high frequency business, court papers show.