Under the agreement reached in June, Google would have placed its advertisements alongside Yahoo search results, bringing Yahoo around $800 million in additional annual revenue.
The U.S. Justice Department told the two companies on Wednesday morning it planned to file a suit against the deal.
"The arrangement likely would have denied consumers the benefits of competition - lower prices, better service and greater innovation," a department press release said.
Google backed out of the deal later in the day, saying it was unwilling to face a legal battle.
David Drummond, chief legal officer at Google, said on a company blog: "it's clear that government regulators and some advertisers continue to have concerns about the agreement."
"Pressing ahead risked not only a protracted legal battle but also damage to relationships with valued partners. That wouldn't have been in the long-term interests of Google or our users, so we have decided to end the agreement."
News that the agreement had fallen through sent Yahoo's shares up 10%, on rumors that it would accept a takeover bid from Microsoft, but sources close to the companies denied the reports.
Microsoft offered $33 per share for Yahoo in an unsolicited takeover bid in May, well above the current share price of around $14.
Analysts have also suggested Yahoo may be forced to merge with AOL.