Prodi, 68, lost the vote of confidence by a mere five votes on Thursday, with 161 senators voting against him as opposed to 156 for. One senator abstained. Prodi, who had led the centre-left coalition for 20 months, handed in his resignation after the results of the vote were announced.
Italian President Giorgio Napolitano must now decide whether to call early elections or appoint an interim government. Prodi will remain as caretaker prime minister until a decision is taken.
Former premier Silvio Berlusconi urged Napolitano to dissolve parliament three years ahead of schedule and call early elections for spring.
"We need to go to the polls in the shortest time possible without delay," the leader of the conservative opposition said after the vote.
Prodi had won a vote on confidence in Italy's parliament on Wednesday, but political commentators and analysts had given him little chance of victory in the Senate.
"Stopping the government's work is a luxury Italy cannot afford," Prodi told the Senate before the crucial vote.
The small, centrist Udeur party had withdrawn support for Prodi before the vote in protest at what it said was a lack of support for its leader, the former justice minister Clemente Mastella, who had earlier resigned after being named in a corruption probe. He denies the charges.
The move left Prodi without a majority and relying on the support of a handful of unelected life senators.
However, Udeur senator Nuccio Cusumano then announced he would support Prodi despite his party's split from the government. He was met with shouts of "traitor" and one senator mimicked shooting him. Cusumano, who was also reportedly spat on, then fainted, before being carried out on a stretcher. He later returned to vote "yes."
