In September a year ago, Merkel's CDU party won the general election but more than 13% of the voting population chose the right-wing nationalists AfD, which are anti-Islam and anti-refugee.
She said the SPD would now propose a "discussion paper" in Berlin demanding concrete progress on key issues over the next year, including pension rights and better childcare, before deciding whether to remain in the coalition.
Electoral momentum is on the side of newer parties, more tightly focused on a narrow range of issues.
Nahles is also feeling the heat from SPD members still disgruntled with their leaders' decision to join Merkel instead of fulfilling an election promise to sit in opposition if they fail to win the federal vote.
The other big victor was the far-right Alternative for Germany (AfD), which entered the Hesse regional assembly for the first time with 12.8 percent of the vote, the ZDF projection showed.
She has said she wants to remain in the post of Chancellor, despite not being chair of her party, but she previous said the two roles should go hand in hand.
Merkel has loomed large on the European stage since 2005, helping guide the EU through the euro zone crisis and opening Germany's doors to migrants fleeing war in the Middle East in 2015 - a move that still divides the bloc and Germany.
Germany's other leader CDU chancellors - Konrad Adenauer and Helmut Kohl - both had messy ends to their time in office.
Arrest in bomb scare file
Other packages sent to prominent Democrats on Wednesday also listed a return address for Wasserman Schultz's South Florida office. He said investigators were able to trace Sayoc after finding a fingerprint on an envelope containing a bomb sent to Rep.
Glenn Hoddle Rushed To Hospital After Falling Ill At BT Sport Studio
BT presenter Jake Humphrey said Hoddle was "taken seriously ill at the BT Sport studio this morning". The England national team sent their well wishes to Hoddle, tweeting: "Worrying news.
Saudi crown prince calls slain journalist's killing a 'heinous crime'
An adviser to Turkey's president said Prince Mohammed had "blood on his hands" over Khashoggi, the bluntest language yet from someone linked to Erdogan.
Merkel's ruling coalition "has lost the confidence of the electorate", said Josef Joffe, publisher-editor of weekly Die Zeit.
The unstable government nearly collapsed twice over the summer, notably when Merkel restrained hardline Interior Minister Horst Seehofer's attempts to toughen up migrant policy.
Of the SPD, he added: "A party on the way down can not suddenly rise from the ashes by going into the opposition".
Even before the results were known, however, leading politicians in the Berlin coalition were signalling to the party rank-and-file to stay the course.
According to an exit poll in Hesse state by public broadcaster ARD, Merkel's Christian Democratic Union (CDU) shed around 10 points for a 28-per cent score, compared with 38.3 per cent in 2013. The mass-circulation daily Bild also reported that she said she won't run again as party leader.
Andrea Nahles, the Social Democrats' leader, said that "the state of the government is unacceptable".
Merkel's coalition partner in Berlin, the Social Democrats (SPD), tanked to 19.8% in a dead heat with the resurgent Green party for second place. She is likely to be reappointed but a weak show of support for her would undermine her authority and accelerate the succession debate.