Stumpf, a 34-year veteran of the bank who took over as CEO in 2007, had previously gained acclaim for navigating Wells Fargo through the financial crisis and keeping it free of scandal.
Stumpf will be replaced by Timothy Sloan, who has worked at Wells Fargo for 29 years - and never served in the troubled community banking branch. Another major change is that Mr. Stephen Sanger, the current member of board of directors, will be switching to the office of non-executive chairman from here-on.
Mr Stumpf's end at Wells Fargo comes a little over a month after the bank was fined 185 million dollars (£151 million) by Californian and federal regulators over its sales practices.
Long considered Stumpf's successor, Sloan has spent most of his career at Wells working with corporations and institutional investors not the retail division, where the fraudulent accounts were opened.
The employees were allegedly doing so in order to meet sales goals.
Even as Wells Fargo's assets crept toward $2 trillion under his leadership, making it one of the largest financial institutions in the country, Stumpf often recalled his upbringing as one of 11 children on a dairy and poultry farm to explain his philosophy for ethical banking.
Theresa May moves to see off 'potential Tory rebellion' over Brexit motion
Shadow Brexit Secretary Sir Keir Starmer said: "There is a mandate for exit - there's not a mandate for the terms. Mrs May has insisted she would not give a "running commentary" over negotiations with European Union leaders.
The San Francisco-based bank has apologized repeatedly, and it reported firing 5,300 employees for misconduct and putting in place more stringent internal controls.
Low morale: Sloan also needs to find a way to fix damage done to employee morale from the scandal and years of alleged mistreatment.
Jeffrey Sonnenfeld, an authority on corporate governance at Yale, believes Stumpf "should be subject to more clawbacks" pending the investigation.
But Khalid Taha, a former Wells Fargo personal banker who left after suing the bank over sales pressures, said: "Wells Fargo's problems go from top to bottom". Cowen & Co. analyst Jaret Seiberg predicts Wells Fargo will be called upon Congress for another hearing next year. He received a $19 million salary previous year.
In less than two weeks of the allegations being leveraged against it, the FBI and federal prosecutors in NY and California began probing the bank, the House Financial Services committee launched an investigation and Stumpf faced blistering questions in front of the Senate Banking Committee.
Stumpf will retire immediately.
Before the fake accounts scheme was uncovered, Wells Fargo was the world's most valuable bank, USA Today reports.