BBC correspondents say protesters are hugging soldiers in the crowd and cheering for them.
One man told the BBC the day was “about a new beginning”.
The rally is supported by the army and members of the ruling Zanu-PF party.
War veterans – who until last year were loyal to the president – are also saying Mr Mugabe should quit.
A BBC correspondent in Harare says this is a watershed moment and there can be no return to power for Mr Mugabe.
A man arriving at the rally said: “As Zimbabweans, we are saying to our army: Thank you very much for the peaceful intervention.
“And it’s time for the masses of Zimbabwe to say: Mugabe must go and must go, like, yesterday. We can’t wait to see his back. For us, it’s about a new beginning.
“For us, it’s about the end of authoritarian rule and we are going to take our Zimbabwe back.”
Mr Mugabe, 93, had been under house arrest since the army takeover on Wednesday, but on Friday he made his first public appearance. He spoke only to open the graduation at a university of which he is chancellor.
Grace Mugabe was not present. It had been thought she had left the country but it emerged on Thursday that she was at home with Mr Mugabe.
The military made its move after a power struggle over the successor for Mr Mugabe, who has held power for 37 years.
He sacked Vice-President Emmerson Mnangagwa last week, apparently to pave the way for his wife Grace Mugabe, who is four decades younger than him, to take over the presidency instead.
The military said it was “engaging” with Mr Mugabe and would advise the public on the outcome of talks “as soon as possible”.
Mr Mugabe has led Zimbabwe since it gained independence from Britain in 1980.
Fear has lifted overnight
Analysis by the BBC’s Andrew Harding in Harare
Euphoric crowds are surging through the centre of Harare, chanting “He must go!” and waving placards demanding President Mugabe’s immediate resignation. People are sitting on their cars, horns blaring, and on top of buses, holding Zimbawean flags.
“This is a revolution,” said one man emerging from a supermarket to join the protesters. “It has been a long time coming.”
For years such scenes have been unthinkable in Zimbabwe, but the army and governing Zanu-PF gave these rallies their blessing, and the fear that held back so many people appears to have lifted overnight.
“We just want change,” said a woman in a long queue outside a bank in the centre of Harare. Others spoke of the country’s deep economic problems and its soaring unemployment, and hoped that a change of leadership might improve people’s lives.
“Mugabe and his wife must leave the country,” read one placard being waved at a crowded intersection in front of the party’s headquarters.
The governing party – now ruthlessly purging itself – will be hoping to retain its iron grip on power in Zimbabwe, but today’s extraordinary street protests may have unlocked forces that will be hard to control.
Who is protesting in Harare?
Christopher Mutsvangwa – the leader of the influential war veterans’ association – earlier this week called for a huge turnout at the rally.
“We want to restore our pride and tomorrow is the day… we can finish the job which the army started.”
“There’s no going back about Mugabe. He must leave,” he said.
Meanwhile, at least eight out of 10 regional branches of the Zanu-PF party voted on Friday for Mr Mugabe to resign as president and party secretary.
In an unprecedented broadcast, several regional leaders appeared on state TV saying that he should step down.
They also called for Grace Mugabe to resign from the party, and for Mr Mnangagwa to be reinstated in the party’s central committee.
The party’s members agreed to mobilise support for and attend Saturday’s rally. The party is planning to hold a special central committee session over the weekend to discuss the crisis.
The BBC’s Anne Soy says the sight of ruling party supporters, war veterans and civil society at the anti-Mugabe rallies is unprecedented.
Separately, the Zimbabwe Defence Forces (ZDF) said in a statement on Friday it had been approached by organisers of the rally, which it described as a “solidarity march”.
The statement went on: “ZDF is therefore advising the nation that for as long as the planned march remains orderly, peaceful… and without hate speech and incitement to cause violence, it fully supports the march.”
Liberal groups opposed to the president have also backed the protest.
How did we get here?
Soldiers seized the headquarters of Zimbabwe’s national broadcaster ZBC on Wednesday, and loud explosions and gunfire were heard.
Maj Gen Sibusiso Moyo then read out a statement on national television, assuring the nation that President Mugabe and his family were safe.
The military was only targeting what he called “criminals” around the president, he said, denying that there had been a coup.
On Thursday, Mr Mugabe was pictured smiling as he took part in talks with an army general and South African government ministers at State House but sources suggested he might be resisting pressure to resign.
What has been the reaction around the world?
- US Secretary of State Rex Tillerson urged a quick return to civilian rule, but also said the crisis was an opportunity for Zimbabwe to set itself on a new path
- Chinese foreign ministry spokesman Geng Shuang said Beijing was hoping for stability and a peaceful “appropriate” resolution
- UK Foreign Secretary Boris Johnson warned against a transition from “one unelected tyrant” to another
- Botswana’s President Ian Khama said regional leaders did not support Mr Mugabe staying in power
- Alpha Conde, the chairman of the African Union, a key regional bloc, said the takeover “seems like a coup” and demanded a return to constitutional order