Along with King Charles and other members of the Royal Family, they processed to Westminster Hall where the Queen will lie in state.
The sight of the brothers together, walking behind the coffin, will evoke poignant memories of their mother Diana's funeral 25 years ago.
Crowds applauded the solemn procession.
The Queen's coffin, carried on a gun carriage, passed below the Buckingham Palace balcony where only three months ago she appeared for the final moments of the Platinum Jubilee celebrations.
Bandsmen played sombre music to accompany this careful choreography of mourning, Beethoven and Mendelssohn alongside the drumbeat of marching feet - exactly 75 steps per minute - and the sound of horses' hooves in the autumn sunshine.
The same gun carriage had carried the coffins of the Queen's father and mother - and as it went past there were ripples of applause and some tears from people crowding along the route.
Walking on foot behind the coffin were the Queen's four children, King Charles, Princess Anne and Prince Edward, in military uniform, with Prince Andrew, no longer a working royal, in a morning suit and wearing medals.
And behind them were William, now the Prince of Wales, and Prince Harry, the Duke of Sussex, also as a non-working royal not wearing military uniform.
This time the brothers were side by side, unlike Prince Philip's funeral last year when they had someone walking between them, in what was seen at the time as a sign of separation.
The brothers appearing together sends a strong visual message of family unity, after so much speculation about tensions and disagreements between them.
Their wives, Catherine and Meghan, Duchess of Sussex and Camilla, the Queen Consort, travelled in cars behind.
A thundercloud of media interest has hung over claims of a strained brotherly relationship, now conducted from different sides of the Atlantic.
But it's not clear what the feelings might really be for a family whose private grief comes with so much public scrutiny.
Catherine is now the new Princess of Wales and the images of the coffin procession will bring back thoughts of another September funeral, when Diana in 1997 was buried from Westminster Abbey.
William and Harry, as young schoolboys, walked in that funeral procession, with their father King Charles and grandfather Prince Philip. In step at a time when their worlds had been turned upside down.
Today's procession was another national moment. Crowds held up mobile phones to capture a glimpse of history taking place before them. Big Ben rang out around the streets, tolling each minute. Queen Elizabeth had left Buckingham Palace for the very final time.
Her connections with the palace went back across the eras. She'd been Christened there 96 years ago and there were guests in that family photo who had been alive at the same time as the Duke of Wellington, victor of Waterloo.
The coffin has been covered with the emblems of the monarchy, the Royal Standard and the Imperial State Crown. But there are also personal connections in the flowers in the wreath. Along with white roses and white dahlias there was pine from Balmoral and lavender and rosemary, a symbol of remembrance, from the gardens at Windsor.
Royal aides have described this procession as a significant symbolic turning point in the funeral journey. The Queen was head of her family, but she was also head of state - and here her family were putting her back into the public realm.
The procession was taking her to the heart of political life, through Whitehall, past Downing Street and to the Houses of Parliament.
At Westminster Hall, the oldest part of the Palace of Westminster, there was a short service of prayers and readings conducted by the Archbishop of Canterbury, Justin Welby.
Soldiers from the Grenadier Guards had lifted the coffin into place on a "catafalque", a raised platform inside the medieval hall.
With the Royal Family standing in the historic hall, a psalm was sung - "O Lord, thou hast searched me out and known me" - and a reading which began "Let not your heart be troubled".
Prince Harry and Meghan appeared to be holding hands as they left.
From 17:00 BST on Wednesday, the Queen's lying-in-state will begin, when the public will have their chance to pay respect, until the morning of the funeral on Monday.
The queues have already started and everyone, like the Queen's family in the procession, will have arrived with, and will leave with, their own memories.
Watch: King Charles and Princes William and Harry walk behind Queen Elizabeth's coffin
Watch: Queen Elizabeth leaves Buckingham Palace for the final time