One Direction went on to have four number ones, before all embarking on solo careers
Tens of millions of people would tune in to ITV every Saturday and Sunday, alongside their friends and family as wannabe pop stars fought for their 15 minutes of fame.
Some of them achieved it. But many walked away with their dreams shattered.
We laughed, we cried and sang along. Many of us thought we could do better ourselves.
And every so often, at the business end of the competition, some genuine stars - One Direction, Leona Lewis, JLS, Little Mix - were born.
In recent years though, audiences fell by nearly half, and this week ITV confirmed it had no plans for the show to return. Its creator Simon Cowell is focussing his attentions on a fresh music panel show, entitled Walk The Line,while channel bosses have fresher formats to concentrate on, like The Masked Singer.
So how did The X Factor go from four yeses to being voted off our screens?
Along with Big Brother and Pop Idol in the early 2000s, The X Factor was part of a new wave of reality TV shows that enabled normal people to become stars in a matter of months.
Not only that but the public could decide their destinies.
"This idea was that not only could you become famous, but also you could choose who the famous person is," says TV critic Scott Bryan. "I think now we kind of take that for granted but [at the time] it still felt quite revolutionary."
There were also a lot fewer channels, he notes, and the internet was still young.
In other words, when The X Factor began in 2004, no-one had smart phones, or access to YouTube, TikTok or Twitter.
"So, in terms of Saturday night's entertainment, it was one of the only shows where you had FOMO [fear of missing out]," he says.
"Even if you hated it, even if you found it a bit contrived, you kind of knew that more than 10 million people were watching it around the UK and it had multi-generational appeal."
Such was the show's cultural power that from 2005 to 2008, every single Christmas number one came courtesy of an X Factor star, whether it was Shane Ward or Alexandra Burke. However, there was a reaction to the contestant's apparently pre-ordained success.
Rage Against the Machine broke the show's winning streak in 2009. Their single, Killing In The Name was sent to number one by a fan campaign that attempted to inform Cowell the British public were no longer going to "Do as you told me".
Perversely, even that counted as a victory for Cowell. His opponents confirmed the show's importance simply by caring about it.
Matt Cardle returned the show to the festive top spot in 2010, after more than 17 million viewers tuned in to see him crowned winner of that year's series, but he was the last of a dying breed.
In a new digital decade, it became harder and harder for The X Factor to prove itself as the ultimate crucible for British talent.
X Factor acts thrived in the CD era. Fans would buy their singles as a memento of the time they'd invested in the series. Even if they never played the music, the song was all but guaranteed to go to number one.
Streaming torpedoed that link. Suddenly, the Top 40 was counting consumption as well as sales - and music fans simply weren't playing the curdled cover versions the show forced upon its winners (even Little Mix have erased their winner's single, Cannonball, from their discography, preferring to call the strident, anthemic Wings their debut).
In 2015, a year after streaming figures were incorporated into the Official Singles Chart, Louisa Johnson became the first X Factor winner to miss the top five. After that, the show never produced another number one.
Around the same time, artists like Taylor Swift, Drake and Billie Eilish redefined pop music, with an emphasis on introspective, diaristic songwriting.
A TV show designed to appeal to the broadest possible audience couldn't replicate that style without losing viewers. Nor could it survive without engaging the dominant mode of pop music. There was nowhere for it to go but down.
The musical conceit wasn't the only element of the show that had reached its sell-by date. Changing attitudes to mental health made the idea of laughing at deluded contestants seem increasingly uncomfortable.
Cowell's barbed comments could border on cruelty, too.
The music mogul told one hopeful he looked "half dead" and another that her mouth was "far, far too big" when she sang. A would-be girl group was dismissed as looking like the same person "before, during and after Weight Watchers".
"The X Factor was on at a time in which entertainment [involved] occasionally punching down, laughing at members of the public," says Bryan.
"If you look at the audition process, some of them, I think, have remained iconic British television moments but I think some of them you just wouldn't be able to do on TV now."
"I think attitudes have changed," he continues. "It's now considered that being an ordinary member of the public and then becoming famous very quickly does have its advantages, but it also can have a lot of drawbacks, particularly on your mental health."
Fellow TV critic Emma Bullimore agrees there is "a trend for a much nicer kind of programming" these days, with shows like The Great British Bake Off a stark contrast to Cowell's "brutal" put-downs.
Aware of the shift, the show tried to get nicer, but by that stage it had already been "going off the boil for quite some time," Bullimore adds.
"It just lost its way a little bit, it felt formulaic, repetitive and like we'd seen it all before."
Semi-successful attempts were made to freshen up the format, bringing in new judges like Nick Grimshaw and Rita Ora, and introducing tension-inducing rounds like the six-chair challenge - but it had already "lost that power" to provide the "water cooler" moments of old, says Bullimore.
"It wasn't appealing to the young generation and it wasn't appealing to the family audience it did before, so therefore there was no-one left really to get excited about the show."
While many X Factor alumni still appear in the charts (hello, Harry Styles!) others, including Rebecca Ferguson, Jedward, Cher Lloyd and Katie Waissel, have been critical of the way they were treated by the show, detailing a variety of bad experiences both on-air and behind-the-scenes.
Their claims have included allegations of sexual assault, mistreatment and bullying.
Misha B, who appeared in the 2011 series, said the show had "created this narrative of me being over-confident because I'm black", which led to her being accused of bullying another contestant on live television.
She said the experience had left her with suicidal thoughts and post-traumatic stress.
Waissel told the BBC News channel on Thursday she was "hoping for a brighter and more safeguarded future" for the next wave of reality TV show stars.
She confirmed she was now advising the government, with the help of other former reality TV stars, in how to implement measures to protect "future generations" of participants.
The singer also hopes her efforts will provide "comfort" to former contestants, in the knowledge that "their voices have been heard and used for greater change".
Another former X Factor star, Rhydian Roberts, told BBC Radio 5 Live this week that he remained "utterly grateful" for his time on the show, but admitted he and others found it tough going at times.
"I'm not saying it's without its flaws, you had to have had a thick skin to go on The X Factor and endure the feedback - invariably it was negative for clicks and TV entertainment," he said.
"But I did witness some slightly... Was it bullying? I don't know.
"But what it did do is it prepared you for the media and the recording industry."
When Little Mix launched their own talent show last year, they pointedly insisted on aftercare for contestants after being left to cope on their own following their X Factor experience.
"You do get thrown into it without a second to think about anything, which can be a lot to deal with," Jesy Nelson told the Radio Times.
"We never had anyone checking on us to see how we were doing mentally, it was all just go, go, go. I personally don't feel like there was anyone who cared."
A spokesman for The X Factor told the BBC that the "duty of care to our contestants is of the utmost importance".
"We take welfare very seriously and have measures in place to ensure that they are supported."
Cowell's upcoming show will arrive into a very different media landscape to the one The X Factor inhabited.
With so many other entertainment options available, Bryan thinks it will have to "prove itself pretty much straight off the bat".
"Before, even primetime shows would have a little bit of time to find their footing," he says. "But you only have to look to see how some shows [today] literally arrive and go as quickly as they appear."
However, he suggests that Cowell, if anyone, "has got the track record" to capture audiences in 2021.
Bullimore, who thinks the Simon-and-Cheryl era marked The X Factor's "golden" years, feels the format will always be remembered fondly by many.
"When Alexandra Burke is doing a duet with Beyonce in the final, that is a big deal," she says.
"And when they are filling Wembley Arena with the final, again, that was a big dea. It was a national event every time X Factor came along.
"You can't take that away from it, even if it waned at the end. It was of its time but you can't take away its success."