Gavan Holohan put the visitors ahead shortly before the interval after referee Thomas Bramall ruled Lyanco had blocked Josh Emmanuel's cross with his arm.
Southampton defender Duje Caleta-Car then caught Danilo Orsi with a flailing hand not long after the restart, allowing Holohan to convert his second spot-kick of the game.
Caleta-Car's close-range finish gave Saints hope and Theo Walcott had a late equaliser ruled out for offside as Grimsby held on to spark scenes of mass celebration among the 4,000 travelling Mariners fans, who can now look forward to a trip to Brighton in the last eight.
Grimsby are just the sixth team from the fourth tier or below to reach the last eight of the FA Cup since the introduction of the division in 1958-59 - and the first since National League Lincoln City shocked Premier League Burnley in 2017.
They are the 13th side from the fourth division or below to have beaten top-flight opponents in the competition since the Football League was rebranded at the start of the 2004-05 campaign.
Having beaten Plymouth Argyle, Cambridge United, Burton Albion and Luton Town in their previous rounds, Grimsby can now look forward to a first quarter-final since the 1938-39 season after a remarkable victory over the Premier League's bottom club.
The final whistle was met by a mix of jubilant cheers from the away end and loud boos from the home supporters, whose wretched season took another downturn against a side sitting 64 places below them in the league pyramid.
Saints manager Ruben Selles made nine changes to the team beaten by Leeds United in the Premier League on Saturday, and those alterations appeared to contribute to a lack of fluency in the hosts' play in the first half as they struggled to break down their plucky opponents.
Sekou Mara should have done better when he shot straight at former Southport goalkeeper Max Crocombe from Romeo Lavia's pass, although the Southampton striker might have been penalised for offside had he managed to find the net.
Mara was denied a goal by the linesman's flag after steering Moussa Djenepo's cut-back past Crocombe, the Saints winger ruled offside in the build-up to the relief of the travelling supporters behind the goal.
The 4,000-strong Grimsby contingent who made the 460-mile round trip to the south coast had little to shout about until five minutes before the interval, when Emmanuel's delivery struck Lyanco on the arm.
An agonising wait followed as VAR reviewed the incident, but Bramall was eventually called over to his pitchside monitor by VAR and after watching the moment of contact from several angles, he pointed to the spot.
Holohan made no mistake, sending Southampton goalkeeper Alex McCarthy the wrong way to send the Grimsby fans wild at the other end of the ground.
Things went from bad to worse for Saints in the opening stages of the second half as Caleta-Car inexplicably swung an arm at Orsi inside the penalty area, giving Bramall little choice but to award another penalty.
For the second time in the game Holohan sent a magnificent spot-kick past McCarthy to put the Mariners on the brink of an astonishing upset.
After Caleta-Car's finish set up a nail-biting final 25 minutes, Southampton thought they had restored parity when Walcott slotted home in the closing stages, but the former Arsenal man was ruled narrowly offside following another VAR review.