Chris Bryant told MPs he had a leaked Home Office document which suggested the Chelsea Football Club owner should not be able to base himself in the UK.
Mr Bryant said it was written in 2019 and "little" had been done since then.
A spokesperson for Mr Abramovich declined to comment.
Replying to Mr Bryant, senior minister Mark Spencer said the government had taken "very strong action against high-profile Russian individuals who are of concern".
Downing Street would not be drawn on the claims about Mr Abramovich made in the Commons. The PM's spokesman said: "I couldn't comment on individuals in that way or on leaked documents of that type."
Earlier this week the government announced it would impose sanctions on three billionaires with close links to President Vladimir Putin as part of the UK's response to Russia's actions in Ukraine.
Mr Abramovich is one of Russia's richest people and is believed to be close to President Putin.
Speaking in the House of Commons, Mr Bryant urged the government to take action against Mr Abramovich, including removing his ownership of Chelsea Football Club and seizing his assets.
Quoting from the leaked Home Office document, he said: "'As part of HMG's [Her Majesty's Government] Russia strategy aimed at targeting illicit finance and malign activity, Abramovich remains of interest to HMG due to his links to the Russian state and his public association with corrupt activity and practices. An example of this is Abramovich admitting in court proceedings that he paid for political influence. Therefore HMG is focused on ensuring individuals linked to illicit finance and malign activity are unable to base themselves in the UK and will use the relevant tools at its disposal, including immigration powers to prevent this'."
Mr Bryant noted that the document had been produced three years ago and said: "Surely Mr Abramovich should no longer be able to own a football club in this country?
"Surely we should be looking at seizing some of his assets including his £152m home?"
As well as owning Chelsea, Mr Abramovich is also the primary owner of the private investment company Millhouse LLC.
He built his fortune in oil following the collapse of the former Soviet Union in the 1990s.
In 2018, Mr Abramovich experienced delays when applying to extend his visa to stay in the UK and eventually withdrew his application.
In October 2021, the businessman's spokesperson confirmed he travelled to London as an Israeli citizen - therefore entering the United Kingdom without the need for a visa.
Tier 1 investor visas, offering foreign investors fast-track residency in the UK, are being scrapped with immediate effect, Home Secretary Priti Patel confirmed last week.