Ben Wallace said there was a "whiff of Munich in the air" - a reference to diplomacy that failed to prevent World War Two.
Minister Brandon Lewis said Mr Wallace was expressing concern that "Russia has continued to move troops" during talks.
Moscow has denied any plans to invade Ukraine but has amassed more than 100,000 troops near the border.
In an interview with the Sunday Times, Mr Wallace said: "It may be that he [Putin] just switches off his tanks and we all go home but there is a whiff of Munich in the air from some in the West."
Prior to World War Two, European powers pursued a policy of appeasement towards Nazi Germany, culminating in an agreement at Munich to allow the invasion of Czechoslovakian territories - a betrayal of allies which failed in its goal of preventing war.
Vadym Prystaiko, Ukraine's ambassador to the UK, said the comparison of diplomatic efforts with Russia to appeasement was unhelpful.
He told the BBC Radio 4's Broadcasting House programme: "It's not the best time for us to offend our partners in the world, reminding them of this act which actually did not bring peace but the opposite - it bought war."
Northern Ireland Secretary Brandon Lewis told the BBC's Sunday Morning programme: "If you look back to that period of time there was a lot of diplomatic engagement, there was an optimism at the time, actually, that there may be a diplomatic way through.
"That eventually turned out not to be the case. It turned out that wasn't the intent or aim of Adolf Hitler at the time."
Mr Lewis said the fact Russia had not yet invaded could indicate talks were working, but he warned there remained a risk President Vladimir Putin may not be engaging sincerely.
In his interview, Mr Wallace also said Moscow could launch an "offensive at any time".
He later said on Twitter that because of the "worsening situation in Ukraine", he had cut short a long weekend abroad with his family.
One of the unofficial rules of British politics is to avoid historical allusions. Especially those relating to World War Two. They are open to interpretation.
So it is with Ben Wallace and his reference to the 1938 Munich agreement. Some have inferred from this that the defence secretary was accusing some allies of seeking to appease President Putin. Mr Wallace's advisers insist not.
They say he was instead suggesting it was possible the Russian leader had already decided to invade Ukraine, in the same way Hitler had already decided at Munich to occupy part of Czechoslovakia.
In other words, he was suggesting that all the current diplomacy may come to naught and the West should be ready for that.
At most, Mr Wallace was implying that some allies might be better to judge Mr Putin on his actions and not his words.
All of this is not to suggest there are no differences among Western leaders. Some European countries are less forthright than the United States in their predictions of a possible invasion of Ukraine.
Others in Europe are fearful of losing access to Russian energy and are more cautious about what sanctions to impose if it were to invade.
That aside, one of the striking aspects of this crisis is how Western powers have managed to remain largely united in their broad strategy of deterrence and diplomacy, regardless of differences over individual tactics and measures.
More than a dozen countries have urged their citizens to leave Ukraine immediately, including Britain.
On Saturday, Armed Forces Minister James Heappey said advice to UK nationals in Ukraine had changed because Russia could now attack "at no notice".
Mr Heappey warned that the RAF would not be able to "fly in" and rescue those who chose to stay, comparing the situation to the evacuation of Afghanistan last summer.
US Secretary of State Antony Blinken has said the "imminent" threat of military action justifies evacuating the US embassy in Kyiv.
Yvette Cooper, Labour's shadow home secretary, said a "very strong and united international response" was needed for "this Russian belligerence".
"It is immensely serious and nobody should be naive about what Russia is up to, and the scale of the troops amassing on the Ukraine border," she told BBC One's Sunday Morning programme.
Ms Cooper said there had to be "extremely strong and swift and severe" sanctions in the face of any incursion.