Richard Lee said he had hoped to meet the prime minister in Downing Street but was "devastated" after a surprise meeting on Monday.
He said he was told about the meeting minutes before and was not prepared.
Katrice Lee vanished on her second birthday in 1981, near a British military base in Germany.
Mr Lee believes his daughter was abducted and the case has been raised again in recent months in the House of Commons.
In November Mr Johnson said he would meet Mr Lee "father to father" and it was hoped the meeting would take place in Downing Street.
However, a previous date for a meeting at Downing Street in March was cancelled due to the situation in Ukraine.
A planned meeting between Mr Lee and Margaret Thatcher in 1982 was also cancelled because of the Falklands War.
Mr Lee said he received a phone call "out of the blue" last week asking him to take his file on Katrice's disappearance on Monday afternoon to the Hartlepool office of local Conservative MP Jill Mortimer to discuss "a few points on the case".
Mr Lee said: "When I arrived I was taken to a conference room where I put my file on the table, ready to clarify a few points and it was then I was told I was meeting the PM in a few minutes.
"To say I was angry and upset is an understatement, I've been fighting for a meeting with a British prime minister for 40 years and I was infuriated it was happening with no notice.
"I had a presentation ready, but it was at home on the kitchen table - luckily I had a copy on my phone."
Katrice had been with her mother Sharon, from Gosport, at a Naafi supermarket in Paderborn in November 1981 when she disappeared.
The Royal Military Police has previously apologised for failings in its investigations, which included delays in interviewing key witnesses and not releasing a photofit of a suspect for 36 years.
Mr Lee said he discussed the case with Mr Johnson for about 15 to 20 minutes and asked for an independent inquiry into how his daughter's disappearance was handled.
He also called for the case to be handed to civilian police and for the government to pledge £1m to the charity Missing People, which helps families trace loved ones who disappear overseas.
Mr Lee said: "I walked out of the office after the PM had looked across at Jill Mortimer and said he could probably help with one of the points, but I can't remember which one as I was so angry.
"I'm annoyed we live in a country where we have freedom of speech and freedom of choice, but both of those things were taken away from me on Monday."
Mr Lee added he was waiting for the prime minister to get back in touch following the meeting.
Both Mr Johnson's and Ms Mortimer's offices have been approached for comment.