Challenging the decision, Prince Harry's lawyer said he had not been aware Sir Edward Young, the Queen's private secretary, played a role.
The decision was "materially prejudiced" as key information was withheld, Shaeed Fatima QC said.
The duke lost full protection after he stepped back from royal duties in 2020.
Without a guarantee of police protection, Prince Harry believes it is too dangerous to bring his family from the US to visit the UK - which his lawyer has previously said "is and always will be his home".
The duke's legal team argued the decision about his security by the Executive Committee for the Protection of Royalty and Public Figures, known as Ravec, was invalid because of "procedural unfairness".
They questioned whether it was "appropriate" for Sir Edward or other members of the Royal Household to sit on the committee, when Prince Harry had been told it was "independent".
Representing the duke, Ms Fatima said: "He didn't know at that stage that the Royal Household was involved at all."
She said Prince Harry should have been able to make direct representations to Ravec to ensure his arguments were properly communicated to the committee and been given an opportunity to respond to points being made by others, including members of the Royal Household.
Ms Fatima said his offer to pay for his own police protection "was not conveyed to Ravec" and so the decision was "materially prejudiced".
This court action is still at an early stage. Mr Justice Swift is being asked to decide whether Prince Harry has an arguable case going forward. That decision won't come for another few weeks.
However - the written and oral submissions heard in Court three this morning are headline-making because of the detail and most notably what were described as "significant tensions" between Prince Harry and the Queen's private secretary, Sir Edward Young.
No specifics were read out in court - sensitive information was kept out of the public hearing. But it does point to the breakdown in relations between the Sussexes and the Royal Household prior to their move abroad.
Whatever Mr Justice Swift decides, there may be further legal action as counsel for the Duke of Sussex has indicated a second judicial review claim has been started into whether he should be allowed to fund the security himself rather than having taxpayers pay for it.
An offer Prince Harry's counsel say was made - including in an email to Sir Edward Young in 2020 - but which was not passed on to Ravec.
Lawyers for the Home Office say Ravec was entitled to make the decision it did - to provide police protection to Prince Harry and Meghan on a case-by-case basis when they visit the UK.
Sir James Eadie QC, representing the Home Office, said in written arguments that personal tensions between Prince Harry and Royal Household officials are "irrelevant" to his change in status when he stepped back from royal duties.
He said there was "no basis" to conclude that the duke making representations to the committee would have led to a different outcome.
Prince Harry has previously argued his private security team cannot replicate the work of police protection in the UK, with their access to local intelligence and legal powers.
He and Meghan have only rarely visited since their move to California. On one visit in July 2021, the duke's car was chased by photographers as he left a charity event.
The Sussexes also stopped in the UK in April, meeting the Queen on the way to the Invictus Games, and attended the thanksgiving service during the Platinum Jubilee last month.