First Minister Nicola Sturgeon wants to hold a referendum on 19 October next year.
But the court ruled unanimously that she does not have the power to do so because the issue is reserved to Westminster.
The UK government has refused to grant formal consent for a referendum.
Court president Lord Reed said the laws that created the devolved Scottish Parliament in 1999 meant it did not have power over areas of the constitution including the union between Scotland and England.
These issues are the responsibility of the UK Parliament, he said, and in absence of an agreement between the two governments the Scottish Parliament is therefore unable to legislate for a referendum.
He also rejected the Scottish government's argument that any referendum would simply be "advisory" and would have no legal effect on the union, with people only being asked to give their opinion on whether or not Scotland should become an independent country.
Lord Reed said: "A lawfully held referendum would have important political consequences relating to the union and the United Kingdom Parliament.
"Its outcome would possess the authority, in a constitution and political culture founded upon democracy, of a democratic expression of the view of the Scottish electorate.
"It is therefore clear that the proposed Bill has more than a loose or consequential connection with the reserved matters of the Union of Scotland and England, and the sovereignty of the United Kingdom Parliament."
Responding to the outcome, Ms Sturgeon said she was disappointed but respected the ruling of the court, and stressed that the judges do not make the law and only interpret it.
She added: "That is a hard pill for any supporter of independence, and surely indeed for any supporter of democracy, to swallow."
The first minister told a media conference that a referendum remained her preferred option, but in the absence of an agreement the SNP would use the next UK general election as a "de facto referendum" in an attempt to demonstrate that a majority of people in Scotland support independence.
The "precise detail" of how this would work will now be a matter for the party to debate, she said, with a special conference to be held in the new year.
Ms Sturgeon said: "We must and we will find another democratic, lawful means for Scottish people to express their will" and accused the UK government of "democracy denial".
Scottish Labour leader Anas Sarwar said there was not a majority in Scotland for either a referendum or independence, but there was a "majority in Scotland and across the UK for change".
The case was referred to the Supreme Court by Lord Advocate Dorothy Bain KC, the Scottish government's top law officer.
Ms Bain said at the time that she did not have the "necessary degree of confidence" that Holyrood would have the power to pass legislation for a referendum without UK government consent.
She said the issue was of "exceptional public importance" and asked the UK's top court to provide a definitive ruling.
The court heard two days of legal arguments from both the UK and Scottish governments last month, with its ruling being delivered just six weeks later - earlier than many experts had expected.
The independence referendum in 2014, in which voters backed remaining in the UK by 55% to 45%, was possible because the UK government agreed to temporarily transfer the necessary powers to the Scottish Parliament to allow the vote to be held through what is known as a Section 30 order.
Supreme Court president Lord Reed read the verdict of the justices