Nicola Sturgeon was speaking as the UK government formally moved to block the legislation that was passed by the Scottish Parliament last month.
The reforms are intended to make it easier for trans people to change their legally-recognised sex.
But the UK government says the changes could impact on equality laws that apply across Great Britain.
Scottish Secretary Alister Jack is to use a Section 35 order to prevent the Gender Recognition Reform (Scotland) Bill from gaining royal assent - the first time this has been done since the Scottish Parliament was created in 1999.
The Scottish secretary is able to block legislation passed by Holyrood if he believes it would have a detrimental impact on areas that are reserved to Westminster.
Mr Jack told the Commons the reforms would have an adverse impact on single sex clubs, associations and schools and protections such as equal pay.
And he said having different processes across the UK would create "significant complications" and could lead to "more fraudulent or bad faith applications".
Ms Sturgeon told BBC political editor Chris Mason that Mr Jack was making a "profound mistake" and accused him of launching a "direct attack on the institution of the Scottish Parliament".
She said it would "inevitably end up in court" and that the Scottish government would "vigorously defend this legislation".
The first minister added: "In doing so we will be vigorously defending something else, and that is the institution of the Scottish Parliament and the ability of MSPs, democratically elected, to legislate in areas of our competence.
"In short, we'll be defending Scottish democracy".
Ms Sturgeon said the legislation would not automatically go to the Supreme Court, as would have happened if the UK government had used another section of the Scotland Act to challenge it.
So the Scottish government is instead likely to seek a judicial review of Mr Jack's decision at the Court of Session in Edinburgh.
The first minister also accused the UK government of attempting to stoke a "culture war" over the issue, and of being "increasingly hostile to the Scottish Parliament".
Ms Sturgeon added: "This is not the first attack on the Scottish Parliament we've seen, but it is the most serious to date".
Mr Jack said he was not making the Section 35 order lightly, and acknowledged it was a "significant decision".
But he said the mechanism had been put in place by the architects of devolution for good reason, with Donald Dewar - Scotland's inaugural first minister - describing it as a "sensible measure to ensure that devolved legislation does not have adverse impacts on reserved matters".
Mr Jack added: "We should be clear that this is absolutely not about the UK government being able to veto Scottish Parliament legislation whenever it chooses as some have implied.
"The power can only be exercised on specific grounds and the fact that this is the first time it has been necessary to exercise the power in almost 25 years of devolution emphasises that is not a power to be used lightly".
The UK government has published a full statement of reasons alongside the order which it says sets out in full the adverse effects it is concerned about.
Mark Drakeford, the Welsh first minister, condemned the UK government's move as a "very dangerous precedent" for devolution.
Mr Drakeford has said he would like to introduce similar gender recognition reforms to Scotland, but the Senedd does not have the powers to pass such legislation.
Earlier on Tuesday, the UK government's education secretary denied being at odds with the prime minister over gender recognition policy after appearing to suggest 16-year-olds are old enough to change their legal sex.
Gillian Keegan appeared to quickly row back on remarks - made as she toured broadcast studios on Tuesday - in which she seemed to indicate that young people could be mature enough at 16 to make the decision.
The lowering of the age from 18 to 16 has been one of the most controversial aspects of the Scottish reforms, with both Rishi Sunak and Labour leader Keir Starmer saying they have concerns about it.
When asked by Sky News on whether she thought 16 was too young, Ms Keegan replied: "No I don't actually. I was working at 16, I was paying tax at 16, I was making decisions for myself at 16."
In a subsequent interview with ITV's Good Morning Britain, she said she does not have a "strong opinion" and was referring to the fact she "felt able to take decisions for myself at 16".
Scottish Labour backed the lowering of the minimum age, with the party's MSP Monica Lennon criticising Mr Starmer's concerns as "unhelpful and ill informed", adding: "I don't think Keir Starmer can comment with the same insight as us.
"He didn't sit in the committee, he didn't sit in our chamber, he didn't have a vote and he didn't follow the evidence like we did."
The Gender Recognition Bill was passed by 86 votes to 39 in the Scottish Parliament last month, and is intended to streamline the process in Scotland for changing someone's legally recognised sex.
The reforms have been highly controversial, with several SNP backbenchers voting against the bill and Ash Regan quitting as community safety minister in protest.
The bill would lower the age that people can apply for a gender recognition certificate (GRC) - a document confirming a change of someone's legal sex - from 18 to 16.
It would also remove the need for a medical diagnosis of gender dysphoria, with applicants only needing to have lived as their acquired sex for three months rather than two years - or six months if they are aged 16 or 17.
Trans campaigners welcomed the reforms, but critics of the plans are worried that allowing anyone to "self-identify" as a woman could affect women's rights and access to single-sex spaces like refuges and changing rooms.
Scottish Labour, who supported the bill at Holyrood, called on Scottish and UK ministers to find a solution to the impasse.
Shadow Scottish Secretary Ian Murray has accused both governments of attempting to "break" devolution and argued that "trans rights and women's rights should not be used as an excuse for SNP-Tory attrition warfare".
Nicola Sturgeon says the UK government is ‘stoking a culture war’ over gender reform bill
Alister Jack says gender reform bill was blocked to protect women's safe spaces