Ash Regan quit as community safety minister, saying her conscience would not allow her to support the plans.
Her resignation came hours ahead of the gender recognition reform bill being passed in principle at Holyrood.
Seven SNP MSPs voted against and two abstained in the biggest rebellion since the party came to power.
It was also the first time an SNP minister had quit over government policy since 2007 but, despite the dissent, the stage 1 vote at Holyrood saw it comfortably passed by 88 to 33 with four abstentions.
The bill aims to speed up the time it takes to obtain a gender recognition certificate (GRC) and drops the requirement for a medical diagnosis of gender dysphoria.
It also lowers the age for obtaining a GRC from 18 to 16.
The Scottish government argues the changes would make the process simpler and less traumatic, but the proposals are controversial with some arguing it would undermine women's rights and safety.
The SNP politicians who voted against were: Stephanie Callaghan, Fergus Ewing, Kenny Gibson, Ruth Maguire, John Mason, Ash Regan and Michelle Thomson. Annabelle Ewing and Jim Fairlie abstained.
The Scottish Conservatives allowed their MSPs a free vote, with many voting against, while Scottish Labour said it would support the general principles but would seek amendments at a later stage.
Ms Regan was one of 15 senior SNP politicians - including current Finance Secretary Kate Forbes and Business Minister Ivan McKee - who signed a letter in 2019 urging the Scottish government not to "rush" into "changing the definition of male and female".
In her resignation letter, she said: "I have considered the issue of Gender Recognition Reform very carefully over some time. I have concluded that my conscience will not allow me to vote with the government at the Stage 1 of the Bill this afternoon.
"Consequently, I am writing to resign my position in the Scottish government as minister for community safety. I have greatly valued the opportunity over the last few years to work in government with colleagues to build a better Scotland."
She later tweeted that she believed the legislation "may have negative implications for the safety and dignity of women and girls".
First Minister Nicola Sturgeon said she accepted the resignation, but noted that Ms Regan had not approached her to raise her concerns.
She said: "I note that at no stage have you approached me - or indeed the cabinet secretary for social Justice - to raise your concerns about the Gender Recognition Reform Bill or the vote this evening.
"However, in circumstances in which a minister is unable to support the government, it is the case that the only options available are resignation ahead of the vote or dismissal thereafter. I therefore accept your resignation."
The Scottish government argues the bill would streamline a process that is currently invasive and distressing for an already marginalised minority group.
Presently, people must apply to a UK gender recognition panel, provide two medical reports as well as proof of having lived in their acquired gender for at least two years. They must also swear an oath that they intend to live in that gender for the rest of their lives.
The new bill would retain the requirement to swear an oath - with potential prosecution for false statements. They would need to live in their acquired gender for three months, followed by a three-month "reflection period" before a certificate was issued, and applications would be handled by the Registrar General for Scotland.
Some campaigners, however, believe that allowing people to "self-identify" could have an impact on rights that women have fought for decades to secure. There is also concern about the impact on women-only spaces, including changing rooms, hospitals and refuges.
One of the most high profile opponents of the proposed changes has been the Harry Potter author JK Rowling who says that her life has been "shaped by being female" and that "removing the concept of sex removes the ability of many to meaningfully discuss their lives".
Ms Rowling recently supported a demonstration against the bill by tweeting a picture of herself wearing a T-shirt describing Nicola Sturgeon as a "destroyer of women's rights".
The Scottish Conservatives urged other parties to allow their MSPs a free vote on the legislation when it returns to parliament.
The party's equalities spokesperson Rachael Hamilton said: "Despite the SNP's attempt to whip their members into a united front, it's clear that many of their own MSPs are deeply worried about these proposals.
"The fact that nine SNP members felt compelled to rebel against this controversial legislation shows the seriousness of the objections held by MSPs right across the chamber."