On live radio, Nimco Ali said she was on a "completely different planet" to the home secretary when it came to women's rights and ethnic minorities.
Ms Ali was appointed when her friend Boris Johnson was prime minister.
A source close to the home secretary said Ms Ali's contract was coming to an end before Christmas.
They added: "The home secretary is determined to make our streets and homes safer for women and girls.
"She will continue to focus on this policy and the rights of women and girls to live safely in our country."
The Home Office said they had not received any formal resignation and were therefore not commenting.
Ms Ali was being interviewed on Times Radio when she said she would not "feel comfortable in serving under Suella".
"Suella and I are on completely different planets when it comes to the rights of women and girls and also the way that we talk about ethnic minorities," she said.
Ms Ali said Ms Braverman had different approaches to "people like me who are from a refugee background".
She also questioned if the home secretary shared her "feminist ideals".
When pressed on whether she would remain as an adviser, she said "ultimately no, I'm not going to continue".
The move comes hours after the home secretary agreed to back a bill criminalising street harassment.
Ms Ali had a major role in preparing the ground work for the bill and led a consultation into making street harassment a specific crime while serving under former home secretary Priti Patel.
Ms Ali was appointed to her role by Ms Patel in October 2020.