Currently councils can charge a second home premium of up to 100% but that will increase to 300% from April 2023.
The move, included in a Labour-Plaid Cymru cooperation agreement, is part of efforts to make it easier for people to afford homes where they grew up.
A homeowners' group said the move was "morally indefensible".
But Climate Change Minister Julie James hit back, arguing it was "the morally right thing to do" to protect communities.
"It's astounding and morally indefensible," Jonathan Martin, a spokesman for the Home Owners of Wales Group, told BBC Radio Wales Breakfast.
"Where do they think we're going to get this 300% from? I can't afford it, that's for sure and I'm quite sure a lot of other people can't afford it. It's just astounding."
Mr Martin, who lives in Altrincham, Greater Manchester, and has a second home in Gwynedd, said most of the group visited their homes regularly.
"They love Wales, they love Welsh people, they love the Welsh language, they love the Welsh culture. That's why they have a home there," he added.
Abersoch-born and bred Huw Hickey works full time but still struggles to pay his rent.
"I've moved into Pwllheli to live, because it's cheaper for me there," he said.
"There's hardly any Welsh people here now."
Even if a 300% council tax hike was imposed he believed people from outside the area would still "come here in their droves".
"Then we'll be driven out and there won't be no Welsh people here soon."
Cath Clarke, who owns a caravan near Abersoch, agrees with the Welsh government's proposal.
"If people have second homes the community is going to die," she said.
"It needs locals. If people really like it here so much then they should move here.
"I think it's really good that there's so many caravan sites, because people can come and holiday there.
"I really do disagree most strongly with second homes. It's an incredibly selfish way of going about things."
Mr Martin also criticised the timing following the pandemic and amid the rising cost of living, but said his group were ready for a fight.
"I think the biggest threat to the Welsh government will be that we've been advised it's absolutely unlawful," he added.
"So I don't know where we go from that but we'll have to have a big discussion as a group. We're financially able to take on the Welsh government if they forced this through without further acquiescence with us."
Gwynedd and Swansea are already charging a 100% premium and Pembrokeshire will do so from next month.
Welsh ministers are encouraging councils to put the extra money raised into increasing the amount of affordable housing.
The Welsh government will also tighten up the rules on self-catering accommodation being liable for business rates rather than council tax from next year.
At the moment properties available to let for a minimum of 140 days in any 12-month period, and actually let for at least 70 days, pay rates not council tax.
Under the changes properties must be available to let for at least 252 days, and actually let for at least 182 days to qualify.
The government said this would make it clearer that the premises are genuine holiday accommodation making a "substantial contribution" to the local economy.
In some parts of Wales, particularly coastal areas, second homes are a contentious issue, with people feeling they are pricing them out of the area they grew up in and threatening the Welsh-speaking culture.
Meanwhile, second home-owners argue they feel like they are being "discriminated against" for buying homes in desirable areas.
According to the Welsh government council tax premiums are being paid on more than 23,000 properties in Wales this year.
Some homes are exempt from the premium - such as those needed for employees.
In the 2022-23 tax year nine authorities will charge a premium, ranging from 25% in Conwy and Ceredigion, 50% in Anglesey, Flintshire, Denbighshire and Powys, and 100% in Gwynedd, Pembrokeshire and Swansea.
Both Pembrokeshire and Gwynedd have the largest number of second homes that are subject to a premium, at 3,746 and 3,794 respectively.
Ministers will also raise the maximum premium for empty homes to 300%.
Speaking to Radio Wales Breakfast, Josh Phillips, chairman of Solva Community Land Trust in Pembrokeshire, said the move was positive for rural communities but warned a balance needed to be struck.
"I think if we start talking about moral outcomes here, we're going to get ourselves in a bit of a tricky situation because what's the moral authority of these people to erode our communities in such a way and destroying our local culture?
"What we can't have is these premiums increasing to 300% and scaring off these people who who do play a vital role in our economy locally.
"So I think there's lots of raised voices at the moment and I think we need to stop and consider where that premium sits and how we utilise it to best benefit our local people."
Speaking to Radio Wales Breakfast in response to Mr Martin's comments, Ms James said: "This is somebody who's fortunate enough to be rich enough to own a second home, who objects to making a contribution to the community in which he wants to come and holiday.
"Obviously, the Welsh government is very clear that we have the legal right to do this and actually, it's the morally right thing to do."
She added: "This is about making sure that people who are lucky enough to own second homes make an appropriate contribution to the communities that they want to come and holiday in.
"We also encourage people to let out their houses in the private rented sector to long term rents for people who are local there."
Plaid Cymru's Sian Gwenllian said: "It's a first, but important, step on a journey towards a new housing system that ensures that people have the right to live in their community."
The Conservatives accused ministers of "punishing aspiration and investment".
Janet Finch-Saunders, who speaks for the Welsh Conservatives on housing, accused Labour ministers of "pandering to their nationalist coalition partners and punishing aspiration and investment in Wales".
"The housing crisis is a direct result of years of successive Labour-led governments failing to provide opportunities and build enough houses, with housebuilding falling below levels before devolution."
Gwynedd council said it welcomed action to address second homes to help people buy homes in their communities.
"It is vital that all dwelling houses are removed from the business rates system so that they pay council tax and any premium," a spokesman said.