Part of the Street Art Trail, that was painted in Port Talbot, was defaced on Friday.
An emergency meeting was organised and the graffiti was cleaned up by members of the ARTwalk community group.
Swastikas, the words "Nazi zone" and a racial slur were painted on the mural. South Wales Police said the "despicable hate crime" was being investigated.
It also included the numbers 1488, a code used by neo-Nazis and white supremacists to broadcast hate speech in a covert manner and to show their alliance with others in their movement.
Jalisa Phoenix-Roberts said the graffiti had "shaken the general community" and caused "uproar".
The mural had been painted to recognise the contribution of people from the Caribbean to the town.
"When it's on your doorstep it feels worse and I guess I was slightly ignorant thinking this kind of thing doesn't go on in Port Talbot any more, but it does and it has," Jalisa added.
"It's really upset the community in general, not just the black community in Port Talbot, massive uproar on my Facebook, with nothing but support which is lovely to receive after such an awful time.
"But it's quite shocking literally a couple of hours after the art was made, I think that's the worst part."
Resident Martin George Andrews said his initial feelings had been of anger but he then began thinking of how his energy could be "best served".
He said he decided to try and use it to raise awareness instead, adding: "Like making people aware of what's going on and then the possibility of educating and having a conversation to try and find understanding, because it baffles me, to be honest.
"Not only is it a mural to represent Caribbean culture within Port Talbot, it's also a memorial. I just find that in itself is levels and levels of disrespect.
"It's hurtful not just for me but for people in the community. It just sheds a terrible light on Port Talbot."
A South Wales Police spokeswoman said: "This is a despicable hate crime that will not be tolerated. We will do everything we can to identify the people responsible for this graffiti."