Officials denied the permit for Civic Center Park due, in part, to concerns over potential violence and lack of security.
Two days after Berkeley Mayor Jesse Arreguin called on his constituents to stay away from a so-called "Anti-Marxist Free Speech" rally in his city, counter-demonstrators say they'll ignore that request.
FULL LIST: Counter demonstrations planned for San Francisco, Berkeley rallies
"We don't have an option to not fight back," said Tur-Ha Ak with the Anti-Police Terror Project and Community Ready Core. "We're not advocating violence, but we are saying you have to fight. So fight from where you stand. You choose the method, but fight."
Amber Cummings, the organizer of Sunday's "free speech" event claims the gathering from 1 to 5 p.m. is not about racism or white supremacy, but a coalition of community activists isn't buying it.
"To the fascists, we do not want you here and we are not going to tolerate you being here," said Dan Siegel, with the San Francisco-Bay Area Chapter of the National Lawyer's Guild and the Oakland Justice Coalition. "You will find that out soon enough."
RELATED: Permit approved for right-wing rally at SF's Crissy Field
In an effort to avoid a repeat of the fights and violence that happened in Civic Park earlier this year, the Berkeley City Council last week passed an ordinance that gives police greater latitude to enforce bans on certain items and set perimeters beyond the rally at Civic Park.
"The police have tons of power," said Judith Mirkinson, Vice-President of the local chapter of the National Lawyers Guild."They didn't need extra power. What they're doing is they're codifying giving the police unprecedented power and mostly its to stop us."
Earlier this week, Berkeley Mayor Jesse Arreguin defended his city's handling of the upcoming rally, saying while he did not agree with the message, his city has no choice but to allow the event in the name of "free speech."
RELATED: New breed of white nationalist leadership based in California
The city is working on a specific list of banned items but has yet to release it. A city spokesperson told ABC7News it will be made public at least 24 hours before Sunday's rally.