The statue depicts Fallon in 1846, raising the U.S. flag in San Jose.
California was still a part of Mexico at the time, and opponents of the statue, unaffiliated with the vandalism, want it removed. Organizers, like Rebeca Armendariz of the CARAS Advisory Board, say it's a symbol of oppression.
RELATED: Mayor Breed calls for review of public art after crowds topple 3 historic statues in San Francisco
"When I pass by the Fallon statue it feels like a slap in the face to all the progress that we've made," Armendariz explained. "To the struggles that we've had."
San Jose Mayor Sam Liccardo says it's now time for residents and leaders to come up with a plan.
He says he's already started talking to historians.
WATCH: Several hundred people topple Francis Scott Key statue in SF
"What we need to do rather than burn is we need to learn," Liccardo said. "Part of that learning may be facing some ugly truths about our history."
There was also controversy when the statue was commissioned in 1988, and the city decided to install four other statues to show more perspectives of city history.