Before the vote, and during the public comment portion, audience members began fighting with each other and the meeting had to be recessed for a period of time.
Hsu has come under controversy in recent weeks over statements she made about black and brown students and their family learning environments.
"I misspoke while trying to discuss these serious issues. And, in doing so, I said things that unintentionally perpetuated harmful stereotypes," Hsu said.
Hsu apologized following the uproar and again at Tuesday's meeting, but for many it wasn't enough.
Reverend Arnold Townsend is the vice president of the San Francisco chapter of the NAACP.
He's calling on Hsu to resign immediately.
Townsend points to a recalled school board member who also faced criticism for racially insensitive comments.
VIDEO: SFUSD Board Vice President Alison Collins apologizes for offensive tweets aimed at Asian Americans
SFUSD Board Vice President Alison Collins apologizes for offensive tweets aimed at Asian Americans
"If Alison Collins' words hurt Asian children, certainly, what does these words do to black and brown children," Townsend said.
His frustration also shared by many others in attendance.
Many people came to say they thought Hsu has no place on the SFUSD board.
"We do love our babies, and we do value education. We do believe in Black excellence and Latinx excellence," said parent Rionda Batiste.
But not everyone felt that way.
Dozens also turned out in support of Hsu, saying her comments were taken out of context.
VIDEO: SFUSD Board VP responds to controversial 2016 tweets aimed at Asian Americans
SFUSD Board VP responds to tweets targeting Asian Americans
"We need someone better. We need Ann Hsu. We need Ann Hsu to speak," said parent Selena Chu.
Many of Hsu's supporters say they think Hsu was simply trying to point out systemic inequalities in education.
Not because she harbors racist views, but instead because she wants to help those children in particular.
"If we want to fix the problem we have to be able to talk about root causes even if they're uncomfortable for some people," said support Diane Yap.