The animated crowd spilled across Polk Street, holding signs with messages including "Stop the Violence," "Asians are not punching bags," and "Stop attacking the elders and vulnerable."
Andy Lee, 39, carried a sign bearing images of Oscar Grant III and Tian Sheng Yu, a 59-year-old San Francisco man who died last month after being beaten in a random attack in downtown Oakland while he was shopping with his son. Two suspects have been arrested in the attack.
Grant, who was shot and killed by former BART police Officer Johannes Mehserle at an Oakland BART station in 2009, is black.
Lee, a San Francisco resident, said he simply wants the suspects in both cases to pay for their alleged crimes.
"It's not a color issue ... there should be justice for what they did," he said.
In San Francisco, there have been at least three recent attacks against Asian-Americans at or near San Francisco Municipal Railway T-Third light rail stops. One of the victims, 83-year-old Huan Chen, died in March after being beaten at a bus stop near Third Street and Oakdale Avenue.
"No matter what, if (the victims) are Chinese, Caucasian, you don't have the right to come out and beat up other people," Lee said.
A number of city supervisors attended the rally, including Board of Supervisors President David Chiu, who called the recent violence "tragic" and "horrific."
Chiu mentioned a number of positive changes since the recent attacks, including meetings between politicians and community leaders, more police officers assigned to the southeastern part of the city and a bilingual drop-in center where Asian victims can go to report crimes.
"The Chinese character for crisis is opportunity, and I want to thank you for being part of the change that we are going to see," Chiu told the crowd.
"This is our problem. This is our city," Chiu said. "We have to take our city back."
Supervisor Carmen Chu said the board understands that people are angry not only about the recent violence but about other traumatic experiences they have had over the years.
She said it is the board's responsibility to make sure residents feel safe doing things like going to the grocery store and riding Muni.
"When you need us to be in the community to listen to you, we will be there," she said.