These allegations come in the aftermath of two high profile incidents in less than a year.
The nurses and the hospital workers are members of SEIU Local 1021and they say the understaffing is especially serious in the emergency room and say it's happening on every single shift.
On September 19, Lynn Spalding, 57, was admitted to San Francisco General Hospital with a bladder infection.
She vanished two days later and was found dead in a hospital stairwell 17 days after that.
On February 14, Mary Perez, 93, was admitted to the hospital with breathing problems.
She was brutally attacked in the emergency room by another patient who was in psychiatric care.
Hospital workers say these two horrific incidents are linked to severe understaffing, especially in the emergency room.
"We need adequate staffing to provide safe care to our patients and I think we don't have that on a day to day basis right now," said Jason Negron-Gonzales, a registered nurse in the ER.
The union that represents hospital workers held a rally Wednesday in front of the hospital's emergency department.
They complained of being stretched too thin, resulting in unattended patients wandering off and others being assaulted by mentally ill patients.
"We actually surveyed nurses and most of them responded that staffing and workplace safety were the main priorities," San Francisco General Hospital nurse Norlissa Cooper said.
Several investigations following the Lynn Spalding case found both the hospital and the sheriff's office, whose deputies provide security, at fault.
But administrators say the union is barking up the wrong tree.
"I want to be very clear with the people of San Francisco that we do not have a staffing issue that's linked to patient safety," San Francisco General Hospital spokesperson Rachael Kagan said.
The hospital admits it's understaffed, but says they're trying to fill those vacancies with temporary nurses and other per diem workers.
Administrators also question the timing of the rally because it's being held at a time when the union is holding contract negotiations with the city.