It happened under the freeway at the Interstate 280 onramp at Bosworth in the Glen Park neighborhood. Firefighters say the group was trapped in a crawl space, where they were seeking shelter on a night when the temperature was in the 40s.
"They couldn't even see their hand in front of their face and it was extremely hot," San Francisco Fire Lt. Jonathan Baxter.
He went on to say that one of the victims was still able to talk at the time of being rescued and essentially led firefighters to the fourth person. Baxter notes they've seen these types of fires before.
RELATED: Lower Nob Hill residents voice concerns over proposal to turn youth hostel into homeless shelter
"We're speculating at this time that this was a warming fire inside this structure," said Baxter. "It definitely has to come from our fire investigators for cause of origin, but anyone who was outside like we were it was an extremely cold morning."
The fatal fire - prompting calls for more action.
"It's devastating that people are living in conditions where they are up inside of a freeway to stay warm. The humanity of it is just heartbreaking," said Lydia Bransten, the executive director of the Gubbio Project, which provides daytime shelter at St. Johns Church in the Mission.
She says she worked with the city to try and find a warm place for people to sleep Tuesday night. But she says even when emergency shelters open on cold nights, it is hard for homeless people to get to them.
RELATED: Has SF hit rock bottom? Former mayor says city's 'humanitarian' ethic is to blame for recent issues
"You have someone who has their belongings, their tent...it's complicated. We all think here's the problem, here's the solution. But we are talking about thousands of people and we are talking about hundreds of beds," she said.
She says, while the issue of homelessness is complicated, she thinks a good start would be creating shelters that have fewer requirements for entry.
"There are lots of reasons people don't go into shelter. Couples for instance can't always stay together. But they can stay together in a tent," she said.
The supervisor for the district where the fire happened says he has tried to address the homeless crisis.
RELATED: Oakland opens new tiny home village near Lake Merritt for the city's unhoused
"More than a year ago I introduce legislation at the Board of Supervisors called 'place for all' to require the city to have some kind of exit from the street, even if it is just a safe sleeping site," said Rafael Mandelman. He says it did not get far, but now he is motivated to try again.
"It got a hearing last April. It was strongly opposed by homeless advocates and it has not moved forward. I am going to make another round of inquiries with some of my colleagues to see if maybe we can get this going again."
He believes those who speak up for the homeless and demand more permanent solutions and housing are well meaning, but he thinks the immediate needs should be addressed as soon as possible.
"We have an immediate, urgent problem. We need shelter, we need to get people off of the sidewalks, out of encampments," he said.
The fire department says the cause of the fire is under investigation.
If you're on the ABC7 News app, click here to watch live