The fire jolted the neighborhood into action. Several people raced to try and save residents from the flames.
"We were just hoping that all the kids got out, that everybody got out and that's not what happened," said neighbor Pat Honneyman.
Fire crews arrived and quickly launched an attack and a rescue operation.
"Some of the people that were able to safely evacuate did report that there were possibly two people that were still inside the structure so we went in to rescue," said Matthew Lucett from the San Mateo County Fire Department.
Thirteen people were in the house when the fire started. Eleven were able to get out -- some from a second-floor window.
One of the children tossed out of the window was a 1-year-old baby. The infant, along with two parents, were taken to San Francisco General Hospital for smoke inhalation.
Throughout the morning, extended family from around the Bay Area came to the fire scene to figure out what happened and console each other.
"The family's not doing too good. They're very sad," said relative Patricia Haro.
Emotions boiled over during the day. One family member tried to get into the house and was held back by other family members and police.
Four generations of the family lived in the house. The grandfather who died could barely walk and had only recently moved to the Bay Area from Mexico about three months ago.