The first 911 call about the crash came in at 4:40 p.m., according to San Francisco Fire.
MORE: 2 dead after car goes over cliff at Devil's Slide in San Mateo Co., CHP says
For some unknown reason, two vehicles traveling on Point Lobos Avenue veered off the roadway, through a fence, and onto the cliff below, SFFD said.
Video from the scene appears to show the vehicle may have hit the building, breaking a few windows and leaving what look like tire marks on the wall.
Both vehicles contained single occupants. One has died and the other received minor injuries.
San Francisco Police have taken over the investigation into the crash.
Point Lobos Avenue remained closed in both directions as of 5:40 p.m.