Fremont Fire Dept. Acting Battalion Chief Dan Brunicardi said the car went through the first floor, which is vacant.
Fremont police said fire crews responded at 5:47 p.m.
"We were able to secure the car in that we were able to extinguish the fire and extract the vehicle from the building to remove all that smoke and heat from the building," Brunicardi said.
MORE: Tesla crashes into back of San Mateo home, police say
Brunicardi said smoke reached the upper floors of the building so everyone was evacuated.
Tom Vo lives on the fifth floor and said the building shook on impact. Once he heard the fire alarm, he grabbed his cat Katzu.
"My window is wide open, I heard this loud screech right before that I basically like - that person or whoever was happening they were hitting the object before they went into that building, and pretty much I heard a loud explosion, I literally thought it was a bomb," Vo said.
Fremont police confirmed the driver died, and no one else was in the car.
MORE: Orinda home gets crashed into for 2nd time in 2 years
"Right now because the vehicle still has some hot spots, we're continuing to apply water to the vehicle to make sure that it's out. And then we're waiting for our Fremont PD to start their investigation into what happened," Brunicardi said.
Debra Martin lives in a nearby building. She said the driver nearly hit her as she was driving back from the grocery store.
"He was going fast I would say like 100 miles an hour - it was fast," Martin said.
Martin described the driver was swerving and losing control of the vehicle.
"I just saw it was a Tesla but I couldn't see who was in it if it was a guy or girl, I have no idea. It happened so quick," Martin said.