Four people living in a unit above the garage managed to get out safely.
Fire officials say the incident off Foss Avenue was reported around 5:45 p.m. Monday.
Graham Flanagan, captain with the San Jose Fire Department said the incident briefly reached to a three-alarm fire.
"We need to make sure things are completely out, the last thing we want is a re-kin so we're going to take our time, put things together in an orderly fashion and get a salvage and overhaul plan make sure the fire is completely cold," Capt. Flanagan said.
The temple is located on a corner tucked inside of a neighborhood, making it somewhat difficult for first responders to access.
"The major challenge was the fact that this was a large dead-end. And so we actually had to use two elevated master streams, we went into a defensive posture," Flanagan said.
Members from the Buddhist temple were across the street still grappling with the amount of damage the fire caused.
One man told me he was one of the four residents inside when he heard the fire break out.
MORE: Investigation underway after 3-alarm fire burns Oakland home, church
He didn't want to go on camera but said he's glad everyone is OK.
Fire officials say the four people displaced are getting services from another Buddhist temple nearby.
Flanagan said this does not appear to be a hate crime.
"Because it's a place of worship, there's always a possibility there's a nefarious element to it - this does not appear to be that," Flanagan said.
San Jose fire says the cause is still under investigation.