The fire was on the 2800 block of Cicero Way.
Nearby structures were evacuated while firefighters responded to the fire.
"The fire was fast-moving, it was dangerous. It was wind driven, there was wind in the area. Really help push it, move it quickly, " says Battalion Chief Robert Culbertson with the San Jose Fire Department.
Culbertson says it took two hours to put out the fire that destroyed the duplex in East San Jose.
Culbertson said, initially, they had reports of people inside.
"With rescue on our mind, we thought there may be somebody in there. We did an aggressive interior fire attack. And it took us about two hours to successful put this fire out," Culbertson said.
MORE: Dozens of Oakland Chinese seniors, tenants in limbo after fire; advocates ask community to help
Just after 3 p.m. calls came in for the fire. Once on scene, Culbertson says they upgraded it to a two-alarm fire.
"And what that means, for our system, is about 40 to 50 firefighting personal. There is also law enforcement and some ambulances with us," Culbertson said.
"I want to say, in maybe 10 to 15 minutes, it went from a small little fire in the front, to engulfing the entire house. I was like, 'Wow.' It surprised me a lot," said Kevin Flores.
Flores grew up in the neighborhood and lives in the next duplex over. At first, he thought the smell of smoke was the neighbors barbecuing. But then, he saw big plumes of smoke. He tried to use his garden hose to spray the fire.
MORE: 1 dead, fire captain injured in Mountain View mobile home fire, authorities say
"I pulled it over, I yanked it, and that's when I tried spraying it away. But in reality, it was so big that the hose doesn't really do much. The firefighters took like a good hour to even get it to calm down, so I was never going to do anything with this tiny little hose," Flores said.
The Red Cross tended to the families, who gathered on the street with a few belongings that they could salvage. Fire officials say everyone made it out alive. But two dogs died in the fire.
Culbertson said along with the wind, how close the homes are built to one another posed another serious challenge.
"The closely arranged patterns of these homes, the garages that had materials in them, you see the trees in between these homes. Also very flammable. Those also contribute to it," he said.
One firefighter was taken to the hospital with non-life-threatening burn injuries. The cause of the fire is still under investigation.