The accident happened around 12:50 p.m. on Paseo Padre Parkway between Chadbourne Drive and Dorne Place.
Gilbert Gil, 34, of Milpitas, was allegedly speeding and lost control of his car, veered off the road and crashed into a tree, police spokeswoman Geneva Bosques said.
Bosques said the impact of the crash split the vehicle in half.
The toddler, Angelo Gil, was in a child safety seat in the back, but was ejected when the car came apart, Bosques said. He was pronounced dead at the scene.
Gilbert Gil was also ejected, and suffered major injuries. His wife, Joie Gil, 31, was critically injured and was pulled from the car by firefighters, Bosques said.
She was pronounced dead this afternoon, police said. Gilbert Gil underwent surgery Tuesday night and was in stable condition today.
The crash shut down a section of Paseo Padre Parkway for more than seven hours Tuesday afternoon, police said.