Sgt. Jason Dwyer said Thursday that a group of young men walked up to a man in his late 30s or 40s and began stabbing him several times in the torso in the city's Kenwood neighborhood shortly after 8:30 p.m.
The victim was later pronounced dead at a local hospital.
Dwyer said police do not yet have a motive as it's still unknown whether the victims knew his attackers or if the incident was random.
No arrests have been made. The victim's name has not yet been released.
It was San Jose's second homicide in two days. On Wednesday, 43-year-old Anna Duong, a mother of two, was shot and killed in her condominium.
No arrests have been made in that killing.