The man fell around 9:40 a.m. Wednesday as he worked on a boom truck in the area of South Second and Martha streets, about two blocks east of Washington Elementary School, Officer Albert Morales said.
Emergency crews pronounced the man dead at the scene a short time later, according to Morales.
"We talked to one witness. They saw what they thought was someone falling from the boom truck, but we don't have anyone that actually saw him fall," said San Jose Police Lt. Paul Spagnoli.
Police say the victim may have been in his 40s.
The fall appears to have been an industrial accident and was referred to Cal/OSHA, or the California Division of Occupational Safety and Health, said Morales.
Residents say they often see crews working in this area fixing power lines, clipping trees or working on the lights.
"I see them all the time putting their cones out, their hard hats, they wear their safety belts, they have the cones pretty much far away from the curb, and where the traffic is passing, so that nobody's in danger," said Kathleen Vigil, a nearby resident.
The worker was employed by facility management company ABM Electrical & Lighting providing electrical and lighting services, according to the California Division of Occupational Safety and Health.
ABM is a subcontractor of OpTerra Energy Services, a San Francisco-based company, which entered a contract with the city last year to retrofit some 18,000 streetlights with LED lighting, according to city spokesman David Vossbrink.
The bulk of the work was completed last month, with a couple hundred conversions left to be completed this summer, according to Vossbrink.
About one-third of the city's total streetlights have been assigned to OpTerra, Vossbrink said.
"We are deeply saddened by this morning's tragedy, and our prayers are with the deceased's family and coworkers," John Gajan, vice president of operations at OpTerra, said in a statement.
"We are coordinating with our subcontractor, ABM, and the City of San Jose to help determine the cause," Gajan said.
ABC7 News Reporter Chris Nguyen contributed to this story.