It is on Martha Street between 7th and 8th Streets and happened around 1:30 a.m.
Video shows flooding and mud all over the road, which is closed while crews clean up.
San Jose Water says 24 customers are affected.
They expect repairs to be completed by 6 p.m., but the cleanup will likely continue into the night.
As of 11 a.m., repairs have arrived on site as crews continue to clean the area and expose the pipe.
Stay with ABC7 News for updates on this developing story.