Zoo officials, who described Gene as gentle and friendly, said he died of kidney failure on Monday.
The zoo's animal care and veterinary staff had been keeping an eye on Gene because his appetite had dropped and he was lethargic, according to zoo officials.
After a recent blood sample indicated that Gene had kidney failure, zoo staff focused on keeping him comfortable for the remainder of his life.
Gene was named after the late Eugene Friend, who served on the Recreation and Park Commission for 24 years, zoo officials said.
Gene was born in Kenya and came to the San Francisco Zoo in 1978 at the age of seven.
During his time at the zoo, Gene fathered five offspring, three of which are now living at other accredited zoos, zoo officials said.
Gene's animal keeper, Julie McGilvray, said he had a good disposition and was very friendly.
"We nicknamed him Big Dog because he loved to be rubbed, either by hand or with a scrub brush, and oftentimes he would lie down and absorb the soothing experience," McGilvray said in a statement.
Zoo officials said black rhinos are a critically endangered species because they are targeted by poachers who covet their valuable horns.
They said that while it is unconfirmed by scientific research, there is a belief in Chinese medicine that black rhinos' ground-up horns help in healing and increase sexual appetite.
Zoo officials said black rhinos use their horns for food gathering to dig up and collect roots, twigs and branches.