One of Ali's most famous fights came in Zaire in 1974, The Rumble in the Jungle. Lee said he sparred with Ali days before that fight.
READ MORE: Watch Muhammad Ali funeral live on Friday
Lee spent a lot of years in the fight game, starting when he got out of the Navy in the mid 1960's.
Under the name "Irish Terry Lee" he punched his way to the California state light heavyweight championship.
RAW INTERVIEW: Local boxer speaks out about sparring with Muhammad Ali
But, the pinnacle came when he went to Oakland to check out the new heavy weight champion -- George Foreman. "He looked at me and said 'hey, you want to spar?' And I said 'sure' and he saw the speed I had and stuff," Lee said.
Foreman needed to spar against someone fast and elusive like Ali, so they took Lee with them to the Rumble in the Jungle in the African Country of Zaire, which is now known as the Democratic Republic of Congo.
READ MORE: Muhammad Ali's life celebrated by celebrities, athletes
Even though he worked for foreman, his heart was in the other camp. "The first time with Muhammad Ali was when I went to the gym over there and asked them if I could spar and he said 'yes, you're fast for a white boy,"' Lee said.
He was ringside the night that Ali recaptured the title with a stunning eighth round knockout when many people thought Ali was washed up.
READ MORE: Muhammad Ali remembered in prayer as icon who pushed for unity
Lee said it was difficult watching the champion's health deteriorate over the years.
It's even more difficult watching preparations for a memorial service in Ali's hometown of Louisville, Kentucky where Muslim prayers are being said in advance of Friday's funeral.
Lee said sparring with the legend taught him a lot. "He was very intelligent and very fast in the ring and he could take a punch. If you can't take a punch when you're a heavyweight, you better give it up," he said.
He said in or out of the ring, there was only one Ali. "He was just the greatest," Lee said.
"Irish" Terry Lee of Concord sparred with Ali in Zaire. His memories on ABC-7 news. pic.twitter.com/nwA3pf500H— Eric Thomas (@ericthomaskgo) June 9, 2016