NEED TO KNOW

• Jack O'Halloran retired from professional boxing in 1974 as he committed to acting
• During his time as the California heavyweight champion, O'Halloran, now 83, was working toward fighting Muhammad Ali
• O'Halloran recalls knowing Muhammad and fighting his brother, Rahaman Ali, in conversation with PEOPLE

Jack O'Halloran had a storied career as a boxer.

The Superman star, 83, was a heavyweight champion in the years before he got into acting. During a recent conversation with PEOPLE, O'Halloran opened up about coming close to fighting late boxing legend Muhammad Ali.

"Ali and I were signed to fight four times. He fought a fellow from Detroit named Alvin 'Blue' Lewis. Blue Lewis fought him in Ireland, went 13 rounds with him. Then Blue Lewis came back and he beat Ernie Terrell and beat another ranked fighter," he explained.

"Then they called me in California and asked me if I wanted to fight Blue Lewis. I went out and I beat him in Detroit and then I went up to camp and saw Muhammad and then we decided it was time to get it on. And he fought Wepner and a few others."

Never miss a story — sign up forPEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

In 1973, O'Halloran was knocked out by Jimmy Summerville, which ended the possibility of fighting Ali. Still, O'Halloran cherished his relationship with his fellow fighter.

"We were good friends. He was an interesting individual," he shares of Ali.

While pursuing the Muhammad Ali fight, O'Halloran did fight his brother, Rahaman Ali, in 1972, in what ended up being Rahaman's final fight.

"We fought in San Diego when I was the California heavyweight champion. I ran off about four or five wins in a row and Muhammad called me on the phone. He said, 'You got to do me a favor.' I said, 'I'll do you a favor. Sign a contract,' " he laughed.

"He said, 'No, no, no, you're fighting my brother Rahaman and you got to get him out of boxing.' I said, 'Rahaman Ali's your brother?' He said, 'Yeah.' I said, 'Well, I better go in the gym a couple of days then.' His brother was undefeated until he fought me and then he never fought again. I knocked him out in the ninth round and he never fought again."

O'Halloran stayed connected to the boxing community, even after moving on to acting.

"When I retired from boxing, I was so used to going to the gym to workout. I kept going and I saw this young fighter come in from Philadelphia, Frankie Liles. I took him and I trained him and managed him for the super middleweight championship of the world. So I trained and managed a champion. I started Freddie Roach out, who became a prominent trainer, trained Pacquiao and a bunch of others."

Read the original article on People