By Haley Draznin, Producer

(CNN) -- The most decorated Olympian of all time, swimmer Michael Phelps rewrote sports history.

But his journey to 28 Olympic medals did not come without challenge. Even in the midst of Olympic perfection, the athlete grappled with mental health issues.

Phelps, 32, opened up about his years-long battle with depression to David Axelrod on "The Axe Files," a podcast from The University of Chicago Institute of Politics and CNN.

"I'm somebody who's gone through at least three or four major depression spells after games that that, you know, I've put my life in danger," said Phelps, whose success at each Olympic Games was followed by drug and alcohol use.

After the 2012 Olympics, Phelps said he wanted to take his life, "I wanted to die. I straight wanted to die."

"We were prescribed Ambien because we were traveling the world and I actually looked back and I had one Ambien left and I'm actually happy I only had one because you had a full prescription you know. Who knows," Phelps said.

Here are a few takeaways from Phelps' conversation with Axelrod:

The US Olympic Committee should do more

"We're competing to represent our country, we're competing to do everything we can to try to win a medal or to try to do our country proud by wearing the stars and stripes on international ground. When we come home from it, you know, they're like kind of 'OK check. Who's the next kid coming in? Where's the next person?' And I think it's sad," he said.

Phelps said a majority of Olympic athletes go through a "post-Olympic depression." Among them, US swimmers Missy Franklin and Allison Schmitt. Franklin also spoke to CNN about her own experiences of depression following her four gold medal wins.

"It saddens me that we still don't have anything put in place to help them make that next transition," Phelps told Axelrod, "The USOC in my opinion hasn't done anything to help us transition after an Olympics, and I think that's sad. I think it's unfortunate and it's something that we're working towards now."

In 2016 the USOC launched Pivot, a program designed to help athletes "discover and cultivate their next passions and goals as they transition out of elite competition." The program is specifically intended to assist retiring athletes.

'I swam with a lot of anger'

Learning to discuss his emotions, Phelps said, was an incredibly important step in recovering.

He had to look back on his upbringing, where he first struggled internally. Phelps' parents divorced at a young age and growing up in a single-family household was "challenging."

He did not have a good relationship with his father, who Phelps said was never around. Phelps carried that "abandonment feeling" throughout his early life.

"I did sort of want a white picket fence and the family dinner. I wanted that as a kid growing up. I saw my friends having it," Phelps said.

Swimming was his escape and he learned to take out his frustrations under water.

"There were moments growing up when I was training where I swam with aggression. I swam with a lot of anger. And yeah part of it was probably coming from home and coming from what I was going through when we were in our home life," Phelps said, "I'd let out a lot of profanity under water."

What he's doing now is "bigger than a gold medal"

Today he understands that "it's OK to not be OK," and he's using his voice to help others struggling with mental health issues through the Michael Phelps Foundation.

"Every day somebody is going to have ups and downs and if we can continue to help people get out and talk about things and open up, for me that was something that completely changed my life and I was able to see a much cleaner happier healthier way of living," Phelps told Axelrod.

Phelps knows that by sharing his experience he has the chance to save lives.

"So if I can honestly save a life or save two lives that's all I want. For me that's way bigger than ever winning a gold medal," Phelps said.