By Chris Isidore and Aaron Smith, CNN Business

(CNN) -- Bernard Madoff, whose name became synonymous with financial fraud, died while serving a 150-year sentence in Federal Prison. He was 82 years old.

His death Wednesday at the Federal Medical Center in the prison in Butner, North Carolina, was confirmed by the US Bureau of Prisons. A cause of death was not released.

In February 2020, he petitioned the courts for an early release from prison, stating that he had terminal kidney failure and a life expectancy of less than 18 months. But the US Attorney's office for the southern district of New York said Madoff's crime was "unprecedented in scope and magnitude" and is "sufficient reason" to deny Madoff's request.

Madoff was the mastermind behind a $20 billion Ponzi scheme -- the largest financial fraud in history.

Judge Denny Chin, who originally had imposed the sentence of 150 years, denied his request for release, calling his crime "one of the most egregious financial crimes of our time," and one that continued to take "a staggering human toll."

"When I sentenced Mr. Madoff in 2009, it was fully my intent that he live out the rest of his life in prison," the Chin wrote in his order last June. "The symbolism of a 150-year sentence was important: the public trust had been eroded by Mr. Madoff's ability to manipulate the system for so many years, he deserved to be punished according to his moral culpability."

One of the few kind words about Madoff Wednesday came from Brandon Sample, one of his attorneys.

"Bernie, up until his death, lived with guilt and remorse for his crimes," Sample said. "Although the crimes Bernie was convicted of have come to define who he was -- he was also a father and a husband. He was soft spoken and an intellectual. Bernie was by no means perfect. But no man is."

He had a legendary career on Wall Street, famously delivering astronomical returns for his investors, which included director Steven Spielberg, actors Kevin Bacon and Kyra Sedgwick and New York Mets owner Fred Wilpon.

He served as chairman of the Nasdaq for several years in the 1990s and amassed beach houses, boats and a Manhattan penthouse. But Madoff was arrested in 2008 and pleaded guilty to eleven felony charges in 2009. He had been using money from new investors to pay back earlier investors. He supposedly had a total of $65 billion under management, but two thirds of that money was a figment of Bernie Madoff's imagination. The rest was his investors' principle.

Madoff founded Bernard L. Madoff Investment Securities in 1960, but no one has been able to prove when Madoff began stealing from investors.

He told CNNMoney in a 2013 interview that it all started in 1987, but he later said the scheme began in 1992. Madoff's former account manager, Frank DiPascali, Jr., said in court testimony that financial misdeeds had been going on "for as long as I remember." He started working at the firm in 1975.

Irving Picard, the court-appointed trustee charged with recovering assets stolen by Madoff, together with the Department of Justice, had recovered $14.4 billion as of March of this year, distributing the vast majority to Madoff victims. In addition, the Securities Investor Protection Corporation has provided $600 million in insurance to victims.

Madoff was born April 29, 1938 in New York City's borough of Queens, where he met his wife Ruth in high school. They had two sons, both of whom worked for their father's firm. Mark killed himself in 2010.

Madoff's brother Peter also served a 10-year prison sentence for his involvement in the scheme. He was sentenced in 2012.