By Ariane de Vogue, CNN Supreme Court Reporter

(CNN) -- The Supreme Court grappled on Thursday with a case concerning an exception to the Fifth Amendment's ban on prosecuting an individual twice for the same offense in a case that could also possibly impact President Donald Trump's pardon power as it applied to the Robert Mueller probe.

Decades ago, the Supreme Court developed an exception to the Fifth Amendment's double jeopardy clause and it is now being asked to rethink precedent.

The so-called "separate sovereigns exception" provides that a person can be tried twice for the same offense if the prosecutions occur in state and federal courts. The rationale is that the states and the federal government are different sovereigns.

Critics contend that in the modern day it leads to harassment of defendants -- especially the poor -- who can't afford to fight on two fronts. They also point to a recent trend they argue has led to an increase of federal prosecutions in areas that had traditionally been left to the states.

In addition, it could also impact the presidential pardon power, leading to a question of what would happen if President were to pardon an individual like his former campaign chairman Paul Manafort for federal offenses. Under the exception, a state could conceivably bring a prosecution for the same crimes. That might not occur if the court were to strike the exception.

In court, the justices worried about the fact that the exception has been on the books for some 170 years.

Justice Elena Kagan stressed that "30" justices have approved it. Justice Brett Kavanaugh noted the instability in the system that can occur when the justices change course. That seemed to be the prevailing sentiment on the court.

At the same time, some justices recognized the complication of the debate and how the exception has been criticized by academics and judges who fear it could, as Justice Ruth Bader Ginsburg said, amount to a "double whammy" for some defendants. Justice Neil Gorsuch pointed to the fact that successive prosecutions are a "problem."

A Justice Department attorney stressed that if the court were to strike the exception it would cause "practical" problems, deter cooperation, and prompt defendants to play the federal government against states.

Gamble prosecution

The case before the justices Thursday was brought by Terance Gamble, who was convicted of second-degree robbery in Alabama in 2008 and 2013. He was subsequently stopped in 2015 and found with a weapon in his car. Federal and state law forbid a convicted felon from possessing a firearm. After convictions in both federal and state courts, Gamble said that his dual convictions prolonged his incarceration by three years.

An appeals court ruled against him citing Supreme Court precedent which, the court said, "has determined that prosecution in federal and state court for the same conduct does not violate the Double Jeopardy Clause because the state and federal governments are separate sovereigns."

Gamble appealed to the Supreme Court, asking it to overrule the separate sovereigns doctrine.

In court papers, Gamble's lawyers argued that "for centuries" federal and state criminal justice systems operated with little to no overlap and that state criminal law was dominant.

In the modern day, they say, that has changed.

Presidential pardon power limits?

Although the case does not touch on the special counsel's investigation, and the presidential pardon power did not come up in oral arguments, some believe that it might have ramifications for Manafort. The President has not ruled out the possibility of a pardon. If the Supreme Court strikes down the exception, a state could in theory prosecute him.

"There's more than nothing to the concern that, if the court overturns the separate-sovereigns doctrine, a state could not then prosecute someone like Paul Manafort for the federal crimes for which he might be pardoned," said CNN legal analyst and University of Texas Law School Professor Steve Vladeck.

"But the criminal jurisdiction of states tends to be so much broader than the federal government that such a move might not close the door to all potential criminal liability in such cases," Vladeck added.

Adam Kurland, a profess of law at Howard University School of Law, doubts the case will impact the Mueller investigation.

"New York law already has a statute that limits some state prosecutions based on the same conduct as a prior federal prosecution," he said in a statement.

The Justice Department has urged the court to uphold the separate sovereigns exception, citing the intent of the framers.

"The framers wrote the Constitution to manifest the sovereign power of the United States and the states, including the power to enforce their own criminal laws," argued Principal Deputy Solicitor General Jeff Wall in court papers.

Wall also argued that under longstanding policy, the government only pursues a federal prosecution when the state case has left "substantial federal interest demonstrably unvindicated."