By Tal Kopan, CNN

(CNN) -- Judge Steven Morley has overseen the immigration case of Reynaldo Castro-Tum for years. But last month when Castro-Tum was officially ordered deported, it wasn't Morley at the bench.

Instead, the Justice Department sent an assistant chief immigration judge from Washington to replace Morley for exactly one hearing: the one that ended Castro-Tum's bid to stay in the US.

The unusual use of a chief immigration judge from headquarters has raised concerns from retired immigration judges, lawyers and the union for active immigration judges. They say the move seems to jeopardize the right to a fair process in immigration courts.

It also highlights the unique structure of the immigration courts, which are entirely run by the Justice Department, and the ways that Attorney General Jeff Sessions -- who serves as a one-man Supreme Court in these cases -- has sought to test the limits of his authority over them.

The saga of Castro-Tum starts in 2014, when he crossed the border illegally as a 17-year-old. The Guatemalan teen was apprehended by the Border Patrol, which referred him to custody with the Health and Human Services Department as an unaccompanied minor. He was released to his brother-in-law a few months later and registered his brother-in-law's address with the government. Multiple notices of court hearings were sent to that address, the government said.

But after the fifth time Castro-Tum failed to appear in court, immigration Judge Morley closed the case until the government provided him with evidence that Castro-Tum had ever lived at the address they were sending the notices to. The Board of Immigration Appeals sent the case back to Morley to reconsider with instructions to proceed even if Castro-Tum failed to show again. His current whereabouts are unknown.

Earlier this year, Sessions referred the case to himself and ruled that immigration judges across the board could no longer close immigration cases as they saw fit. The attorney general said immigration judges lack the authority to make such "administrative closures" of cases.

Sessions gave Morley 14 days to issue a new hearing notice to Castro-Tum. The Philadelphia-based immigration attorney Matthew Archambeault, who had begun following the case, appeared in court and volunteered to represent Castro-Tum, as well as to track him down. He asked the judge to postpone the case a bit longer to give him time to do that, which Morley granted.

Archambeault told CNN that the address given for Castro-Tum was a trailer park in Pennsylvania known to be used by a transient immigrant community. A group familiar with the area was not able to find Castro-Tum. Archambeault said there are some indications he may have returned to his home country shortly after he was released from custody years ago, though no one has been able to confirm that.

But when Castro-Tum's hearing came up July 26, Morley had been replaced.

Assistant Chief Immigration Judge Deepali Nadkarni, reading from a computer screen, ordered the young man deported, Archambeault told CNN.

Morley was working and heard all his other cases as scheduled, Archambeault said. Castro-Tum's was the only case that was reassigned.

"She was very polite and professional, she read the decision from the computer screen, she referenced the brief I had written and the arguments I made, then she ordered him removed, and then she allowed me an opportunity to make a statement, I made a statement, and that was kind of it for the hearing," Archambeault said. "It was a really weird and odd thing, to be honest."

Nedkarni did not explain why she was holding the hearing, he added.

The Justice Department did not respond to multiple requests to explain why the judge was sent to Philadelphia to decide this single case.

Archambeault said the move was "disturbing" but indicated he doesn't think Morley would have ruled any differently. According to data maintained by Syracuse University's TRAC project, Morley has an asylum denial rate of 46%, just under the national average of 53%. Asylum denial rates are one measure of frequency that judges side with immigrants.

"We don't have a lot of due process in immigration court. We're kind of used to it," Archambeault said. "But we don't want it to be a completely rigged process. When we get a bad decision, we want it to be the judge's bad decision. We don't want it to be decided from above. ... This was clear to send a message to immigration judges that they need to get in line, and that's not the way courts should be run."

Those sentiments were backed up by statements from retired immigration judges and the union that represents immigration judges.

"Our organization is extremely concerned and dismayed at the actions of the executive office for immigration review in this unprecedented incursion on judicial independence," said immigration Judge Dana Leigh Marks, speaking in her capacity as the president emeritus of the National Association of Immigration Judges.

Marks said that generally judges should be taken off cases only if a motion to recuse is filed, but nothing of the sort was done in this case. She said the organization intends to pursue legal action.

"It appears, unfortunately, to be a method by which to influence the outcome of this case and others," Marks said. "And it's just completely inappropriate. ... It's our worst fears come true."

In a letter from 15 retired judges and former Board of Immigration Appeals members, the group said this instance of interference from the Justice Department was "unacceptable."

"Immigration Judges must be afforded the independence to conduct fair, impartial hearings," the group wrote. "For this reason, some important due process safeguards are required in deportation proceedings, and errors should be corrected through the appeals process, not through interference by managers."