Analysis by Michael Warren, CNN

(CNN) -- One of the more interesting revelations to come out of last week's nationally-televised hearing with President Donald Trump's former lawyer Michael Cohen (and there were many) was that a top Democrat and Republican lawmaker are actually close friends.

That news came toward the end of the marathon session, when House Oversight Committee chairman Elijah Cummings stood up for Republican Rep. Mark Meadows in the face of accusations of racism. The chairman referred to Meadows as one of his "best friends."

"I know that shocks a lot of people," Cummings said.

Having two best friends from opposing parties is certainly a rarity on Capitol Hill these days. But what makes this relationship particularly interesting is the role of each man: Cummings is leading one of the most aggressive oversight investigations of a sitting president in recent memory, and Meadows is among the President's closest allies and staunchest defenders in Congress.

Their unlikely friendship could prove to be one of the most consequential relationships in Washington over the next two years and could end up being a rare example of bipartisanship and civility. It will also likely be put to the test in the coming months as Cummings digs into any number of aspects of the Trump administration, from the granting of security clearances at the White House to the administration's communications with Russia to lobbying by White House officials for a nuclear power deal with Saudi Arabia.

As Cummings' investigations unfold, there will be more subpoenas, more hearings and more opportunities for the President to come under harsh scrutiny. And that will give Meadows all the more reason to defend Trump and undercut Democratic efforts, as he did during the Cohen hearing, when he brought Lynne Patton, a black, high-ranking political appointee in the Department of Housing and Urban Development.

Patton's appearance was meant to rebut Cohen's claim Trump is a racist. But Rep. Rashida Tlaib, during her question time, argued Meadows had used Patton as a "prop" and called the act racist. Meadows objected and reminded Cummings that he "knew" him well, prompting Cummings to declare his friendship to ease the tension. (Tlaib said she did not mean to call Meadows a racist, and the two members later embraced on the House floor.)

An Unlikely Friendship

Despite their differences, Cummings and Meadows do appear to have a real affinity for each other—even if "best friends" is a bit of an exaggeration. They've worked together on a number of governmental reform bills and have grown to respect each other during six years serving together on the Oversight committee. "There's a bond of trust that exists," said Rep. Gerry Connolly, a Virginia Democrat who also sits on Oversight.

Meadows and Cummings are far from an obvious pair. Cummings, 68, is a native of inner-city Baltimore, while Meadows, 59, grew up in Florida before moving to the foothills of the Appalachian Mountains in western North Carolina. Cummings' district is nearly 60% black, urban and heavily Democratic. Hillary Clinton won more than 75% of votes there in 2016.

Meadows's district is more than 90% white, rural, and deeply Republican— 63% of its voters supported Trump for president. Meadows, first elected in 2012, is chairman of the hyper-conservative, rabble-rousing Freedom Caucus, while Cummings is a House institutionalist with more than two decades on Capitol Hill.

The distinctions on the politics of the moment are even starker. Meadows is more than a typical supporter of Trump's—he remains one of the President's closest confidants in the House. Trump speaks with Meadows on the phone several times a week. Meadows was even considered for the job of White House chief of staff.

In fact, one of the attributes that Meadows cited while he was floated for the post in December was his relationship with Cummings, which he suggested would have been valuable as Cummings' panel scrutinized the Trump administration. Asked about Meadows' comments at the time, Cummings praised his relationship with Meadows.

"I have a good relationship with Meadows. We don't agree on probably 85% of things, of issues, but he is not disagreeable," Cummings told reporters. "So we're able to sit down and work out things we agree on. And he has always maintained a high-level of civility, which I don't see how you can even get anything done unless you at least have that."

Cummings is part of a Democratic House caucus that believes its wins in last year's midterm elections came with the implicit mandate to investigate the President. Just as many Tea Party Republicans felt like they were sent to Washington to oppose President Barack Obama, many freshman Democrats believe their top priority is to resist the policies of Trump. Several of those freshmen, like Tlaib, Katie Hill and Ayanna Pressley, are on the Oversight committee.

Despite that, the Cummings-Meadows relationship has thrived, forged over several whistleblower and postal-reform bills they've worked on together. They've also grown close during their shared time on the Oversight committee, where Cummings had been ranking member until the Democrats won the majority. Meadows was often tapped as the "fill-in" chairman when the top Republican was absent for a hearing.

"There were times when a Democratic witness would stumble or would inadvertently begin making a point he knew the other side wouldn't want them to make, and he'd mumble under his breath to Cummings, 'They're drowning!'" Graham Haile, a former Meadows staffer, told CNN. "And that was always Cummings's cue to jump in and redirect the line, the questioning, which he'd always allow. I think they both got a kick out that."

Meadows, Connolly says, has consistently shown a courtesy to Cummings and the Democratic side that he says was absent with earlier Republican chairs.

"Mark is not confrontational," said Tom Davis, the former Virginia congressman who served as the top Republican on the Oversight committee during much of the George W. Bush administration. That sticks out, Davis told CNN, when confrontation is the coin of the realm on Capitol Hill.

If Meadows continues to take drastic measures to defend Trump, patience among the progressive freshmen on the committee may overpower that kind of bipartisan relationship. And if Trump begins to direct his fire toward Cummings, the man who will be looking under every rock to find wrongdoing within the administration, will Meadows return the favor and defend him? Or will he side with the President?

Mia Love, a former GOP House member and a CNN contributor, says the intensity in both camps—to protect the President on the Republican side and to take him down on the Democratic side—won't be good for what she says is a real friendship between Cummings and Meadows. "The friendship will end up in the crosshairs, and no one wins," she said.