Brian Lowry

(CNN Money) -- During a Monday night appearance on "The Late Show with Stephen Colbert," Billy Bush again verified the authenticity of the "Access Hollywood" tape featuring President Trump's crude comments about women.

Bush was hosting "Access Hollywood" in 2005 when then-reality-star Donald Trump made remarks in a video that leaked during the 2016 presidential campaign. Bush reiterated Monday what he wrote in a Sunday night New York Times op-ed piece -- that it was in fact Trump's voice in the tape. The issue had resurfaced because of a New York Times report quoting sources who said the president has privately questioned whether the tape is real.

"I would also like to say that's not me on that bus. You don't get to say that," Bush said, adding in response to Trump, "That is your voice. I was there. You were there."

Colbert, who replayed the tape, noted that the issue wouldn't have been raised again if the president hadn't opened that door.

"It's really the dumbest thing he could have done," Colbert said, adding wryly -- "He's a dumb, dumb person."

Bush said the reports of Trump's denial "infuriated" him, not only because of his own involvement, but because of the women who had stepped forward accusing Trump of sexual harassment.

Billy Bush to Trump: Yes, you said that

"You're reopening wounds on them too," Bush said. "Enough's enough."

Colbert asked Bush about the apparent irony of NBC firing him for his role in the situation, but continuing to employ longtime "Today" host Matt Lauer. Lauer was recently fired after reports arose regarding his own alleged sexual misconduct.

Bush said Lauer told him at the time that he had privately lobbied the network to keep Bush, who had just segued to the morning show from "Access Hollywood" -- a show which NBC also distributes.

Bush noted that because Trump hosted the hit show "The Apprentice," everyone at NBC at that time -- during which the taped conversation took place -- had to "kiss the ring a little bit of The Donald."

Bush added that he didn't necessarily take Trump literally, likening his comments to "a crass standup act. ... like an Andrew 'Dice' Clay performance," referring to the once-controversial comedian.

Bush said that it took some time after his departure before he began reading the accounts of the women who had publicly leveled allegations against Trump.

"I said, 'Twenty women don't get together and say, "You know what would be really fun, let's take down a powerful guy together."' No, they don't," Bush said.

Colbert did not ask Bush about his career plans, which have been a subject of renewed speculation in light of the op-ed and his appearance.