Jordan Valinsky

(CNN Money) -- Papa John's founder John Schnatter said it was a "mistake" for him to resign.

In letter to the pizza chain's board obtained by CNNMoney, Schnatter accused the board of not "doing any investigation" and said its decision to remove him as chairman was based on "rumor and innuendo."

Last week, the Papa John's founder resigned after he admitted to using the N-word on a conference call with marketing agency Laundry Service.

In his letter, he said Laundry Service wanted to hire rapper Kanye West to represent Papa John's in advertising materials, but Schnatter wasn't on board with the idea because West uses the N-word in his lyrics.

Schnatter claimed the agency asked if he was racist, and he used the N-word in describing how KFC's Colonel Sanders used to talk. Schnatter said he wouldn't ever use that word -- although he just had.

He also alleged that Laundry Service tried to extort Papa John's by threatening to go public with Schnatter's comments. Laundry Service didn't immediately return CNNMoney's request for comment.

Schnatter has hired Patricia Glaser, an attorney to represent him to the board. She sent a separate letter saying any attempt to remove Schnatter from the board without a "proper vote of the shareholders will be null and void." He still owns roughly 30% of the company's shares.

Papa John's has been distancing itself from the cofounder since the scandal erupted last week. He was kicked out of his office and the brand has scrapped Schnatter's image from its pizza boxes and pulled ads that featured him.

Read the full letter below:

Dear Fellow Board Member

I am writing because I believe it is important that you hear directly from me the facts and circumstances surrounding the events that were initially reported and mischaracterized in the July 11 Forbes story, "Papa John's Founder Used the 'N' word on Conference Call" and ultimately was carried in media across the country.

On May 14, Steve Richie, Mike Nettles, I and others in the company met with executives and staff of The Laundry Service, who shared their creative and strategy, at their offices in New York. As you know, we had been testing with significant success, my returning to the company's advertising. On May 22, at their strong suggestion, I participated in what The Laundry Service called "diversity media training." The idea was to prepare me for questions I might get as a result of my reappearance at NHRA on Saturday, May 26 in Chicago. (The Laundry Service, for those of you who don't know, is an advertising and marketing agency which is part of the Wasserman Media Group.) During and after that meeting, The Laundry Service leadership strongly urged that our company retain Kayne West as my co-spokesman in the television spots and other promotions. I told them that would not work because he uses the "N" word in his lyrics.

During this diversity media training, which covered a wide number of topics, I was asked whether I was racist. I, of course said no -- which is a truthful statement as those of you who know me well will attest and of course, if you felt otherwise you would not be sitting on the Papa John Board. I was asked if I was not racist, then why did I say what I did about the NFL situation? I said if you look at what I said, it was in no way racist. (The fact is, we completely mishandled the NFL situation from a public relations standpoint -both the Board of Directors and company leadership.) I then said something on the order of, Colonel Sanders used the word "N," (I actually used the word,) that I would never use that word and Papa John's doesn't use that word. Earlier, I gave an example of a scarring experience I read about in Texas when I was growing up which further cemented my existing abhorrence of racism. The thought of this situation to this day sickens me. Let me be very clear: I never used the "N" word in that meeting as a racial epithet, nor would I ever.

I have talked to a Papa John's employee who was in that room with me who confirmed my recollection of these events.

The next day, May 23, the company made the decision -- not me -- to fire the Laundry Service, with their last day being July 2. We owed them approximately $1.3 million. Of course, we said we would pay them what was owed, but they said they wanted $6 million because they claimed some of their people had been offended by what I had said. Moreover, one of their attorneys said they would conduct a smear campaign against the company and me unless we paid them what he was asking for. Unfortunately, the company gave in to this extortion attempt and offered them $2.5 million or roughly $1.2 million more than they were owed.

On July 10, we got a call from the Forbes reporter who wrote the above-referenced story. The reporter gave me 15 minutes to give him our comments and said he then was publishing the story. It published the next day. Please be assured, I am going to get the facts of this situation out, but we want to make sure we do it correctly.

The Board asked me to step down as chairman without apparently doing any investigation. I agreed, though today I believe it was a mistake to do so. I have checked with corporate governance experts who tell me that this was not a proper action by the Board. At the last meeting, a few of you raised the issue of whether I should step down as a director. Once again, those individuals were acting on rumor and innuendo, without any investigation—let alone a third-party investigation of the facts. And once again, the corporate governance experts with whom I consulted said this is not the proper action of either a director or the board.

I am confident that an examination of the facts will bear out what I have written in this letter and show that once again our company has demonstrated that it does not know how to handle a crisis based on misinformation. I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted.

John