By Curt Devine, Drew Griffin and Jamiel Lynch, CNN

(CNN) -- A lawsuit filed in a Georgia federal court by five state voters asks a judge to strip sitting Secretary of State and gubernatorial candidate Brian Kemp of his powers over the midterm election -- including any potential runoffs.

A motion for a temporary restraining order asks a judge to order Georgia's governor to appoint a replacement who is not affiliated with Kemp or his campaign. It also asks that the Secretary of State's Office remove from its website any reference to allegations against Georgia's Democratic Party.

No court action was expected Tuesday night, according to two of the plaintiff's lawyers.

Kemp has been under fire over allegations of conflict of interest for overseeing the election in which he is a candidate -- and for a last-minute decision over the weekend to level claims that Georgia's Democratic Party had been involved in an attempt to hack state voter data.

The charges originated with a private citizen's discovery of what he believed was a flaw in the system and sought to bring to the attention of authorities, and Georgia Democrats called the episode an abuse of Kemp's power.

Kemp's decision to directly accuse the opposing party of wrongdoing while running for the state's highest office further inflamed deep-seated worries over voting rights in Georgia at the height of a historic campaign by his Democratic opponent, Stacey Abrams, to become the country's black female governor.

The plaintiffs in Tuesday's lawsuit are being represented by Protect Democracy, former US Attorney for Middle District Michael J. Moore and former Department of Justice voting rights section attorney Bryan L. Sells.

"Kemp's conflict of interest was apparent when he got the nomination, but in the last few days he's gone from a potential conflict to actual misuse of his office," Sells told CNN.

One of the plaintiffs, Katharine Wilkinson, told CNN that she decided to join the suit because she believes that what Kemp is doing is illegal.

"You shouldn't be a judge in your own case. You shouldn't be a referee in a game that you're playing. ... There's really a need for a neutral person to be overseeing this election," she said.

Wilkinson said she joined the lawsuit after being contacted by Protect Democracy, a nonpartisan watchdog group founded by a former Obama administration lawyer. Wilkinson is a Georgia resident and voted early in the election.

"For me, it really is just a pretty simple question of fairness," she said. "We should be able to trust our secretaries of state to help ensure that."

Candice Broce, a spokesperson for the Georgia Secretary of State's Office, told CNN: "We have notified Secretary Kemp of this lawsuit. This 12th-hour stunt will not distract us from fulfilling our responsibilities and working with county officials to ensure a secure, accessible and fair election for all eligible Georgians."