By Veronica Stracqualursi, CNN

(CNN) -- The Tennessee Supreme Court on Wednesday overturned a lower court ruling that would have allowed all eligible voters to vote via absentee ballots due to the coronavirus pandemic.

Under the ruling, those with underlying medical conditions who have determined it's "impossible or unreasonable" to vote in-person due to Covid-19, and their caretakers, are eligible to vote via absentee ballot in the fall election. State law already allows those who are hospitalized, ill or physically disabled and their caretakers to vote absentee. Fear of contracting or spreading the virus will not apply as an excuse to vote absentee in Tennessee.

The decision comes as several states plan on holding a primarily vote-by-mail election in November, and other states push to expand mail-in voting amid the Covid-19 pandemic, which has infected over 4.8 million Americans and led to over 158,000 deaths.

Tennessee is one of several states that require an excuse to cast an absentee ballot -- such as being 60 years old or over, jury duty, observing a religious holiday, out of town or enrolled as a full-time student at a college outside the county.

In April, as it became clear the pandemic could affect voting, Tennessee released an election contingency plan stating that "a person who is quarantined because of a potential exposure or who has tested positive to Covid-19 should vote absentee by-mail as a person who is ill."

Two groups of Tennesseans, who said their health would be at risk if forced to vote in-person, sued the state in May, seeking to expand vote-by-mail options to eligible voters during the pandemic. They argued that the state's interpretation of the law of who is eligible for absentee voting burdened their right to vote.

The state's justification for refusing to expand voting by mail during the pandemic was voter fraud and that it's not "fiscally nor logistically feasible."

Last month, a county court judge issued a temporary injunction, ordering the state to provide absentee ballots to voters wishing to vote by mail to avoid spreading or contracting Covid-19, so long as the pandemic is ongoing. The judge also found that it's "feasible" for the state to provide voters with a vote by mail option and that "increased voter fraud is not a material concern."

The lower court's order paved the way for any eligible voter to vote absentee to avoid spreading or contracting Covid-19 in Tennessee's primary. But because its ruling came the night before Thursday's primary, the Tennessee Supreme Court said all absentee ballots submitted on time under the lower court's injunction "shall be duly counted."

During oral agreements before the Tennessee Supreme Court, the state said individuals with a "special vulnerability" to Covid-19, or their caretakers, already meet the requirements under law to be allowed to vote by absentee ballot.

Secretary of State Tre Hargett, a Republican, said in a statement that he appreciates the Tennessee Supreme Court "agreeing with our analysis of Tennessee election law."

In its Wednesday ruling, the Tennessee high court said while it is "understandable" that some of those without underlying medical conditions would want to vote by mail, the risk to these voters is "significantly less" due to health measures the state is taking to ensure safe in-person voting, like mandatory masks for poll workers and plexiglass shields at check-ins.

As part of its evaluation, the court said, however, it agreed with the lower court that an increase in absentee voting would not increase the risk of fraud.

The court said that because it views the risks to voters without underlying conditions as moderate, rather than severe, it would not interfere with the decision of the Coordinator of Elections, saying those decisions would be "judged by history and by the citizens of Tennessee."

"We find that the State's interests in the efficacy and integrity of the election process are sufficient to justify the moderate burden placed on the right to vote of those plaintiffs and persons who neither have special vulnerability to COVID-19 nor are caretakers for persons with special vulnerability to COVID-19," Justice Cornelia A. Clark wrote in the majority's opinion.

In an opinion that dissented in part, Justice Sharon Lee wrote that the court's ruling doesn't go far enough.

"In the midst of this pandemic and while Tennessee remains under a state of emergency, qualified Tennessee voters with no underlying medical or health conditions should not be left with the impossible choice of voting in person and risking getting COVID-19 or forfeiting their constitutionally protected right to vote. Tennessee voters deserve better," Lee wrote.

The ACLU of Tennessee said Wednesday that it's "troubled" that the court did not expand mail-in voting "so that all Tennessee voters could decide how to best cast their ballot to protect their health and safety."

"The COVID-19 pandemic is hitting Black communities and urban areas the hardest," Hedy Weinberg, ACLU of Tennessee executive director, said in a statement. "Failing to expand mail-in voting in the midst of the pandemic is nothing short of voter suppression."

One of the plaintiffs, Ben Lay, said he and his wife are "pleased" the court affirmed that they and others with underlying health conditions can vote by mail. Lay said he's a cancer survivor who suffers from a compromised immune system and his wife is immunosuppressed.

"However, we believe that no American -- underlying health conditions or not -- should ever be faced with that most impossible choice on Election Day: Do I stay home to protect my health and the health of my loved ones, or do I risk my life to exercise my constitutional right to vote?" Lay said in a statement.