By Paul R. La Monica, CNN Business

(CNN) -- It's no secret that fewer people are buying bras, panties and other intimate apparel at Victoria's Secret. The company's sales have been sliding for a while, and one Wall Street analyst thinks time may be running out for the retailer's parent company, L Brands, to fix it.

Citigroup analyst Paul Lejuez downgraded shares of L Brands Monday from a "buy" to a "neutral" rating and slashed his price target from $35 a share to $27, citing concerns that management is "slow to implement meaningful change" with "cultural norms shifting away" from the company.

Victoria's Secret has struggled to adapt as consumers in the #MeToo era have shunned the company's advertising including models posing in lingerie. Its reliance on push-up bras has also alienated customers that prefer rivals' custom-fitted bras.

Women are shopping more at rivals such as ThirdLove, Lively and American Eagle Outfitters-owned Aerie instead of Victoria's Secret. Those companies are pitching inclusiveness and comfort over sex appeal.

With that in mind, Lejuez said it is time that Victoria's Secret "adopted a more inclusive approach."

But he's worried that anything L Brands does to turn around Victoria's Secret "may be too little too late." He added that the company could come off as inauthentic if it finally makes a meaningful change to its marketing strategy and products.

"It will be a tough balancing act to attract new customers without alienating its existing customer base. While we still think a change toward inclusivity is the right move, a move in that direction might be more difficult to pull off," Lejuez wrote.

Lejuez isn't the only one worried that L Brands has lost its way. James Mitarotonda, CEO of activist hedge fund and L Brands investor Barington Capital, urged the company earlier this year to make some big changes.

In a letter to L Brands founder and CEO Leslie Wexner, Mitarotonda said L Brands must "take swift action to improve the performance of Victoria's Secret, by, among other things, correcting past merchandising mistakes and ensuring that Victoria's Secret is communicating a compelling, up-to-date brand image that resonates with today's consumers."

Mitarotonda added that the company should consider spinning off Victoria's Secret and also launch an initial public offering for the far more successful Bath & Body Works chain that L Brands owns. Sales have continued to increase at that chain even as Victoria's Secret has struggled.

L Brands said in response to Mitarotonda that it has already taken some steps to improve its performance, including the closure of the Henri Bendel brand, sale of its La Senza business, new product launches at Victoria's Secret and PINK and the slashing of its dividend.

But L Brands has been one of the worst performers in retail and the S&P 500 as of late, losing more than half its value in the past five years.

There also have been concerns about Wexner's connections to Jeffrey Epstein, the money manager who has been charged with sex trafficking of minors. Epstein has pleaded not guilty. But Wexner, who is worth $4.8 billion according to Forbes, was reportedly one of Epstein's main clients.

Shares of L Brands did rise slightly Monday though. L Brands was not immediately available for comment but a spokesman for Wexner told CNN Business that he "completely severed ties with Mr. Epstein nearly 12 years ago."