By Jordan Valinsky, CNN Business

(CNN) -- Kanye West's threat to walk away from his fashion company's fledgling partnership with the Gap has sent shares sharply lower.

Gap's stock fell nearly 6% Monday following West's comments he made at a public event Sunday. Videos taken at the event show the rapper criticizing Gap and Adidas, another apparel company his Yeezy brand partnered with.

West noted he was not on either company's board. "That has to change today or I walk away," he said.

Adidas declined to comment and Gap didn't immediately respond to CNN Business' request for comment.

Gap, which has been struggling lately, announced a 10-year partnership with Yeezy last month for a new line of clothes that are "modern, elevated basics for men, women and kids at accessible price points" due out next year. It's unclear if he's even allowed to walk away from the deal.

The initial announcement sent Gap's shares soaring nearly 19% on June 26. The stock has lost half of those gains since then.

The Yeezy deal could give Gap a much-needed financial and creative boost. Gap has been struggling: Sales plunged 43% in the quarter that ended May 2, but online sales for May were double what they were a year ago.

Gap also didn't pay rent in April because of the temporary closures. It said cash-saving measures taken during the quarter left it with $1 billion in cash on hand as of May 2, a bit more than it had a year ago.

West has had great success in the fashion space. He became the first non-athlete to launch a full collaboration with Nike for the original Air Yeezy sneakers in 2009, and the brand's sneakers often go for several hundred dollars. His major partnership with Adidas that began in 2013 earned him raves.

-- Oliver Effron contributed to this report.