Originally Published: 27 APR 23 11:43 ET

By Samantha Murphy Kelly, CNN Business

(CNN) -- Less than a few hours after Snapchat rolled out its My AI chatbot to all users last week, Lyndsi Lee, a mother from East Prairie, Missouri, told her 13-year-old daughter to stay away from the feature.

"It's a temporary solution until I know more about it and can set some healthy boundaries and guidelines," said Lee, who works at a software company. She worries about how My AI presents itself to young users like her daughter on Snapchat.

The feature is powered by the viral AI chatbot tool ChatGPT -- and like ChatGPT, it can offer recommendations, answer questions and converse with users. But Snapchat's version has some key differences: Users can customize the chatbot's name, design a custom Bitmoji avatar for it, and bring it into conversations with friends.

The net effect is that conversing with Snapchat's chatbot may feel less transactional than visiting ChatGPT's website. It also may be less clear you're talking to a computer.

"I don't think I'm prepared to know how to teach my kid how to emotionally separate humans and machines when they essentially look the same from her point of view," Lee said. "I just think there is a really clear line [Snapchat] is crossing."

The new tool is facing backlash not only from parents but also from some Snapchat users who are bombarding the app with bad reviews in the app store and criticisms on social media over privacy concerns, "creepy" exchanges and an inability to remove the feature from their chat feed unless they pay for a premium subscription.

While some may find value in the tool, the mixed reactions hint at the risks companies face in rolling out new generative AI technology to their products, and particularly in products like Snapchat, whose users skew younger.

Snapchat was an early launch partner when OpenAI opened up access to ChatGPT to third-party businesses, with many more expected to follow. Almost overnight, Snapchat has forced some families and lawmakers to reckon with questions that may have seemed theoretical only months ago.

In a letter to the CEOs of Snap and other tech companies last month, weeks after My AI was released to Snap's subscription customers, Democratic Sen. Michael Bennet raised concerns about the interactions the chatbot was having with younger users. In particular, he cited reports that it can provide kids with suggestions for how to lie to their parents.

"These examples would be disturbing for any social media platform, but they are especially troubling for Snapchat, which almost 60 percent of American teenagers use," Bennet wrote. "Although Snap concedes My AI is 'experimental,' it has nevertheless rushed to enroll American kids and adolescents in its social experiment."

In a blog post last week, the company said: "My AI is far from perfect but we've made a lot of progress."

User backlash

In the days since its formal launch, Snapchat users have been vocal about their concerns. One user called his interaction "terrifying" after he said it lied about not knowing where the user was located. After the user lightened the conversation, he said the chatbot accurately revealed he lived in Colorado.

In another TikTok video with more than 1.5 million views, a user named Ariel recorded a song with an intro, chorus and piano chords written by My AI about what it's like to be a chatbot. When she sent the recorded song back, she said the chatbot denied its involvement with the reply: "I'm sorry, but as an AI language model, I don't write songs." Ariel called the exchange "creepy."

Other users shared concerns about how the tool understands, interacts with and collects information from photos. "I snapped a picture ... and it said 'nice shoes' and asked who the people [were] in the photo," a Snapchat user wrote on Facebook.

Snapchat told CNN it continues to improve My AI based on community feedback and is working to establish more guardrails to keep its users safe. The company also said that similar to its other tools, users don't have to interact with My AI if they don't want to.

It's not possible to remove My AI from chat feeds, however, unless a user subscribes to its monthly premium service, Snapchat+. Some teens say they have opted to pay the $3.99 Snapchat+ fee to turn off the tool before promptly canceling the service.

But not all users dislike the feature.

One user wrote on Facebook that she's been asking My AI for homework help. "It gets all of the questions right." Another noted she's leaned on it for comfort and advice. "I love my little pocket, bestie!" she wrote. "You can change the Bitmoji [avatar] for it and surprisingly it offers really great advice to some real life situations. ... I love the support it gives."

An early reckoning over how teens use chatbots

ChatGPT, which is trained on vast troves of data online, has previously come under fire for spreading inaccurate information, responding to users in ways they might find inappropriate and enabling students to cheat. But Snapchat's integration of the tool risks heightening some of these issues, and adding new ones.

Alexandra Hamlet, a clinical psychologist in New York City, said the parents of some of her patients have expressed concern about how their teenager could interact with Snapchat's tool. There's also concern around chatbots giving advice and about mental health because AI tools can reinforce someone's confirmation bias, making it easier for users to seek out interactions that confirm their unhelpful beliefs.

"If a teen is in a negative mood and does not have the awareness desire to feel better, they may seek out a conversation with a chatbot that they know will make them feel worse," she said. "Over time, having interactions like these can erode a teens' sense of worth, despite their knowing that they are really talking to a bot. In an emotional state of mind, it becomes less possible for an individual to consider this type of logic."

For now, the onus is on parents to start meaningful conversations with their teens about best practices for communicating with AI, especially as the tools start to show up in more popular apps and services.

Sinead Bovell, the founder of WAYE, a startup that helps prepare youth for future with advanced technologies, said parents need to make it very clear "chatbots are not your friend."

"They're also not your therapists or a trusted adviser, and anyone interacting with them needs to be very cautious, especially teenagers who may be more susceptible to believing what they say," she said.

"Parents should be talking to their kids now about how they shouldn't share anything personal with a chatbot that they would a friend -- even though from a user design perspective, the chatbot exists in the same corner of Snapchat."

She added that federal regulation that would require companies to abide by specific protocols is also needed to keep up the rapid pace of AI advancement.