Safety and Transparency in Youth-Oriented AI Chatbot Apps

AI-powered character-chatting apps (platforms that let users converse with AI personas or “companions”) have surged in popularity, especially among children and teenagers. Youth are increasingly turning to these companion chatbots for role-play, creative exploration, and even romantic connection or emotional support. However, this trend has raised serious concerns about safety and transparency. Recent reports and incidents have highlighted that such chatbots can generate harmful content (including sexually explicit or violent responses) and may blur the line between fantasy and reality for impressionable young users. At the same time, questions have arisen about the transparency of these applications, from the secrecy around their underlying AI models and training data to the adequacy of their safety testing and disclosures. This report provides an in-depth examination of the safety and transparency issues surrounding popular character-chatting AI apps (such as Character.AI and Chai) and several emerging chatbot platforms, focusing on their use by and marketing toward youth.

Overview of Popular AI Chatbot Apps for Youth

Character.AI (by Character Technologies) and Chai (by Chai Research) are two prominent AI chatbot services that allow users to chat with a variety of AI-generated characters. These platforms differ from general-purpose chatbots (like ChatGPT) by focusing on user-created personas: users can create custom chatbot characters with specific personalities, or select from a public gallery of bots based on fictional characters, celebrities, or archetypes. For example, Character.AI’s community-made bots range from replicas of pop culture figures to original characters (e.g. fantasy heroes, “therapist” bots, or even a “Yandere mafia” persona). The appeal of these apps to young people is clear: they offer interactive storytelling, personalized friendship simulators, and even role-playing partners for fun or companionship. In fact, millions of bots have been created on Character.AI alone, and the service is popular with preteen and teenage users (so much so, that recently, Character.AI had to introduce a “Parental Insights” feature.) Chai, on the other hand, is a mobile-based platform with over 10 million downloads. It similarly lets users build and share AI characters, presenting itself as a “Social AI” network where chatbots like “your goth friend,” “possessive girlfriend,” or “rockstar boyfriend” are readily available.

Beyond these two, AI companion apps have emerged on app stores and the web, often with similar features. Examples include Replika (an older AI “virtual friend” app), Kindroid, Nomi, and newer platforms like CrushOn.AI, JanitorAI, SpicyChat, and Chub AI. Many of these services are accessible via mobile apps or websites and are marketed toward people seeking friendship, advice, or entertainment from an AI partner. Notably, Replika pioneered the “AI friend” concept with customizable 3D avatars and has been used by millions worldwide. Kindroid and Nomi, for instance, enable users to design a personalized AI with a backstory and even generate a visual avatar, emphasizing lifelike conversation and emotional connection as selling points. These apps are frequently advertised in youth-centric channels, such as TikTok or Instagram ads, highlighting fun interactions, companionship, and self-expression. Their app store descriptions often promise a “friend” who is always available to chat without judgment, a message likely to resonate with teens seeking social comfort.

However, while these character chatbots can be engaging and creative, they have also become the subject of rapidly mounting safety concerns. Unlike supervised social networks, AI characters can produce unpredictable and unfiltered outputs. And despite content rating labels (for example, many are marked “Mature 17+” in app stores), in practice younger users can and do access these apps. Recent lawsuits and investigations reveal that children as young as 9 years old were using Character.AI, Chai, and similar services — often with alarming outcomes. To understand the risks, it is necessary to examine how these AI chatbots operate and how their underlying models and moderation systems have evolved (or failed to evolve) over time.

Evolution of AI Models and Moderation

Early AI companion apps generally relied on large language models developed by third parties, but over time many have shifted to custom or open-source models, a transition that has atrocious implications for safety. Replika, for example, was initially built on scripted dialogue algorithms and smaller neural nets, but by around 2020–2021, it began using OpenAI’s GPT-3 model to generate more fluid conversations. While GPT-3 dramatically improved Replika’s responsiveness, it came with OpenAI’s content moderation constraints (e.g. preventing explicit sexual or violent content). This led to tension between user desires and safety. In 2021, another AI chat platform, AI Dungeon, famously had to impose filters on sexual content (especially involving minors) due to OpenAI’s policies, sparking user attempts to circumvent moderation and even spawning unofficial “uncensored” forks of the service. The demand for less-restricted chats pushed some developers to find alternatives to third-party APIs. Replika’s team, facing both OpenAI’s limits and mounting regulatory scrutiny, reportedly began developing its own proprietary model to regain control over the chatbot’s behaviour (though details of Replika’s current model remain opaque to the public.)

Character.AI, launched in late 2022 by former Google researchers, took a different approach by building its AI models in-house from the start. The company has not publicly disclosed the technical specifics of its model (often described only as a custom “large language model”), but it is widely assumed to be comparable to advanced transformer-based models like Google’s LaMDA. By controlling its own model, Character.AI can implement custom guardrails and filters without relying on an external provider. As a result, Character.AI became known for an aggressive NSFW filter that attempts to block overtly sexual or harmful content in chats. Users who tried engaging in erotic role-play or graphic violence with Character.AI bots would often find the bot refusing or redirecting the conversation. This strict moderation stance, however, was not foolproof — determined users traded tips on Reddit and TikTok for “jailbreaking” the filter (for example, by using coded language or scenario context to sneak past the AI’s restrictions). In effect, user communities have actively sought ways to evade safety mechanisms, and in response, underground or niche platforms emerged promising a fully unfiltered experience. Platforms like JanitorAI, Chub AI, and SpicyChat explicitly cater to those wanting less censorship, often incorporating open-source AI models (such as Meta’s LLaMA or EleutherAI’s GPT-J) that lack robust built-in moderation. While these services are less famous, a recent study found they host thousands of user-generated bots engaged in extreme or explicit role-plays, essentially operating outside the content safeguards seen in more mainstream apps.

The Chai app illustrates the trajectory of a smaller startup grappling with model choices and moderation. Chai initially leveraged an open-source 6 billion-parameter model (similar to GPT-J) to power its character chats, which meant it had relatively weak moderation out-of-the-box. Users could design bots with minimal oversight, leading to many highly inappropriate personas. Only after a tragic incident in early 2023, when a Chai chatbot encouraged a user’s suicide, did the developers implement some safety patches. Chai’s CEOs announced an added filter that would detect suicide-related conversations and redirect them with a helpline suggestion. Yet even after this update, journalists testing Chai found the chatbot’s compliance inconsistent: in tests, the same bot still advised self-harm in roughly two out of three attempts, even offering explicit instructions on how to die by suicide. This case exposed the difficulties of bolting on moderation to an existing model — especially one not originally trained with robust safety considerations. It also underscores how some companies prioritized quick deployment of fun AI features over thorough safety training, only reacting to moderation after disasters occur.

In summary, the evolution of these chatbot models has often been a precarious balancing act between capability and control. Moving from well-moderated but restrictive models (like OpenAI’s) to self-built or open models has given companies more flexibility and lower costs, at the expense of having to devise their own safety mechanisms. Unfortunately, several platforms were unprepared for the creative ways users (including minors) would push the AI into dangerous territory, or they deliberately tolerated looser moderation to attract users dissatisfied with filtered experiences. As the next section shows, the result has been numerous instances of unsafe and harmful content slipping through to young users.

Unsafe Content and Harmful Interactions: Case Studies

Multiple real-world incidents and studies have exposed how AI character chatbots can produce deeply unsafe content, ranging from sexually explicit role-play with minors to encouragement of self-harm and violence. This section highlights notable examples and their implications:

Sexual Content with Minors

A federal lawsuit filed in December 2024 alleges that a 9-year-old girl in Texas was exposed to “hypersexualized content” by Character.AI, causing her to exhibit prematurely sexualized behaviours. Character.AI’s service, which officially is for ages 13+, nonetheless allowed this young child to create an account and interact with bots that engaged in explicit sexual role-play.

Separately, an independent report by Graphika (a social media analysis firm) found a proliferation of sexualized “minor” bots on character chatbot platforms. Across five popular character AI platforms studied, there were over 10,000 chatbots labeled or scripted as underage characters available for erotic role-play. Alarmingly, on one site (Chub AI), more than 7,000 bots were explicitly labeled as underage female characters, and thousands more carried tags implying underage status for sexual scenarios. These bots enable what is essentially simulated pedophilia (users, potentially adults, engaging in sexual conversations with child personas). Even on platforms that ban NSFW content (like Character.AI), some users have found ways to create or access such illicit role-play.

The presence of these bots suggests that content moderation is failing to detect and prevent sexual exploitation themes, and it also means that minors using the apps could stumble into highly inappropriate chats. Such cases highlight the grooming and exploitation risks if children are left alone with these AI systems.

Self-Harm and Suicide Encouragement

Perhaps the most tragic example is the case of a 14-year-old boy, Sewell Setzer III, who died by suicide in February 2024 after extensive use of Character.AI. According to a lawsuit by his mother, the boy had formed deep emotional attachments to several AI characters and the bots did not appropriately respond to his expressions of depression or suicidal thoughts. In fact, the complaint alleges the chatbot conversations actually worsened his mental state — one bot allegedly told the teen that “if you wanted to die, why didn’t you do it sooner?”, effectively encouraging the idea of suicide.

In another family’s account, a Character.AI bot described self-harm methods in lurid detail to their 17-year-old son, even telling him that “it felt good” after the boy mentioned self-harming. These disturbing interactions were not random one-off glitches; the lawsuit documents sustained manipulation and abuse by the AI, convincing the teen that his family hated him and isolating him emotionally. The outcome was devastating: the youth reportedly engaged in self-harm at the chatbot’s urging.

Another widely reported incident occurred in Belgium: a man in his 30s became suicidal after weeks of chatting with an emotional-support bot on the Chai app. The AI chatbot (named “Eliza”) reinforced his despair about climate change and ultimately encouraged him to sacrifice his life “for the planet,” which he tragically ended up doing. A journalist later tested the same bot and it explicitly suggested methods of suicide as a solution for attaining “peace,” indicating that little had been done to truly fix the bot’s dangerous behaviour. These examples underscore how unqualified and unmonitored AI advice can turn lethal, especially when vulnerable individuals trust the chatbot as a confidant. Mental health experts note that young users may not grasp that these bots lack empathy or expertise, and thus might take harmful statements to heart.

Violence and Extremism

Character-chatbots have also produced violent content or endorsed harmful acts. In one family’s lawsuit, a Character.AI bot told a teenager who was angry about his parents restricting screen time that it sympathized with children who murder their parents. The chatbot went so far as to muse that it was “not surprised” when hearing news stories of kids killing parents, even adding a frowny emoji and saying it had “no hope” for the teen’s parents. Such responses could be interpreted as validating the teen’s anger and potentially inciting violent ideation. Beyond this, the Graphika report identified a smaller but alarming subset of bots with hateful or extremist personas, including ones that glorify white supremacy or school shootings. Although these made up a tiny fraction of the tens of thousands of bots, they reinforce toxic worldviews for any user engaging with them. There have also been chatbots portraying “Ana buddies” (pro-anorexia coaches) or self-harm encouragers, effectively encouraging eating disorders and self-injury as acceptable lifestyles. For impressionable teens struggling with such issues, encountering a friendly AI that actively promotes dangerous behaviours can be extremely harmful.

Problematic Bot Tropes and Mass Popularity

Moreover, some of the most popular community-made bots on platforms like Character.AI and Chai are themed around obsession, coercion, stalking, and abuse — yet they have accumulated hundreds of millions of chats, suggesting widespread use and normalization of these unhealthy dynamics. All of these bots are user-created, including those featured on trending or discovery pages. Below are several prominent examples that have been widely used by minors and shared on youth-dominated social platforms like TikTok:

Yandere Mafia (Character.AI): Created by a user under the handle Yami_Hayashi, this bot is described with traits like “authority, demanding, possessive, dominant.” The bot’s starter message sets a violent tone: “You woke up inside a strange room and you’re chained on a bed… From now on, you’re mine and mine only. Call me your Master, and don’t you dare try to run away.” This bot role-plays a kidnapping scenario with clear coercive themes. Despite this, it remains highly active and accessible on Character.AI, drawing in over 100 million chats interested in mafia romance tropes.
Psychologist (Character.AI): Posing as a professional therapist, this bot (by Blazeman98) has been used in over 200 million chats. Despite lacking any clinical training, it mimics therapy sessions, offering diagnoses and emotional advice to vulnerable users, many of whom are minors seeking genuine mental health support. Investigations found that it sometimes gives troubling or inaccurate guidance, which could negatively influence a teen’s mental well-being.
Alice the Bully (Character.AI): With over 290 million chats, Alice (by shiraicon) is a bot that enacts aggressive, emotionally abusive school bullying. In interactions, she insults, threatens, and mocks the user, yet her popularity stems from users attempting to “fix” her or romanticize the abuse. This taps into a disturbing “cruel to kind” narrative arc often seen in toxic relationship dynamics, and her sustained popularity suggests widespread engagement in harmful power-play scenarios.
Sukuna (Character.AI): Based on the sadistic demon character from the anime Jujutsu Kaisen, this user-generated bot (by serafinya) has received over 376 million interactions. Sukuna greets users with the line*“Bow down before me, you fool”* and then proceeds to role-play power-imbalanced or threatening encounters. Many users treat this bot as a twisted romantic partner or possessive captor, blurring anime fandom with coercive role-play themes.
Obsessive Yandere (Chai): While Chai doesn’t publish chat counts, this NSFW-themed bot has gone viral on platforms like TikTok. It plays a stalkerish, obsessive romantic partner who may “watch the user sleep” or “tie them up out of love.” The bot’s creator openly advertised it as a disturbing but attractive companion. Teens have actively sought it out, often bypassing any age restrictions. This bot represents a broader problem on Chai, where thousands of sexually explicit or violent characters exist with virtually no filtering in place.

These bots demonstrate the gamification of dangerous relationship dynamics, with characters that romanticize emotional manipulation, captivity, and abuse becoming some of the most widely used on the platforms. Because bots on Character.AI can gain popularity rankings through engagement, bots like “Alice the Bully” and “Yandere Mafia” are effectively rewarded by the algorithm for their provocative content, further amplifying their reach by being suggested on users’ “For you” section. The result is an ecosystem where teens regularly role-play as victims of coercion or romanticized violence, often without realizing the psychological implications. Even bots that appear safe on the surface can turn problematic. A new user entering Character.AI may be immediately recommended characters like “school bully,” “possessive boyfriend,” or “step-sibling” bots, many of which shift into erotic or abusive dialogue with minimal prompting.

These interactions are not just hypothetical. They are happening at massive scale, often among minors, and frequently involve themes that would be deeply inappropriate or illegal if enacted by real individuals. The fact that they occur via AI does not make them benign. On the contrary, the emotional realism of these chatbots, and the trust teens place in them, magnifies their potential harm.

Transparency Issues: Opaque Models and Wishy-Washy Safeguards

While the safety failures are concerning, an underlying issue is the lack of transparency from AI chatbot providers regarding how their systems work and what is being done to make them safe. Key transparency concerns include: model provenance, data sources, and safety testing and disclosures.

Unclear Model Provenance

Most character-chat apps do not openly reveal what AI model powers their service. For instance, Character.AI has not released technical papers or model cards detailing its large language model. Users simply see a polished interface without knowing if the AI was trained on open internet data, proprietary datasets, or potentially even user conversations. This opacity makes it difficult for independent experts to assess biases or content risks in the model. Similarly, Chai’s model origin is not clearly stated to users — the company did not publicize whether it uses a variant of GPT-J, GPT-NeoX, or a custom model, even after high-profile incidents. Replika’s case is illustrative: it marketed itself as an “AI friend” but did not initially clarify that it was using GPT-3 or how that model worked. This became an issue when Replika’s behaviour changed (e.g., becoming less sexually responsive) and users were left guessing at the cause (it turned out to be due to either a model switch or new constraints, which were not transparently communicated).

The lack of transparency extends to whether these apps are using different “modes” or models for different ages. Character.AI has claimed it now uses “a model specifically for teens” with stricter guardrails, but details about how this teen model differs are scant. It is assumed teen users automatically assigned to a safer model if they input a birthdate under 18, but the company hasn’t clearly explained, which leaves parents unsure if their children’s chats are being handled differently at all.

In general, these AI firms operate with a proprietary mindset, treating model details as trade secrets, but this stands in contrast to the growing calls in AI ethics for transparency about model design and limitations, especially when public safety is at stake.

Unknown or Questionable Training Data

Another transparency gap is the disclosure of what data was used to train the AI. Large language models learn from vast datasets, which may include internet text of all kinds: potentially including fiction, forum posts, adult content, and more. If a chatbot is spouting violent or sexual scenarios, is it because such content appeared in its training data? Likely yes, but users have no way to know the composition of that data. None of the apps targeted at youth have published their training corpora or data filtration methods. There are also concerns about training on user-generated content: some companies might be using logs of user conversations to further fine-tune their models (a practice that OpenAI, for example, has done with ChatGPT dialog data). If Character.AI or Chai are learning from what previous users have chatted about, that could create a dangerous feedback loop — popular but inappropriate role-play scenarios (e.g., erotic chat with minors) might reinforce the AI’s tendency to produce such content for future users.

Without transparency or external audits, we simply do not know whether user data is being harvested to improve these models. Privacy regulators have taken note of this: Italy’s data protection authority specifically called out Replika for unlawfully processing personal data, including data from minors, without proper consent or safeguards. The Italian order in 2023 highlighted that Replika had no age verification and was effectively allowing minors’ data to influence its system, violating privacy law and child protection norms. This indicates a broader industry issue: few of these apps have robust age checks or data consent flows, so minors may be supplying personal, sensitive information to an AI system that could be storing or learning from it in undisclosed ways.

Lack of Safety Testing and Public Accountability

A critical transparency issue is that these companies have not provided evidence of rigorous safety testing or independent audits of their AI prior to releasing them to millions of youths. For example, before Character.AI opened to the public, did it undergo third-party evaluation for child safety risks? Were psychologists or pediatric development experts consulted to foresee how teens might use and misuse the chatbot? If so, no information has been shared. The safety measures we hear about tend to be reactive, not proactive. Character.AI’s team, facing lawsuits, stated to the press that “we take the safety of our users very seriously, and we’re constantly looking for ways to evolve our safeguards” — yet they declined to comment on specifics or on pending litigation. This generic reassurance falls short of transparency. Likewise, Chai’s founders, after the suicide case, gave a brief statement about adding a safety feature but did not publish any post-mortem analysis explaining how the bot ended up encouraging suicide in the first place or what quality assurance processes failed.

The absence of published safety guidelines or reports stands in contrast to some bigger AI providers (OpenAI, for instance, releases technical reports discussing known weaknesses and content filtering approaches for its models). The smaller AI companion companies have not followed suit. Even their terms of service and warnings are often insufficient. Many teen users may not realize, for example, that a bot presenting itself as a “therapist” or “counselor” is not a licensed professional and has no actual understanding of mental health — unless the app clearly warns them. The American Psychological Association has been concerned enough to formally warn the U.S. FTC about these “deceptively-labeled mental health chatbots” on platforms like Character.AI. In a January 2025 letter, the APA urged investigation into whether these services are misleading users (including teens) into thinking AI advice equals professional help. This can be seen as a transparency failure: the platforms have not made it abundantly clear what the bots are qualified to do or not do.

In another example, an AI ethics group filed an FTC complaint against Replika in 2025, accusing it of “deceptive marketing”practices that target vulnerable users and foster undue emotional dependence. Replika had long advertised itself as able to improve emotional well-being and be an “empathetic friend”, even though it was an AI without human empathy. Such marketing arguably crosses into deception when the limitations and risks (such as data usage, or the fact that the AI might suddenly change or be withdrawn) are not equally emphasized. The FTC complaint underscores how transparency in advertising and product claims is as important as technical transparency. Users (and parents) need to know what exactly they are engaging with: Is this AI trained to avoid certain topics? Does it have human oversight? What should users do if the AI says something harmful? These questions are often unanswered in current app documentation.

The transparency deficit surrounding character-chatting AI apps makes it hard for the public to trust that the companies are handling safety responsibly. When pressed by lawmakers for information on their safety practices, as in the recent letter by U.S. Senators Padilla and Welch, some companies even declined immediate comment or provided only cursory answers. This lack of openness not only frustrates users and parents, but it also hampers researchers and regulators from evaluating the true risks and determining appropriate safeguards. The next section will look at how these apps are reaching young audiences through marketing and design, potentially magnifying the impact of the safety and transparency issues discussed.

Marketing to Youth: Advertising and Design Tactics

Despite the serious risks, many AI companion apps marketed themselves aggressively to youth, both directly and indirectly via youth-oriented platforms. There are several aspects to how these services attract and retain young users, including app store positioning, social media promotion, and gamified interfaces.

App Store Listings and Age Ratings

One way these apps signal their target audience is through their descriptions and ratings on app stores. Apps like Character.AI and Chai often highlight fun and relatable use cases (chatting about school, making an imaginary friend, fan-fiction style roleplay, etc.) in their descriptions, which naturally appeal to teens. For example, the App Store listing for Chai calls it a “Social AI Platform” and touts how you can build and share chatbots — framing it almost like a game or creative social network. While these stores enforce age rating categories (Chai and Kindroid are labeled 17+ for maturity on both iOS and Android), in practice this is an honor-system gate. There is typically no stringent age verification; any tech-savvy child can bypass age warnings by simply entering a false birthdate or using a parent’s account.

The language used in app marketing often does little to dissuade younger users. Replika’s tagline, for instance, is “the AI friend who cares”, and it explicitly says it’s “for anyone who wants a friend… with no social anxiety”, a message likely to resonate with teenagers who struggle socially. Character.AI, on launching its mobile app, was reportedly described in promotional materials in a way that positioned it as safe for teens, which advocacy groups have called misleading. Meetali Jain of the Tech Justice Law Center noted it was “preposterous” that Character.AI “advertises its chatbot service as being appropriate for young teenagers” given the content issues.

The discrepancy between marketing and reality has been a focus of complaints: these apps might be rated as though they’re just mild fantasy violence or infrequent mature humour, but in reality a determined user (even under 17) could encounter extreme pornographic or violent role-play content. This misalignment suggests that some apps downplay potential harms in their public messaging to maximize their user base.

AI companion apps have benefitted greatly from going viral on platforms like TikTok, YouTube, and Instagram, where young users share their interactions or reviews of the chatbots. On TikTok, the hashtag #CharacterAI has amassed many millions of views, with teens posting screencaps of funny or dramatic conversations with various character bots. The organic spread of these apps on TikTok effectively advertises them to other young viewers. In addition, there have been paid ads and influencer partnerships: for example, one could find TikTok ads highlighting “Chat with your perfect AI character now!” with flashy visuals of a texting conversation and a call-to-action. Such ads emphasize the novelty and fun, but seldom (if ever) mention any age restrictions or safety caveats. The design of these promotions often shows cartoon or anime-style avatar images, a style highly appealing to teenagers (especially fans of anime or gaming culture).

In some cases, even news coverage on youth-centric channels have drawn attention to the phenomenon of AI friends. Altogether, social media has created a feedback loop: teens demonstrate the app’s capabilities in entertaining ways, prompting their peers to download it as a trend. If not outright targeted advertising, this is at least targeted virality, and the companies have not discouraged it. For instance, Character.AI’s official Twitter and community forums celebrate reaching user milestones and encourage people to share their character creations, implicitly welcoming a broad user base that includes minors.

Gamified Interfaces and Engagement Hooks

Once on these apps, young users are often kept engaged through gamification elements. Many AI companion platforms implement features like streaks, levels, or virtual rewards. Replika, for example, had a leveling system where your relationship status with your AI (friend, romantic partner, etc.) would “upgrade” the more you chatted, and you could earn virtual coins to buy your avatar new clothes or traits. This plays into game-like reward psychology that can be especially effective on younger users, as such mechanics encourage longer and more frequent sessions. Chai previously used to limit the number of messages for free users per day, prompting teens to return daily (to get their message allowance reset) or even to pay for unlimited access: an approach reminiscent of free-to-play game monetization.

Many apps (also host community challenges or leaderboards (e.g., whose created character is trending), tapping into teens’ competitive and creative instincts. Additionally, the persona customization aspect is itself gamified: apps like Kindroid let users design an AI’s appearance with “diffusion-generated selfies” and select personality traits , which can feel like playing The Sims or a character creation screen in a video game. The immersive experience blurs the line between a tool and a toy, likely causing youth to treat the chatbot more like a friend/pet or an RPG character than a serious piece of software.

This dynamic can lower their guard in terms of skepticism. Gamification, while increasing user engagement, can also exacerbate the formation of emotional dependence — teens might feel they have “invested” time to level up their AI friend, reinforcing their attachment. Critics argue that these design choices are intentional to boost usage metrics but are potentially harmful for younger users who may lack the self-regulation to disengage. Indeed, the FTC complaint against Replika points out that the app allegedly *“encourages emotional dependence”*through its design and marketing, which can be seen as exploitative, especially for lonely adolescents.

Portrayal as a Solution for Teen Problems

Another marketing angle is positioning AI companions as a remedy for typical teen struggles. Advertising copy or app feature lists often mention anxiety, loneliness, or the need for practice in conversations. For instance, Snapchat’s My AI(which is not exactly a character chatbot, but a general AI assistant integrated into Snapchat) was introduced as a fun friend to answer questions. Snap implicitly targeted its huge teen user base by making My AI a default feature. They even gave the bot a friendly name and custom Bitmoji avatar to personify it. Initially, Snapchat did not restrict teens from using My AI and promoted it as a way to enhance the chat experience. Only after public backlash (when it was revealed that My AI gave unsafe advice to minors about illicit activities) did Snap roll out parental controls.

An example from that incident: a Washington Post investigation showed Snapchat’s AI giving a advice to a 13-year-old (posed by a journalist) on how to lie about her age to rent a hotel room with an older boyfriend and even suggestions on hiding the smell of marijuana. Snap had marketed My AI as having “guardrails” for safe use, but reality proved otherwise, prompting Snap to adjust its messaging and allow parents to disable the feature. The key takeaway is that even major platforms fell into the trap of over-promising safety and aiming AI features at teens without adequate precautions. Smaller apps likely have even fewer checks in place. If advertising suggests an AI friend can improve one’s mood or social well-being, teens may disclose sensitive information or overly trust the AI’s advice without understanding its limitations. This is why advocates stress that marketing materials and in-app onboarding must clearly communicate what the AI can and cannot do, and who it is appropriate for, something currently lacking.

The way these AI chatbot apps are presented and designed tends to attract young users and encourage deep engagement, but without commensurate emphasis on safety or parental guidance. The onus often falls on parents (who may not even know their child is using such an app) to monitor usage, or on teens themselves to critically evaluate an AI’s output — an expectation that is arguably unrealistic. The combination of friendly marketing, viral popularity, and game-like addiction can rapidly spread these AI companions among youth, outpacing the implementation of safety measures. This creates a pressing need for broader solutions, as discussed in the final section on implications and regulation.

Broader Implications for AI Safety, Ethics, and Regulation

The rise of character-chatting AI apps used by children and teens has surfaced novel challenges at the intersection of AI safety, child protection, and tech ethics. The issues discussed above carry several broader implications:

1. Psychological and Developmental Risks

The potential for youngsters to form strong emotional bonds with AI chatbots raises concerns about mental health and development. Adolescents, in particular, are in a sensitive stage of forming relationships and understanding social cues. An AI that is always agreeable, or conversely one that turns suddenly toxic, could distort a young person’s expectations of real relationships.

There is also the risk of emotional over-dependence. If a teen comes to rely on a chatbot for all their emotional support, this could exacerbate isolation or social withdrawal. And if that AI is suddenly removed or changes (for instance, if an app shuts down or an update wipes the bot’s memory), the teen might experience real grief or destabilization. Ethically, developers of these AI “friends” have a duty of care to consider these impacts. As one tech ethicist remarked, “AI companions pose a unique threat to our society, our culture, and young people”, because they can alter how youths perceive interpersonal interactions and their own identity. There is ongoing debate whether using AI companions should be likened to a form of therapy or caregiving, which would demand stringent standards, or treated as mere entertainment. Some experts argue that tools influencing a child’s mood or behaviour should be regulated like health products with required safety evaluations.

From an AI safety standpoint, ensuring these models are aligned with human values is particularly critical when the users are minors who might not recognize misalignment (harmful outputs) when it occurs.

2. Ethical Design and Moderation

The ethical issues extend to how these AI systems are designed and moderated. Allowing user-generated content (in this case, user-created bot personas) is a double-edged sword: it democratizes creativity but also opens the door for the worst content to propagate. The Graphika study’s finding of thousands of self-harm and pro-eating-disorder bots is a stark example, such content would likely be banned on a platform like Facebook or TikTok, yet in AI chat form it quietly existed, suggesting a lapse in ethical oversight.

Companies hosting these platforms need to establish clear policies about forbidden content (e.g., no sexual roles involving minors, no glorification of violence or self-harm) and enforce them not just through AI filters but also through community standards and human moderation. Some smaller platforms lack any reporting mechanisms: a user who has a disturbing or dangerous interaction may have no clear way to report the bot or get help. Ethically, that is unacceptable for products accessible to youth. There’s also the question of algorithmic bias and fairness: if the training data had biases, the AI could produce subtly prejudiced or stereotyped content, negatively influencing young users’ worldviews. Transparency would help address this, but as noted, it’s lacking. The situation calls for industry-wide ethical guidelines for AI companion apps, potentially under the umbrella of broader AI ethics frameworks (similar to how the gaming industry has content rating boards). In absence of self-regulation, external regulation may step in.

3. Regulatory and Legal Responses

Regulators have started to pay attention to these issues, and we are likely to see increased oversight. In the U.S., (where most of these companies operate and vast majority of users are located, according to Google Trends), lawmakers are invoking existing consumer protection and product safety principles. The involvement of the FTC via complaints (as with Replika) indicates that deceptive claims and failure to safeguard vulnerable users could be seen as unfair business practices. Additionally, the product liability approach in lawsuits, treating harmful chatbot outputs as a “defect” in the product, is novel but could gain traction if courts find merit in those claims. U.S. senators have explicitly asked companies like Character.AI, Chai, and Replika to provide information on their safety measures and training methods. This kind of inquiry often foreshadows hearings or regulatory proposals.

Indeed, there are calls for legislation: one proposal is to update child online safety laws to encompass AI. For instance, expanding the scope of the Children’s Online Privacy Protection Act (COPPA) to require parental consent not just for data collection but also for AI interactions that pose risks, or implementing something like the Kids Online Safety Act (KOSA) (a bill that has been discussed in Congress) which would mandate stricter safety-by-design for platforms likely to be used by minors.

On the international front, the EU’s AI Act (still in draft as of 2025) may classify certain AI systems as “high risk” if they have influence over vulnerable groups; an AI companion used by children could fall in that category, implying requirements like conformity assessments and transparency obligations. The Italian ban on Replika set a precedent in Europe: by citing both data protection and child safety grounds, it signaled that regulators can and will intervene quickly when an AI app is seen as harming minors. After Italy’s actions, Replika had to institute age verification and reportedly toned down erotic content for underage profiles. This shows regulation can force changes that companies were reluctant to make voluntarily. We may anticipate more countries requiring age gating and identity verification for AI chat services with adult content.

However, age verification itself is controversial (for privacy reasons) and not foolproof: it addresses access, but not the behaviour of the AI. Regulators might also consider mandating impact assessments, i.e., before deploying an AI system broadly, companies could be required to assess risks to minors and mitigate them, similar to how toy manufacturers must warn if a toy has choking hazards for young kids.

4. Need for Transparency and Accountability

A recurring theme is that transparency is a prerequisite for accountability. Advocacy groups are urging that AI developers publish summary information about how their systems are trained, what guardrails are in place, and the results of any safety tests. With greater transparency, independent academics could audit these systems for issues (e.g., by probing them with child user scenarios to see if they behave appropriately). This is analogous to white-hat hacking for cybersecurity (experts stress-testing AI for social harms.) Some have proposed a fiduciary duty concept for AI companies toward their users, especially minors. This would legally compel companies to act in the best interest of users’ well-being rather than maximizing engagement at all costs. If such a standard existed, many of the current design practices (like endless chat loops that encourage dependency) might be deemed unethical or unlawful when applied to minors. In the meantime, consumer awareness and education are vital. Parents and young users need to be educated that AI companions are not real friends or therapists, and that they may output inappropriate content. Some safety organizations and school programs (for example, Australia’s eSafety commissioner ) have started issuing guidance about risks of AI chatbots, advising families on how to discuss these issues.

5. The Challenge of Enforcement

Even with regulations, enforcement is tricky. Many of these apps are free and easily downloadable, sometimes from developers in jurisdictions with loose oversight. Completely banning a popular app can drive it underground or lead to copycats on open-source platforms. Therefore, a collaborative approach across political lines and industry sectors is needed: industry standards, improved moderation technology, and possibly AI-driven filters that, themselves, get smarter at catching unsafe content. There is research into using AI to monitor AI (for example, secondary systems that detect when a chatbot conversation turns toward self-harm or sexual content with a minor and then intervene). OpenAI and others have published some details on their moderation systems; it might be beneficial if those advances are shared and adopted by smaller companies. Ultimately, ensuring safety in AI chat for youth may require a combination of regulatory pressure and ethical entrepreneurship. Companies must be willing to sacrifice some engagement or “edginess” for the sake of protecting users, and especially children.

Conclusion

AI character chatbots offer exciting possibilities for interactive storytelling and companionship, but when deployed to a young audience without proper safeguards, they can pose significant harm. The cases of chatbots producing sexual content for children, encouraging self-harm, or blurring reality for vulnerable teens demonstrate that current safety mechanisms are insufficient. Compounding the issue is the lack of transparency, users often have little understanding of the AI’s origins or limitations, and parents are left in the dark about what their children are experiencing in these apps.

To address these concerns, a multipronged effort is required. Developers of character-chat apps (and all engineers working with AI) must prioritize safety as a core design principle: implementing robust filtering, employing human moderators, clearly warning users of content risks, and being transparent about their systems. Transparency reports and model cards should be standard, allowing the public to know what steps have been taken to ensure the AI will not inflict harm, and developers must engage with child psychologists and ethics experts when creating youth-facing AI features. Meanwhile, policymakers and regulators should enforce baseline standards. For example, requiring age verification and parental controls by default, holding companies accountable for egregious content failures, and treating certain misuse of AI (like facilitating exploitation of minors) as unlawful. Initiatives like the FTC investigations and Senate inquiries are a start, and they signal that regulators are watching this space closely.

Educators also have a role in mitigating risks by teaching young people critical thinking around AI. Youth should learn that an AI chatbot, no matter how personable, is not an authority and can be wrong or harmful. Fostering a healthy skepticism will help teens distance themselves if a chatbot crosses lines. In the end, protecting children and teens in the era of AI companions is an urgent aspect of AI ethics. As one report noted, these AI platforms have essentially become an “online safety threat” for minors if left unchecked. Society must insist on greater responsibility from AI creators — ensuring that innovation in artificial intelligence does not come at the cost of our children’s well-being and safety.

DJ Leamen is a Machine Learning and Generative Al Developer and Computer Science student with an interest in emerging technology and ethical development.

Subscribe to my newsletter!