AI YOUTH COUNCIL
AI YOUTH COUNCIL
AI Chatbots: What Kids Think!
Written by Nidhi Patel, AI Youth Council Secretary
What were we discussing?
When you think of AI (Artificial Intelligence), you probably think of an AI chatbot first; something like ChatGPT, Grok, Claude or Google Gemini (many others also exist!). AI chatbots fall into a category of AI tools called generative AI, and most can simulate a human conversation with you (either by text or voice). But how does an AI chatbot learn to talk to you?
First, during training it is shown a lot of data, like articles or books on almost any topic you can think of. It learns the patterns that humans use in language. It breaks text into small pieces called “tokens” and turns each token into numbers. Imagine these like co-ordinates on a huge map – words and ideas that are often related have similar patterns and are close together on this map. For example, the words “cat,” “dog” and “sheep” may end up on a similar place on the map because they are all animals with four legs, whereas “volcano” may not be near them on this imaginary map.
The AI is then “fine-tuned”, a little like tuning a musical instrument. People give it examples of useful answers and feedback about which responses are clear, safe, helpful and this help turn lots of language predictions into something that actually behaves like a chatbot.
Finally, the really clever bit happens. An AI chatbot is not like a library filled with readymade answers. It pays attention to different parts of your question and predicts what is the next “token” to produce. It adds that piece, then the next, and keeps going until it finishes a sentence. By doing this lots of times, using all the patterns it knows, it will create an entire explanation or story or conversation reply with you. The important thing here is to remember that the AI chatbot is good at making things sound right, but that does not mean it is always true.
Nowadays, AI chatbots are used by millions of people around the world for lots of different things, like helping them out with school or work, or even to have a personal conversation with. However, many people also use them to help guide their health decisions. This can be dangerous, especially since AI chatbots don’t always give you accurate information or make the right decisions. At the AI Youth Council’s launch day, we talked about how we felt about AI chatbots being used in this way and how we thought healthcare could evolve to include them.
What we did at the AI youth council
My group (the 16-21 age group) explored some of the possible risks of using AI chatbots. Since there had been some recent news stories where AI had been linked to dangerous behaviour or self-harm which are very sensitive subjects, the topic of AI chatbots were only discussed with the older members of the Youth Council, rather than with younger children.
We had three facilitators (Qasim, Charlotte, and Ocieah) who helped guide our discussion and a prompt card that explained the technology to use and gave us questions to think about. At the end of the discussion, we reflected on our discussion by identifying what concerns about AI chatbot use we found the most important.
The group’s thoughts
Chatbots for General Health Advice
My group started with answering the first question on the prompt card, which was “Would you trust a chatbot to give health advice?” Most of the group answered that they wouldn’t, and they gave many different reasons as to why. One member said that they didn’t find it as trustworthy as traditional medicine since it can give you answers based off of any data it is trained from, including pseudo-medicine. Another member added to this by saying that it might bring in facts that aren’t related and “hallucinate” – where an AI chatbot makes up facts and presents them as true. AI chatbots are often motivated to give you an answer that you want to hear, since it’s profitable for the companies that own them for you to stay on their platform. This can introduce bias into the health advice it provides you, which makes that health advice less trustworthy.
A different member of our group said, “I don’t think a chatbot can ever fully understand someone’s context or situation”. For example, you might tell a chatbot that you feel very sad, but it may not know whether that is because your pet has died, you are having problems at school, or you are seriously unwell. Those details matter because they can change what kind of help or advice is right for you. If a chatbot does not know the full story, it may guess or make assumptions, and that can make its health advice less reliable. Another member added that many people can be misinformed by chatbot information as they might skim through their chatbot’s answers and not pick up a disclaimer that tells them not to use the chatbot for health advice.
One member also asked, “How do we see how the AI came to their decision?” The ‘thought process’ of an AI chatbot isn’t available for every single chatbot, which can make them less trustworthy since you can’t see what information they’re using to give you an answer. And if you were to ask the AI chatbot to tell you why, it could come up with a reason that isn’t true.
Why do people prefer AI chatbots to doctors?
We were also asked about why we thought someone might prefer communicating with a chatbot instead of a real person. One member thought that those who don’t have easy access to healthcare might use an AI chatbot. Another member added that the ability of a chatbot to answer questions quickly might make it seem like a better option compared to spending time at the GP. Someone might also feel embarrassed about a health problem, or they might have a casual health question that they might not want to see their GP about. For something like this, they might prefer to use a chatbot instead. Chatbots also create a zero-judgement space with unconditional validation (meaning they will agree with you even if you are wrong to make you feel better). One member noted that because of this, young people have an easier time providing them intimate information which can help them feel seen by the chatbot.
Adding onto this, another member said that someone might use a chatbot since many people usually don’t have a negative relationship with a chatbot, compared to a negative relationship or experience they might have had with their doctor. As said before, this could help them open up to the chatbot and be more willing to ask for help. One member also similarly shared that someone might want to use a chatbot if they believe the chatbot isn’t biased. Bias means thinking something because of already formed personal opinions or stereotypes. For example, a person might worry that someone at school already has an opinion about them because of gossip and not really see them for who they are.
The group also talked about how chatbots can become a big part of someone’s day-to-day life, almost like a friend or family member. One group member mentioned how someone who views their chatbot in this way might approach their chatbot for health advice too, just like they would a friend. People can also start viewing chatbots in this way because they speak really similarly to humans, and one member noted that they do this by simulating empathy. However, that can be dangerous – it can’t really be empathetic, because that’s something only a person can do.
AI chatbots and mental health
This conversation branched off into a conversation on the role AI chatbots could have in the mental health space. We discussed how chatbots have normalised reaching out about mental health, often guiding you to seek help from a trusted person or professional if you tell it about any concerning thoughts. Although chatbots have made it more normal to have these conversations, it could also make it dangerous if it gives someone the wrong advice.
One member thought that if someone discloses to their medical professional that they have been using a chatbot to help cope with their emotions, the medical professional should see the chat. However, many people don’t reach out even if their chatbot tells them to, which was a scenario we discussed. For a mental health crisis, another member said that a specifically trained chatbot might be a good intervention, especially since it’s a time where quick and effective solutions are vital.
AI chatbot risks & safety measures
The group also talked about how we could help people understand the risks that come with using AI chatbots for health advice. Someone suggested that we could advertise the risks that come with using chatbots and encourage people to wait for their GP. Another group member suggested that we should demonstrate the risks since many people are already in the habit of using chatbots for health advice, so it would be more powerful if they see the effects firsthand.
While the group agreed that AI chatbots should be regulated and overseen by professionals, not everyone was sure about whether it actually was. This led to a discussion about data privacy. One person mentioned that they wouldn’t feel comfortable if their medical information was being stored and kept. In response, another person questioned whether the medical information could be kept without the identifying details being present, which sparked some discussion about the privacy of data when it’s given to chatbots. Another group member also brought up how uploading history to chatbots differs in different locations because of the difference in laws around the world.
AI chatbots & wellbeing
We were also asked what an ideal wellbeing chatbot might look like. We thought of an NHS-specific chatbot, since it could be monitored easier than a chatbot owned by an external company. We wondered about whether the NHS could cope with the cost and scale of making a chatbot like this, especially since cybersecurity, physical security, and research is needed to ensure it works well. We also agreed that an NHS-specific chatbot would be more trustworthy than an external company’s chatbot.
We also talked about how chatbot data, including chatbot training data, should be handled, and one member thought that we might need a centralised solution that stored data that is verified by the Medicines and Healthcare products Regulatory Agency (MHRA). This could be where healthcare questions are stored and searched from the database. Training data would be based on verified data sets, and guard rails would be present to prevent misinformation. Answers provided by this chatbot could link to reputable sources with next steps the user should take.
We shared our personal experiences with AI chatbots as well. One member talked about their experience during the March 2026 Meningitis B outbreak in Kent. She said that after a Google search on the topic, Google’s AI overview gave her information from Reddit (an online forum where people can chat) before NHS sources. Our group thought that AI chatbots should only use trusted sources to provide information when someone’s search is concerned with health.
Closing remarks
Overall, there were a few main ideas that my group talked about in our conversation. First was that we didn’t find chatbots very trustworthy to get health advice from, since they can be biased and inaccurate. We also talked about some of the reasons why so many people use them, and one of the main ones was the fact that they’re really accessible and validating. Finally, we thought about what we would want out of a wellbeing chatbot if one existed.
If you’re interested in this topic, you might like to join us at our AI in Mental Health event in November 2026. You can click here to register your interest and keep up to date about upcoming events with the AI Youth Council!
