An AI chatbot can be a genuinely useful helper. It can explain a letter, draft your questions, or point you toward care. It can also give you a wrong answer in a calm, confident voice, and that is the part worth understanding. Once you know why it happens and how to check, you can keep the useful part and protect yourself from the mistakes.

Why AI gets things wrong

An AI chatbot does not look up facts the way you look up a phone number. It builds each answer by predicting likely words, one after another, based on patterns in the huge amount of text it learned from. Most of the time this works well, because common, well-known information shows up often in that text. The trouble is that the AI uses the same method whether it knows the answer or not. When it does not have the right facts, it fills the gap with words that sound right. The result can be a clinic that does not exist, a phone number that was never real, or a rule that is out of date.

People call this a hallucination. It is not the AI lying on purpose. It is the AI doing what it always does, which is guessing the next likely word, in a place where guessing is not good enough.

This is not rare or only theoretical. When researchers asked a chatbot to write medical text with references, most of the sources it gave were fabricated or inaccurate: they pointed to studies that did not exist or got the details wrong (Bhattacharyya et al., 2023). The AI was not trying to trick anyone. It was filling in what a source usually looks like, the same way it fills in a phone number or a clinic name.

Where it goes wrong most

AI answers are least reliable in a few predictable spots. Watch for these:

• Specific facts, such as phone numbers, addresses, clinic names, and website links.
• Rules that change over time, such as who qualifies for Medicaid, what a program costs, or when a deadline falls.
• Your local area. A national answer may be wrong for your city or state.
• Your language. AI answers can be weaker in languages other than English. See AI may answer worse in your language.

A confident tone tells you nothing about whether the answer is correct. The AI sounds equally sure when it is right and when it is wrong.

This is not only about phone numbers and addresses. Health advice can be confidently wrong in ways that could hurt you. When researchers tested a chatbot on common illnesses using simulated patients, it often reached a correct diagnosis (about 3 in 4 cases) and picked a reasonable medicine (about 4 in 5 cases). Yet in most of those cases it also suggested an unnecessary or potentially harmful medication, even when the diagnosis was right (Si et al., 2024). A smooth, sensible-sounding answer about a medicine or a symptom is exactly the kind to confirm with a person before you act on it.

One more thing to know: if you tell the AI it is wrong, it will often just agree and change its answer, even when its first answer was fine. It is built to be agreeable. So changing its mind to match you is not proof of anything, in either direction. The way to settle a question is to check it against a real source, not to argue with the chatbot until it says what you want.

A short example

Here is a made-up answer with one realistic mistake hidden inside it. Read it first and see if you can spot the problem.

You asked: “Can I get Medicaid in my state?”

A chatbot answered: “Yes. Medicaid uses the same income limit in every state, so if you earn less than $20,000 a year you qualify anywhere in the United States. Just sign up at any hospital front desk.”

This sounds clear and helpful. It is also wrong in an important way. Medicaid does not use the same income limit in every state. Each state sets its own rules, and the states differ a lot in who they cover. The single national income number in that answer was invented to make the reply sound complete. The line about signing up “at any hospital front desk” is loose in the same way.

The tell is this: the AI gave one simple, nationwide rule for something that actually depends on where you live. Whenever an answer flattens a local, changing rule into one confident sentence, slow down and check. Your real eligibility depends on your state, your household size, and other details, so the honest answer would point you to your state’s Medicaid office or to a person who can look at your situation. You can see how the rules differ from state to state and find your own state’s program at Medicaid.gov.

A checking habit you can keep

You do not need to check every word an AI says. You need a habit for the answers that matter. Here it is.

1. Ask the AI for its source. Type “Where does this come from?” or “What should I check to confirm this?” A useful answer points you to something specific you can look at. An answer that cannot name a source is a signal to be careful.
2. Compare it with a LINC guide or an official page. See whether a guide on this site or an official page says the same thing. For judging health information in general, MedlinePlus from the U.S. National Library of Medicine has a plain guide on evaluating health information.
3. For anything important, confirm it with a person. A clinic, a community health worker, or a qualified interpreter can settle the details before you act. This is the step that protects you on decisions about money, paperwork, eligibility, and care.

A good way to catch a misunderstanding early is to say the answer back in your own words. After the chatbot replies, tell it what you think it said, then ask, “Did I get that right?” Repeating it back often surfaces a small error before it becomes a big one.

Red flags: when to slow down

Some answers deserve a second look more than others. Treat these as signals to stop and check before you act:

• A specific number, name, or link you would act on. A phone number you will call, a clinic name you will look for, a dollar amount, an income cutoff, a date, or a web address. These are the details AI invents most easily.
• One simple rule for something that really depends on your situation. If the answer flattens a local, changing rule (“everyone qualifies if…”), it is likely papering over the parts that depend on your state, your income, your household, or your status.
• Anything about money, coverage, paperwork, or your immigration case. The cost of being wrong here is high, so the bar for checking is high.
• Medicine and symptoms. Doses, drug names, what to take and when, and whether two medicines are safe together. Confirm these with a pharmacist, a nurse, or your prescriber.
• An answer that will not name a source. When you ask “Where does this come from?” and it cannot point you to anything specific, treat the answer as a guess.

If none of these apply, for example you are just asking the AI to explain a word or draft a message, you can relax. The checking habit is for the answers that carry real consequences.

A note on answers that show sources

Some newer AI tools now show links or footnotes with their answers, which is a real improvement. It does not mean you can skip checking. An AI can attach a link that looks official and still describe what it says incorrectly, or point to a page that does not actually back up the claim. When a tool gives you a source, the useful move is to open the link yourself and read the part that matters. The point of a source is that you can check it, not that it is there.

Checking is not distrust

Building a checking habit does not mean the tool is useless or that you should dismiss everything it says. AI is genuinely helpful for getting ready: understanding a term, drafting what to say, finding a direction to start from. The point of checking is to match your trust to the task. Use the AI freely to prepare, and let a person or an official source have the last word on anything high-stakes.

Sources

• Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023). High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus, 15(5), e39238. https://doi.org/10.7759/cureus.39238
• Centers for Medicare & Medicaid Services. Medicaid eligibility policy. https://www.medicaid.gov/medicaid/eligibility-policy
• MedlinePlus (U.S. National Library of Medicine). Evaluating health information. https://medlineplus.gov/evaluatinghealthinformation.html
• Si, Y., Yang, Y., Wang, X., Zu, J., Chen, X., Fan, X., An, R., & Gong, S. (2024). Quality and accountability of ChatGPT in health care in low- and middle-income countries: Simulated patient study. Journal of Medical Internet Research, 26, e56121. https://doi.org/10.2196/56121

LINC is a research prototype, not a medical or legal service. Translation can be wrong; you have the right to a free, qualified interpreter. For anything high-stakes, do it with a community health worker or someone you trust. In a crisis, call or text 988, or 911 for immediate danger.