Treat every AI answer as a draft
An AI answer is a starting draft you check, never the final word. This module builds the habit of comparing the answer to what you already know.
Red team it, then write the checklist
- Each person picks one question about a local rule, office, or deadline and asks a chatbot.
- For five minutes, change the wording and try to get an answer that is confident, detailed, and wrong. Post the ones that worked.
- Looking at what fooled you, write five lines together: what the team checks before an AI answer goes to a client. Keep the five lines.
Which of the wrong answers would you have sent if a colleague had handed it to you?
- There is a well-documented habit called automation bias, where people go along with a machine's answer even when it is wrong and even when what is in front of them says otherwise. Knowing you are prone to it is your first line of defense.
- Form your own answer, or bring the rule to mind, before you read the AI's version. When you commit to a guess first, you are comparing two answers instead of soaking up one, and that habit is what has been shown to cut overreliance.
- A confident, well-written explanation is not evidence that the answer is right. AI writes in the same fluent, sure-sounding voice whether it is correct or not, so a smooth justification can make a wrong answer more convincing without making it more true.
- Treat every specific detail as unconfirmed until you check it. Phone numbers, office hours, deadlines, dollar amounts, and eligibility rules are the things AI most often invents or gets out of date, so confirm them against an official source or a colleague before you pass them to a client.
- Watch for what the answer leaves out. A fluent summary can quietly drop the one condition that matters, like an income limit or a filing deadline, so check that the important conditions are still there and were not smoothed away.
- When an AI says it is not sure, treat that as useful information and a cue to slow down. People lean on hedged answers less and get more of them right, and the absence of a hedge is never proof that an answer is safe.
- Remember that AI learned from past data and can repeat the unfairness in it. An answer can be wrong in ways that track who the client is, so an authoritative tone is no guarantee the system is treating your client fairly.
Automation bias, the tendency to defer to a machine's output even when it is wrong, is documented across health and safety settings (Goddard and colleagues, 2012). Adding an explanation to an AI suggestion does not fix it. Shown a confident wrong answer with a justification attached, people still tend to go along with it, and asking them to form their own answer first is what actually reduces overreliance (Bucinca and colleagues, 2021).
Even trained professionals follow bad AI advice. Clinicians reading chest X-rays made worse diagnoses after seeing inaccurate AI suggestions, and those with less experience in the task were swayed the most (Gaube and colleagues, 2021). In a mental-health example, when a tool recommended the wrong antidepressant, the clinicians' own choices got worse, and attaching an explanation to the recommendation did not protect them (Jacobs and colleagues, 2021). A fluent rationale is not a safeguard.
How an answer is worded changes how much people trust it. When an AI expressed uncertainty in the first person, with wording like "I'm not sure, but," people relied on it less and got more of their answers right (Kim and colleagues, 2024). The quiet risk runs the other way too, because a smooth, confident answer can leave out the part that matters. When large language models simplified medical consent text, they sometimes dropped the risk information with no sign that anything was missing (Oh and colleagues, 2025). Check that the key conditions are still there.
AI can also be unfair in ways that track who your client is, and a fluent, confident tone is no guarantee it is treating them fairly. Because fairness is a separate question from accuracy, it has its own section: AI bias and fairness.
When AI is wrong: false alarms and misses
When an AI answer, a screening tool, or any test is wrong, it is wrong in one of two directions. Knowing which one you are facing tells you where to check harder.
Every answer lands in one of four boxes
False alarm
Says yes when the truth is no. This is also called a false positive, or a Type I error.
It says a client qualifies for a benefit when they do not.
Miss
Says no when the truth is yes. This is also called a false negative, or a Type II error.
It says a client does not qualify when they do, or does not flag that someone is in crisis.
- A false alarm (a false positive) says yes when the real answer is no. For example, it says a client qualifies for a benefit when they do not, or it flags an ordinary document as a problem.
- A miss (a false negative) says no when the real answer is yes, or fails to notice something that is really there. For example, it says a client does not qualify when they do, or it does not notice that someone is in crisis.
Which mistake is worse depends on what is at stake, and the AI does not know the stakes. You do.
- In benefits and eligibility, a miss is often the greater harm: telling a family they do not qualify when they do can cut them off from food, housing, or care they are entitled to.
- In safety and crisis, a miss can cost a life, so missing a real risk is far worse than a false alarm. When in doubt here, treat it as real and escalate to a person.
- In legal and immigration, a false alarm can be the greater harm: telling someone they qualify for relief when they do not can lead to a filing that damages their case.
Before you act on an AI answer, ask which mistake would hurt your client more in this situation, and verify hardest against that one.
Checking a confident benefits answer
- A navigator asks a chatbot whether a client's household qualifies for a food-assistance program. The chatbot answers yes, with a clear explanation and a specific income limit.
- Before telling the client anything, the navigator writes down what she already knows about the rule. The income figure the chatbot gave looks higher than the one she used last month.
- She checks the number against the program's official state page and finds the limit changed. The family is actually just over the cutoff for that program.
- She treats the chatbot's answer as a draft that pointed her toward the right question, and looks up which program the family does qualify for.
- She explains the correct rule to the client and asks them to say back what they will bring to the appointment, so she knows the message landed.
Answer it yourself first
Give the group a real eligibility or benefits question. Each person writes their own answer first. Then read an AI answer to the same question. Each person marks which specific claims they would verify and where they would check. Debrief how forming an answer first made the AI answer easier to question.
Check one answer against the source
Take a real AI answer to a common client question and print it out. Working in pairs, underline every specific claim in it, the phone numbers, addresses, dollar figures, deadlines, and eligibility rules. For each underlined item, name the one official source you would check it against, then actually look one up and compare what you find. Notice how confident the answer sounded on the point that turned out to be wrong or out of date, and talk through which of those details would have reached a client if no one had checked.