Level 3Unit D · Session 22
22

Unit D · AI and chatbots

What a chatbot is, under the hood

What we are making: an accurate model of the machine. You have spent seventeen sessions refusing to trust anything you could not explain. Apply the same discipline here.

One sentence

A chatbot is a program that predicts the next chunk of text, over and over, until it has produced an answer.

That is not a simplification for school. That is the mechanism, and everything surprising about these systems — the good and the bad — falls out of it.

YOUThe capital of France isTHE CHATBOTParis.
No lookup happened. It predicted the most likely continuation.

How it got that way

  1. It was shown an enormous quantity of text — a large fraction of the public internet, books, code.
  2. It was repeatedly given a passage with the next chunk hidden, and asked to guess it.
  3. Every wrong guess nudged billions of internal numbers slightly. Every right one reinforced them.
  4. Repeat for months, on thousands of machines.
  5. What is left is a very large set of numbers that encode which text tends to follow which text.
New words
ModelThe trained program: its architecture plus billions of learned numbers.
ParameterOne of those numbers. Modern models have hundreds of billions.
TokenThe chunk it works in — a word, part of a word, or a punctuation mark.
Training dataThe text it learned from. You cannot inspect it, and neither can most of its makers.
InferenceWhat happens when you use it: repeated next-token prediction.
HallucinationA confident, fluent, entirely fabricated output.

Compare it with what you built

In Unit C you wrote a classifier. You chose the threshold, you can point at the line, and you can say exactly why it decided what it decided. That is worth holding onto as a reference point.

Your classifier~30 lines of C++You chose every constantYou can trace any decisionFails in ways you predictedYou can prove it correctRuns on a $9 boardA language modelHundreds of billions of numbersNobody chose them individuallyNo one can trace a decisionFails in surprising waysCorrectness is not even definedRuns in a datacentre
Both decide things. Only one can be explained. That gap is a real open research problem.
⚠ Careful
When a chatbot explains its reasoning, it is still predicting tokens. It is generating a plausible-sounding explanation, not reporting on its own internals — it has no more access to those than you do. This is worth knowing before you ever cite an AI's explanation of itself as evidence.

Why it is right so often

Because for an enormous amount of ordinary text, the most likely continuation is the true one. Paris really does follow “the capital of France is” in almost everything ever written. Truth and likelihood line up most of the time.

The failures happen precisely where they come apart: things rarely written about, things written about after training stopped, and things where a fluent answer exists but a true one was never recorded.

Do it — probe the mechanism

  1. Ask the same question in two separate conversations. Compare word for word. Why are they different?
  2. Ask it to continue a sentence you invent, that appears nowhere. Watch it produce something fluent anyway.
  3. Ask it something highly specific about a library you used this year. Verify against the real documentation.
  4. Ask it to explain why it gave a particular answer. Then ask whether that explanation is a report or a reconstruction.
  5. Ask it for a citation, then try to find that source.

The engineering question

You have spent the year asking one question about every device: under what conditions does this stop being reliable? Ask it here.

Kind of questionHow reliableWhat to do
Common, well-documented factsVery reliableStill verify anything that matters
Explaining a conceptExcellent — its strongest useCheck it against one other source
Code in a popular languageUsually good, plausible when wrongCompile it. Test it. Never paste unread
Anything local or recentUnreliable, and does not say soTreat every specific as unverified
Its own confidenceNot correlated with correctnessConfidence is not evidence

What you learned

  • It predicts the next token, repeatedly. That is the whole mechanism.
  • It learned from text nobody can fully inspect.
  • Truth and likelihood usually coincide. The failures are where they do not.
  • Its explanation of itself is generated, not reported.
  • Its confidence carries no information about its correctness.
Challenge optional — only if you finish early
  • Ask a chatbot how it works, then check from a real source how much of that was true.
21. Design it for somebody else