Back to Writing

Chain of thought, and where it breaks

Telling a model to work through a problem step by step does make it more accurate. Nobody mentions the second half, which is that you usually don't want your users to see any of that working.

2 min read

Chain-of-thought prompting gives a model the route instead of just the destination. You break a problem into steps and have it work through them, rather than asking for the answer and hoping. It’s the model showing its working, and unlike at school, the working is useful to you.

How it works

Say you want a model to predict a stock movement. The naive prompt is “Will Acme Corp go up or down tomorrow?” A chain-of-thought prompt walks it through instead:

  1. Analyse Acme Corp’s latest financial reports.
  2. Consider current market trends affecting its industry.
  3. Evaluate recent news and announcements.
  4. Based on the above, predict the likely direction tomorrow.

You’re guiding the process rather than pulling out a verdict.

Why it helps

  • Better accuracy. Breaking the problem up makes illogical leaps and hallucinations less likely.
  • Explainability. You can read the steps and see where it went wrong.
  • Harder problems. The model handles more nuanced tasks when it has room to work through them.

Where it breaks

Most advice stops before this bit. Chain of thought is easy in a chat window, where you’re reading along anyway. Inside a deployed application it’s harder, because the reasoning is now part of the output, and usually you don’t want it there.

I made the same point in simple prompting tips:

If you are generating something that will be shared with an audience, you don’t want the step-by-step thinking in there.

“Think step by step” works, but only when the reasoning is something you’re happy to show. To get the accuracy without leaking the working into production output, you need one of these:

  • Ask for the reasoning inside <thinking> tags, then strip them out in code before anything gets rendered.
  • Use multi-shot prompting. Pass an initial output back to the model to refine, and surface only the final pass.

Either way it comes back to the same thing that governs prompting generally: be clear about what you want the model to produce, and check that the technique you’ve reached for actually fits the format you have to ship.