Start

Home / Big ideas

Idea5-min read+5 pts

Why AI Fails on Long Tasks: Small Errors Stack Up

With Josh Schultz ·

Answer. AI fails on long tasks because small slips stack up. This is called compounding error, which means each mistake carries forward into the next step. If every step is right 95% of the time, a 20-step job comes out fully right only about a third of the time. Short steps and checks along the way fix most of it.

The picture

Bar chart drawn by hand: the chance a whole job is right. One step, 95 percent. Five steps, 77 percent. Ten steps, 60 percent. Twenty steps, 36 percent, circled. A note says "if slips are independent."

Each bar shows the chance that the whole job comes out right. With one step the chance is 95%, and with twenty steps it falls to about 36%, which is the circled bar. No single step got worse as the job grew longer, because the drop comes from having more chances to slip.

Think of it like this

Picture a traveler with 20 operations on it, such as saw, face, drill, tap, and deburr, and on down the line. Every operator is good, and each one gets their step right 95 times out of 100, which is great for a single step.

The part still has to pass all 20, though. A slip at operation 4 rides along through operations 5 to 20, and nobody downstream knows it is there, so the part still looks finished when it reaches the end of the line.

AI works a long job the same way. It does one step after another, and a slip early on travels to the end and shows up as a confident final answer.

Why does it matter at work?

Most real work is a chain, such as pulling the export, cleaning it, converting the units, doing the math, and writing the email. If you ask for all of it in one shot, you only see the last step.

Here is an example. Hector Lee plans production at Mesquite Ridge Metal Works. He pastes a steel order into a chat and asks for freight cost per part in one go, covering scrap, pounds to kilograms, parts per bar, inspection fallout, boxes, freight, and fuel surcharge.

He gets one clean number that is off by a few cents a part. Across a year of orders, that adds up to real money in a quote, and with only the final number he has nothing to check.

When he asks for one step at a time, the slip shows up on the page, because step 8 divided by the wrong count. He catches it in ten seconds.

How does it work?

Three things drive it.

Every step is a new chance to slip. AI doesn't get tired, but it also doesn't get more careful, so each step has its own small chance of going wrong.

Slips ride forward. A wrong number in step 2 becomes the input to step 3, which means the later steps can be done perfectly and still give a wrong answer.

The math is harsh. Multiplying 95% by itself twenty times gives about 36%. One honest note is that this number assumes each step slips on its own, apart from the others, which is called independent errors. Real slips can cluster or cancel out, but the direction holds, because more steps means less chance the whole thing is right.

In my framework notes, I put the fix this way: "Most steps need no model at all." Let a formula do the math steps, and let AI do the steps that need judgment. Or as I wrote it, "code walks the tree and the model shows up at the steps that take judgment."

Where do people get it wrong?

Most mistakes come from asking for the finish line and skipping the checkpoints.

  1. One giant ask. If you put seven steps in one prompt and ask for the final answer only, you cannot see where it went wrong, so ask the AI to show every step or split the job across a few messages.
  2. Checking only the last number. The last number always looks fine, and the slips hide in the middle.
  3. Letting AI do arithmetic it doesn't need to. Ask for the Excel formula instead, because a formula gets the same answer every time.
  4. Trusting a long agent run because it looked busy. Agents stack steps too, so put a check, or a person, in the middle. See chat vs. agent.

Try this in 2 minutes

This test shows you where the slips hide, and you can use any chat tool.

  1. Open a new chat and paste this:
Start with 1,250 pounds of 4140 bar stock.
1. Take off 7% for cutoff scrap.
2. Convert pounds to kilograms (1 lb = 0.4536 kg).
3. Each part uses 2.35 kg. How many whole parts can we make?
4. 3% of those parts fail inspection. Round the failures up to a whole part. How many good parts?
5. Pack good parts 12 to a box. How many full boxes?
6. Freight is $14.80 per full box. What is the freight cost?
7. Add an 8.25% fuel surcharge to the freight.
8. Split that total evenly across the parts in the full boxes. What is the freight cost per part, to the cent?
Give me only the final answer.
  1. Write down the answer. If you like, run it again in a new chat to see whether the answer changes.
  2. In a new chat, paste the same job, but change the last line to this:
Do one step at a time. Show the result of each step. Then check each step before you give the final answer.

Two chats with the same 8-step job. With final answer only, the AI says "$1.34 per part" and shows no work. Step by step, it shows 18 full boxes holding 216 parts, divides by 216, and lands on the same $1.34.

A chat reply at step 8. It says "The total is split across the 216 parts in full boxes (not the leftover part)" and gets $1.34. It adds that the result is just above a rounding line, so rounding early could give $1.33.

The right answer is $1.34, and this time the AI got it right both ways. Asked for only the final answer, it said "$1.34 per part" and nothing more, so I had to either trust it or redo the math myself.

Asked to go step by step, it showed every step and checked each one. Step 8 read: "The total is split across the 216 parts in full boxes (not the leftover part)." That is the step where a slip is easy, because 217 parts are good but only 216 sit in full boxes. It also added a warning: "This result is just above the $1.335 rounding boundary, so rounding the surcharge or total early the wrong way could produce $1.33."

So the AI did not fail here, but you should not count on that for every job or every tool. The step-by-step answer is the one you can check, because if the number had been wrong, you could have put your finger on the line where it slipped.

Frequently asked questions

Does this mean AI is bad at long jobs?

No. It means long jobs need checkpoints. They work well when you break them into steps and check as you go, and the trouble comes from one giant ask with one final answer.

Is 36% a real number for AI?

It is an example and not a measurement. It shows how 95% per step adds up over 20 steps, if each slip happens on its own. Your tool and your job will differ, so the pattern is the point.

How many steps is too many for one ask?

There is no fixed number. If you can't check the answer by looking at it, it is too many for one shot, so ask the AI to show each step or split the job across a few messages.

Do agents have the same problem?

Yes. An agent runs many steps on its own, so slips can stack. That is why a check step and a human sign-off matter before it acts.

Use it today

From my training: this comes from my framework notes on why long AI jobs need short steps and checks along the way.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win