Check legacy code with AI: separate where to fix from whether to fix
Take the fix AI finds as a candidate, and let a person decide on the change after checking the scope and the receivers.
Key points
- AI's fix is a candidate, not a go-ahead
- Log AI's scope; keep the unread parts
- Outside code, ask how the CSV is read
- Tests pass, yet a person OKs the change
Contents
AI's fix is a candidate, not a go-ahead
A&A perspective
The fix location that AI returns is a candidate for where to change the code, not a decision that the change is safe. Investigating an existing system here means having AI read the code of an in-house system that is already running, to find out how far a change would reach.
Hypothetical example
As a hypothetical example, take a company with a few employees whose order-management system was built by an outside vendor several years ago. The owner wants to drop one field that is no longer used from the spreadsheet-style file (CSV) the company sends to a business partner every night.
Hypothetical example
The owner asks Claude Code to look through the code. The AI searches for the field name, finds the code that writes the export, and replies: "Delete this one line and you're done."
Hypothetical example
That answer only tells you where the field is written. It does not tell you whether the same code also serves the download button on screen and the nightly automatic send, or what the partner's system expects to receive.
From the sources
Sourcegraph, a company that builds code search tools for developers, introduced an AI coding assistant called Cody in 2023. According to the case study Anthropic published, Cody answers questions with the whole codebase in view, and helps developers understand complex interactions in their code.
From the sources
Sourcegraph then made Claude 3 Sonnet the default model for Cody's free users. After that, the case study says, the rate at which developers inserted Cody's suggestions directly into their files rose by about 75 percent.
From the sources
The case study presents this as a sign of better code quality. The page gives no publication date and dates from the Claude 3 era.
A&A perspective
Automate & Augment (A&A), the company writing this article, reads that number as the result of developers deciding for themselves whether to use each suggestion. The people who chose to insert the code were developers who could read it.
A&A perspective
If an owner who does not read the code closely adopts the AI's answer as it is, nobody is left to judge whether the change is safe. So take the AI's answer as a candidate for where to fix, and decide separately whether to fix it.
Log AI's scope and keep the unread parts
A&A perspective
When you ask AI to investigate, have it write down what it searched and what it could not read, along with its answer. A request might read: "Find out what happens if we remove this CSV field. Do not change the code yet. Cite file names and the exact places as evidence, and list the terms you searched for and anything you could not read."
A&A perspective
Also have it record which version of the code it looked at. If the vendor is still working on the system, the code may change after the investigation.
Hypothetical example
In the hypothetical example, the AI's record falls into three groups.
A&A perspective
A record that leaves out the bottom two rows of the table, "Partly" and "Unknown", may look tidy, but it is not enough to decide on a change. When the AI searched and found nothing, have it say "not found within this scope". That is a different conclusion from "not used anywhere".
From the sources
Anthropic has published a technical article on how to build AI agents, titled "Building effective agents". An AI agent here means AI that uses tools on its own to move a task forward. The article says it matters for agents to check their progress at each step against the results of running tools or code.
From the sources
It also says agents can stop and ask for human judgment when they get stuck or reach a checkpoint set in advance.
A&A perspective
A&A reads this as a reason to have even an investigation keep "what we learned by reading the code" apart from "what we confirmed by running it". The places the AI could not read are the checkpoints where it waits for a person. You do not need to hand the AI production settings or customer data for an investigation.
| Group | AI's record (hypothetical) | Who to ask next |
|---|---|---|
| Confirmed | The on-screen download uses a shared routine that builds the CSV | No one |
| Partly | The nightly auto-send settings point to the same export target. The code that does the work is elsewhere and has not been read | The vendor who built the nightly job |
| Unknown | Whether the partner's system reads columns by name or by position | The partner's contact person |
Outside the code, ask how the CSV is read
A&A perspective
The code you have cannot tell you how the receiving side reads the CSV columns. So ask the partner or the vendor. The "Partly" and "Unknown" rows of the table are what you ask about here.
Hypothetical example
If the receiving system reads columns by name, it is enough to check that it does not use the field you are removing. If it reads them by position, removing one column shifts every later value one column over, into the wrong field. The quickest way to find out which is to ask the partner's contact person.
Hypothetical example
The nightly automatic send is the same. If the code that does the work sits somewhere else and the AI was never given it, ask the vendor who built it whether it uses the same export routine.
Hypothetical example
There is more than one option. If you leave the field in place but empty, the number and order of the columns stay the same. If you publish a new format alongside the old one, the current receiver keeps working, and you agree with the partner on when to switch. If the receiving side changes at the same time, line up both sides' work dates. Choose after you hear back from the receiving side.
From the sources
According to the case study, Sourcegraph began an experiment to bring information from outside the code, such as system monitoring metrics, into the screen where developers write code.
A&A perspective
Giving AI more material widens what it can read. Even so, promises made only by email or in meetings, such as an agreement with a partner, are not in the code. Asking about them is a person's job. Once you fill in the "Who to ask next" column of the table, you can see which questions go to the vendor and which ones you take to the partner yourself.
Tests pass, yet a person OKs the change
A&A perspective
Even when automated tests pass, a person decides whether to put the change through. If the tests only check that a CSV can be produced, nobody has checked that the partner can read it.
From the sources
The same Anthropic article says code can be verified with automated tests, and that agents can keep revising their work using test results as feedback.
From the sources
It adds that while automated testing helps verify that things work, human review remains essential to make sure the solution fits what the wider system requires.
From the sources
The article was published in December 2024. A note at the top says much of the development tooling it describes has changed since then.
A&A perspective
A&A recommends asking the vendor or the AI what the existing tests actually check before a change goes through. Base the expected test output on a sample agreed with the partner, not on a copy of what the AI produced after the change.
A&A perspective
Before deciding, have a second person open the evidence cited in the record and see whether they can explain it the same way. Even with unread places left, the change can be prepared once you have settled who will ask what, and by when. But put the change through only after those answers are in.
A&A perspective
On the other hand, for a small change where you can see every place that uses the code and every receiver, this record takes a few lines. Size the record to the number of receivers you cannot see, not to the size of the change.
Summary
A&A perspective
The fix location AI returns is a candidate for where to change the code. Have the AI write down what it searched and what it could not read, and ask the receiving side, outside the code, how it reads the CSV. Even when tests pass, a person decides whether to put the change through.
A&A perspective
How to check the business result after the change goes through is covered in "Check AI's “done” and what arrived separately: 4 client-side steps".
AI's answer is a candidate for where to fix, and a person decides whether the change goes through. Before deciding, check that the record covers what was searched, what could not be read, and who to ask next.
FAQ
If AI says a change has no impact, can we go ahead?
Not right away. First ask what scope that answer covers. If some places could not be read, the answer is not "no impact" but "nothing found within this scope".
Does the same apply when our vendor uses AI to investigate?
Yes. Ask the vendor to hand over what was searched, what could not be read, and who to ask next, along with the findings. Whether the vendor or the AI did the investigating, the company that ordered the change decides whether it goes through.
Do we need every one of these checks for a small change?
No. If you can see every place that uses the code and every receiver, a record of a few lines is enough. Spend the effort when a receiver sits outside the company or when unread places remain.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Sourcegraph Claude Platform (API) case study
Anthropic · Publication date not stated; historical Claude 3 case
Accessed 2026-10-23 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-10-23