Why AI That Worked in the Lab Fails in the Field, and How to Add Data Site by Site
The AI you built with a university or a vendor showed good accuracy in the PoC. But when you connected it to the cameras in the field, it did not hit nearly as often as expected. This happens at many companies that try to bring AI into real operations.
The cause is usually not the model itself but the fact that the “data conditions” in the lab and in the field are different. This article explains why AI that worked in the lab misses in the field, and how to raise accuracy by adding data site by site, without rebuilding the model.
Table of contents
- The short answer: measure under field conditions before you rebuild
- Five conditions that differ between the lab and the field
- Overall accuracy alone won’t show you the cause
- How to add data site by site: five steps
- How to avoid rebuilding every time you add a site
- What to avoid when collecting data
- Frequently asked questions
- Summary
The short answer: measure under field conditions before you rebuild
An AI model performs well on data that looks like what it was trained on, and poorly on conditions it has never seen. If the data you collected in the lab differs from the data in the field, a drop in accuracy is to be expected.
You don’t need to rebuild the model or switch to a different method right away. Start with these three steps:
- Measure the current model’s accuracy on data from the same conditions as the field
- Identify the conditions where it misses
- Add data only for those conditions and retrain
In many cases, this alone gets you close to usable accuracy. The key is to turn “it just doesn’t work well” into “it misses under these conditions.”
Five conditions that differ between the lab and the field
Lab data and field data typically differ in the following ways.
| Condition | Lab (PoC) data | Field data |
|---|---|---|
| Capture | Fixed camera, angle and distance | Angles and distances vary with where cameras can be mounted. Some cameras are low resolution |
| Lighting and environment | Constant lighting | Appearance changes with morning, noon and night, sun and rain, and the seasons. Backlight, reflections and dirt appear |
| Types of targets | Mostly the parts and model numbers you had on hand | New model numbers and target conditions the lab never saw get mixed in |
| Background and surroundings | Clean, uncluttered background | People, vehicles and materials appear in the frame and can partly hide the target |
| Judgment criteria | Criteria set for the research goal | Operations decide where a defect starts, and that can differ from the research criteria |
The one most often overlooked is the last: judgment criteria. A model can be correct from a research standpoint and still be seen as “missing” if it does not match what the field expects. Before adding data, align with the people on site on what counts as a hit.
Lab data also tends to be biased toward images that are easy to capture and easy to judge. A model trained only on clean images is weak on the hard-to-see data that shows up in the field.
Overall accuracy alone won’t show you the cause
When you measure accuracy in the field, an overall accuracy figure doesn’t tell you what to fix. Even if the model does well on daytime footage and misses most of the time at night, the overall number can still look “okay.”
To find the cause, you need to measure accuracy separately for each condition. For example:
- By site (Plant A vs. Plant B, Site 1 vs. Site 2)
- By camera, angle or distance
- By time of day or weather
- By target type or model number
Also count missed detections (target present but not detected) and false alarms (detected when nothing is there) separately. The overall count can match simply because misses and false alarms cancel each other out. Which one hurts the field more also determines which data you should add.
How to add data site by site: five steps
1. Build an evaluation set under the same conditions as operations
First, create test data for measuring accuracy (an evaluation set) from the data that actually flows in from the field. Use only items whose correct answers are confirmed by field records or by staff on site. Do not use the evaluation set for training, and use the same set every time you measure.
2. Measure accuracy by condition and identify where it misses
As in the previous section, break accuracy down by site, camera, time of day, target type and so on. Once you can put the problem into words, such as “many misses on night footage” or “many false alarms on the new model number,” you know which data to add next.
3. Add data only for the conditions it misses, then retrain
Collect field data for those conditions, label it and add it to the training data. Narrowing the conditions keeps the workload down while raising accuracy efficiently. As a rule, keep the training data you made in the lab and train on it together with the field data.
4. Test on another site to see how far it generalizes
Once accuracy is good at one site, measure it on data from a site with different conditions. Knowing whether the model works as is, or needs a little more data, lets you estimate the effort of rolling out to the next site.
5. After adding data, measure again on the same evaluation set
After retraining, measure accuracy again on the evaluation set you built in step 1. Check not only that the weak conditions improved, but also that accuracy on conditions that were already fine has not dropped. Repeat, and start using the model in the field for each process that reaches the pass line.
Even after accuracy improves, conditions in the field keep changing. We cover who should handle retraining after launch in this article.
Who Retrains Your AI After the PoC? How to Keep It from StallingWhy AI needs retraining after a PoC, three options for who handles it, and what to agree on before handing the work over.annotation-support.com
How to avoid rebuilding every time you add a site
Rebuilding the model every time a site or part is added drives up both effort and cost. Setting things up in one of these ways from the start keeps rollout manageable.
| Approach | Good fit when | Watch out for |
|---|---|---|
| Keep one shared model and add data from each site | Sites look similar and use the same judgment criteria | Each time you add one site’s data, check that accuracy at other sites has not dropped |
| Start from a shared model and fine-tune per site | Cameras and environments differ a lot between sites | The number of models grows, so you need to track which version runs at which site |
| Cover only the weak spots with another method | A few processes or locations are hard to see no matter what | Don’t make the AI judge everything. Combine it with location data or field records |
Whichever approach you choose, two things always help:
- Use one labeling guideline across all sites: if correct answers are assigned differently by site or by person, accuracy won’t stabilize no matter how much data you add
- Keep condition information with the data: if you record site, camera and date and time, you can later measure accuracy by condition and find the data you’re missing
What to avoid when collecting data
- Collecting only clean shots: hard-to-see, hard-to-judge data is exactly what causes misses in the field
- Throwing away unclear data: keep unclear items aside, confirm the criteria with people on site, then label them
- Mixing evaluation and training data: training and evaluating on the same data makes accuracy look better than it is
- Targeting image count alone: adding more data for conditions the model already handles barely moves accuracy in the field
Frequently asked questions
Q. How many images do we need before the model is usable in the field?
It varies widely with the target and conditions, so there is no single number. We measure current accuracy on an evaluation set, then add data for the conditions it misses and measure again, and find the right amount that way.
Q. Do we have to rebuild the model when new sites or parts are added?
Usually not. We measure accuracy on data from the new site and retrain with data added only for the conditions where it dropped.
Q. Is the training data we created in the lab no longer useful?
It is still useful. As a rule, keep the lab data and train on it together with field data. Field data fills in the conditions the lab data doesn’t cover.
Q. How should we handle data that is hard to judge?
Collect unclear items separately, confirm the criteria with the people on site, then label them. Add each decision to the labeling guideline so the same question is answered the same way next time.
Q. Is it safe to share field data with an outside team?
We sign an NDA and agree in advance on how data is transferred and stored. See our security practices for details.
Summary
When AI that worked in the lab misses in the field, the cause is usually not the quality of the model but the difference in data conditions between the lab and the field.
Before rebuilding, measure accuracy on an evaluation set built under operating conditions, identify the conditions where it misses, and retrain with data added only for those conditions. Repeat this site by site, and you can widen where the model works without rebuilding it.
AI that misses in the field, brought to usable accuracy. Start with a technical consultation.
