Data Enrichment Automation: Keeping Two Systems From Quietly Disagreeing
Enrichment is the easy part. Deciding which record is correct when two sources disagree, automatically and at volume, is where these systems earn their keep.
Data enrichment sounds like a one-directional problem: take a thin record, add fields to it from external sources, end up with a fuller one. In practice the harder half of the job is deciding what to do when two sources describe the same thing differently, and doing that automatically, at volume, without a person checking every case.
Most enrichment pipelines are built for the easy direction and quietly break on the disagreement, because that part was never designed, just assumed away.
Enrichment from more than one source
A single enrichment source is comparatively simple: look up a record, pull back fields, write them in. Real systems usually pull from more than one source, because no single provider covers everything, and that is where the real design work starts.
A lead sourced from a business registry and the same lead sourced from a web scrape will rarely agree on every field. A phone number formatted differently, a company name with or without its legal suffix, an address that is technically correct in both records but written two different ways. None of these are wrong exactly. They are just not identical, and a naive pipeline that treats “found a match” as “done” ends up with duplicate records that a downstream system, a CRM, a sales tool, a scheduling system, then treats as two different things.
Matching has to happen before enrichment can mean anything. That means a defined set of rules for what counts as the same underlying entity across sources, applied consistently, not eyeballed case by case.
When sources disagree, someone has to win
Once two sources are confirmed to describe the same entity, the harder question is which value to keep when they conflict. This is not a technical problem so much as a policy one, and skipping the policy is how enrichment pipelines end up quietly corrupting good data with bad data from a less reliable source.
A workable approach ranks sources by reliability for each specific field, not as a blanket ordering. A government registry is usually the better source for a legal business name. A recently scraped website is usually the better source for a current phone number, since registries update slowly. Treating one source as universally authoritative produces a pipeline that is right about some fields and confidently wrong about others, in a pattern that is hard to notice until someone downstream acts on the wrong one.
Reconciliation is not a one-time project
The instinct once duplicate or conflicting records are found is to run a cleanup pass and consider the problem solved. That works exactly once. New records keep arriving from every source, and without continuous reconciliation the same drift starts accumulating again the day after the cleanup finishes.
This is the same lesson from a different angle as the spreadsheet ceiling: a one-time fix to a recurring problem is not a fix, it is a delay. Reconciliation needs to run as part of the pipeline itself, checking new and updated records against what already exists, every time, not as an occasional maintenance task someone remembers to run.
Surface the disagreements, do not just resolve them silently
Automatic resolution should handle the large majority of cases cleanly, using the reliability ranking described above. But some disagreements are close enough, or important enough, that silently picking a winner is the wrong move. A discrepancy in a field that drives a real decision, a contract value, a compliance status, deserves to be flagged for a person rather than resolved by a rule that happens to exist.
This is the same principle as the review queue in automating document processing: the goal is not zero human involvement, it is human involvement concentrated on the small number of cases that actually warrant it, with everything confidently resolvable handled automatically and consistently.
What this looks like when it works
A mature enrichment pipeline is boring in exactly the right way. Records from multiple sources merge into one clean entity without duplicates. Conflicts resolve according to a rule someone actually decided on, not whichever source happened to write last. The genuinely ambiguous cases land in front of a person with enough context to resolve them in seconds, not minutes. And the whole thing keeps running quietly in the background, so nobody has to notice it working, which is the actual goal. Automated reporting is the same reconciliation problem one level up, applied to numbers instead of records.
We build enrichment and reconciliation pipelines that keep running without becoming someone’s part-time job. See how we approach automation, or read about the routing logic these queues share with alerting.