10B Observations to 400M Unique Jobs: Deduplication at Scale

The One Job, Ten Listings Problem

When a Fortune 500 company posts a Software Engineer opening, that single job appears on LinkedIn within hours, shows up on Indeed the same day, populates Workday and Greenhouse career portals by the end of the week, and eventually surfaces on Google Jobs, SimplyHired, and a handful of regional aggregators. One job. Ten listings. Each with slightly different formatting, company name variations, and location strings.

At Canaria, we process 10,003,672,873 raw job observations drawn from Indeed, LinkedIn, 200,000+ employer ATS career portals, and dozens of additional sources. Those collapse in two distinct stages.

Stage one, within-source dedup. The same link, scraped again and again over the weeks a listing stays open, resolves to one posting. 10.0 billion observations become 1,011,115,180 job postings. Roughly 90% of what we observe is a re-observation of something we already have.

Stage two, cross-source semantic dedup. One real job posted to Indeed, LinkedIn, and the employer's own Workday portal is three postings but one opening. Multi-signal entity resolution collapses those into 433,666,433 unique jobs, each carrying 100+ structured fields. Only 42.89% of postings survive as distinct openings.

Getting both numbers right is one of the hardest problems in job market data, and simple hash-based matching cannot solve either stage.

The Naive Approach: Hash Everything

The most intuitive strategy is straightforward. Concatenate a few key fields, hash them, and treat identical hashes as duplicates. Pick job title, company name, and location. Run SHA-256. Done.

This works surprisingly well for a single source. Until you add a second one.

A naive hash of title + company + location produces three entirely different hashes. Three records in your dataset. One real job. If your analytics team reports that Amazon is hiring three software engineers when there is actually one, your demand signal is inflated by 200%.

This problem compounds at scale. With 20+ source families feeding into a single pipeline, even small per-source false negative rates (failing to match true duplicates) accumulate into millions of phantom records.

What We Actually See in Production

Our pipeline ingests observations from over 20 distinct source families. Stage two is where the commercially interesting variation lives: how much of a given source's inventory turns out to be a job somebody else already listed. The table below is measured on the current build.

Three patterns emerge from this data.

Aggregators collapse hardest, because they are republishers. ProJobFinder loses 65.6% of its postings to a canonical record held elsewhere, SimplyHired 65.0%, LinkedIn 60.5%, Indeed 49.9%. This is not a knock on those sources. It is what an aggregator is: a place where jobs originating elsewhere are re-listed. If you buy aggregator data and count rows, roughly half to two thirds of what you are counting is a job you already have.

Direct ATS feeds sit at the opposite extreme. iCIMS collapses 2.6%, UltiPro 1.1%, Dayforce 1.3%. These are employer-operated career portals, so their postings are the origin, not a copy. Workday sits higher at 27.8% because large Workday tenants also syndicate to the aggregators. For data buyers evaluating ATS-sourced feeds, this is the number that matters: near-zero collapse means near-zero wasted volume. The ATS and aggregator comparison page shows this distinction in more detail.

Stage one runs in the opposite direction. ATS portals are small, stable inventories scraped at high frequency, so a single ATS posting may be observed dozens of times before it closes, while an aggregator listing turns over faster. Roughly 90% of all raw observations are re-observations, and that share is highest exactly where stage-two collapse is lowest. The two stages measure different things, and quoting one as though it were the other is how vendor volume claims get inflated.

Why Hash Matching Breaks Down

The fundamental limitation of hash-based deduplication is that it treats the problem as exact match. In reality, it is an entity resolution problem. Four categories of variation make exact matching fail systematically.

Company name variations are pervasive. "Macy's" and "Macys Inc" and "Macy's LLC" are the same employer. Our company canonicalization pipeline processes 1.8 million company name variants using MinHash/Jaccard similarity. Without this step, every name variant creates a false split. The glossary defines the entity resolution terminology used throughout our documentation.

Job titles are unstable across platforms. "Software Engineer II" on LinkedIn becomes "SDE II" on the company's career page. Title normalization requires NLP, not string comparison. We map raw titles to canonical forms so that variations like "SW Eng II" and "Software Engineer, Level 2" resolve to the same standardized representation.

Location representations are inconsistent. "New York, NY" and "NYC" and "Greater New York Area" all refer to the same metro area. Some platforms include zip codes, others use metro area names. Our geography of hiring post covers the location parsing challenge in depth, including how ambiguous city names like Portland (Oregon vs. Maine) and the 26 US Springfields complicate geographic analysis downstream of dedup.

Descriptions get reformatted. Indeed reformats employer-submitted descriptions. LinkedIn applies its own rendering. The same content can differ by hundreds of characters while conveying identical information.

Multi-Signal Entity Resolution

Our deduplication pipeline uses multi-signal entity resolution rather than relying on any single matching strategy. The methodology page covers the technical details.

• Semantic vector similarity for descriptions. Even when descriptions are reformatted, their semantic content remains stable. Dense vector embeddings and cosine similarity identify postings that describe the same role regardless of exact wording.
• MinHash and Jaccard similarity for company names. Locality-sensitive hashing efficiently identifies name pairs that are variations of the same entity.
• Title normalization through NLP. Our model maps raw titles to canonical forms, collapsing surface-level variation into standardized representations.
• Geographic clustering. Location strings are parsed into structured components and clustered by proximity. The full field definitions are in our data schema.
• Graph-based transitive matching. If posting A matches B, and B matches C, our graph processing unifies all three, even if A and C would not have matched directly. This transitive closure step is where multi-signal approaches pull furthest ahead of hash-based methods. Isolated pairwise matching misses chains of similarity that only become visible in a graph.

The Dedup Key Design

Our primary deduplication key, jtsId, is a SHA-256 hash of four components: the job key, the job title, the source platform, and the source region. Company name is deliberately excluded because company name inconsistency would cause false splits.

This is a design tradeoff worth explaining. Including company name in the hash would increase precision (fewer false merges) but dramatically reduce recall (more false splits). Given that company name is the single most inconsistent field across sources, excluding it from the dedup key and handling company resolution separately produces better results.

Across all 1,011,115,180 postings in our delivery table, the count of duplicate jtsId values is exactly zero. This is enforced by a LIMIT 1 BY jtsId clause in the merge pipeline and verified by automated QA on every run. You can inspect the output schema, including jtsId, in the schema documentation.

Why This Matters for Analytics

The consequences of poor deduplication cascade through every downstream analysis.

• Inflated demand signals. If the same job appears five times, your hiring demand estimates are 5x too high for that position. At 433,666,433 unique jobs, even a 1% false negative rate means over 4 million ghost postings distorting demand models.
• Distorted source comparisons. Comparing Indeed job counts to LinkedIn counts is meaningless without cross-source dedup. The table above shows Indeed at 256.6M postings vs. LinkedIn at 198.4M, a 29% gap. After cross-source resolution the unique counts are 128.7M vs. 78.3M, a 64% gap. Raw row counts do not just inflate volume, they distort the ranking between sources.
• Incorrect time-series analysis. A job that persists for three months and gets re-scraped weekly appears as 12 new postings. Your time-series shows 12 hiring events. Reality shows one persistent opening.
• Skewed geographic analysis. Multi-location postings may appear as separate jobs in two cities, doubling apparent demand in both locations. This interacts with the location parsing challenge described in our geography of hiring post.

At the scale Canaria operates, 400M+ unique jobs distilled from 10B+ raw observations, even small percentage errors translate to millions of ghost records. When clients build hiring models, compensation benchmarks, or labor market indices, the accuracy of the underlying dedup determines whether those products reflect reality or amplify noise.

If you want to see what deduplicated, structured job data looks like across 100+ fields, request a sample or explore available datasets. For the full field-by-field breakdown of the 1B+-record dataset, see the Anatomy of 400 Million Job Postings; for why ATS portals carry lower duplication than aggregators, see ATS vs Job Boards: Data Quality.