Skip to content
Canaria

Data Collection and Enrichment Methodology

How we collect, standardize, enrich, and deduplicate job market data at scale.

Platform Overview

Observations10B+ scrape observations of job postings
Unique Job Postings1B+, unique by URL and source job key
Fields per Record100+
Historical Coverage2022-present
Acquisition & DeliveryCollected daily to hourly per source; job postings delivered at least daily
Primary SourcesIndeed, LinkedIn Jobs, 200,000+ employer ATS portals (Greenhouse, Lever, Workday, iCIMS)
Geographic ScopeUS-primary plus 240+ other countries and territories
Delivery FormatsParquet (default) or CSV, via Amazon S3, Google Cloud Storage, Azure Blob Storage, Google Drive or SFTP

Data Collection Philosophy

Canaria treats job postings as observable signals of employer labor demand, not as direct measures of hiring outcomes. A posted job reflects stated intent to hire; it does not confirm a hire occurred, a position was filled, or that a single headcount was added. Researchers should apply this distinction when using posting counts as economic indicators.

The platform focuses on making what is observable reliable, standardized, and analytically usable, while explicitly preserving uncertainty and avoiding inference beyond the data's scope. Raw job postings are ingested through a distributed scraping infrastructure spanning Indeed, LinkedIn Jobs, and 200,000+ employer career portals (Greenhouse, Lever, Workday, iCIMS, and others). Coverage is US-primary plus 240+ other countries and territories, with data beginning in 2022. Sources are collected daily to hourly (near-real-time acquisition); job postings are delivered at least daily, and company, salary, and skills products on the cadence each product states.

Each posting passes through a multi-stage enrichment pipeline combining deterministic parsing, machine learning classification, and job-level entity resolution. All raw, parsed, and enriched fields are retained for transparency and auditability. When a field is absent or unparseable, the record carries a NULL rather than a synthetic fill-in; the only model estimates that fill a gap are labeled as estimates, such as predicted salary, which the salarySource column marks on every row. Fill rates are measured on every delivered field. The field priority rule throughout the pipeline is: parsed structured data takes precedence over scraped text, which takes precedence over calculated or model-predicted values.

Pipeline Architecture

Every job posting passes through an 8-stage pipeline: ingestion, deduplication, NLP classification, location parsing, salary normalization, skills extraction, title normalization, and company enrichment.

Step 01

Raw Scraped Data

10B+ observations of 1B+ unique postings (by URL and source job key)

Step 02

Deduplication

Near-duplicates grouped across sources and over time; every posting kept and flagged

Step 03a

Location Parsing

City / state / country / zip / coordinates

Step 03b

Salary Normalization

Annual USD min / avg / max

Step 03c

Description Extraction

Skills, benefits, clearance, contact

Step 03d

ML Classification

SOC, seniority, employment, remote

Step 03e

Title Normalization

Standardized titles + confidence scores

Step 03f

Company Enrichment

29M companies, industry, size, HQ

Step 04: Final Output

Final Merged Delivery

1B+ postings with their duplicate-group flags, 100+ fields, all enrichment stages joined with field priority logic

Machine Learning Enrichment

Our enrichment pipeline is a microservice-based NLP system that transforms raw job text into structured, analytics-ready fields. Each model operates independently, allowing separate scaling and iteration.

1

Title Normalization

Canonicalizes noisy titles (abbreviations, typos) to standardized forms. Outputs confidence scores so researchers can filter or weight results by precision.

Accuracy: 84.58% top-5, 69.94% top-1 across about 100,000 roles

Example: SW Eng II → Software Engineer

2

SOC Classification

Assigns 6-digit Standard Occupational Classification codes using title + description context. Enables occupational analysis aligned with the BLS 2018 SOC taxonomy.

Accuracy: 93% top-5, 73% top-1 across 867 codes

Example: 15-1252 (Software Developers)

3

Seniority Detection

Classifies career level into eight bands (intern, entry, mid, supervisor, senior, staff/principal, director, executive) from the title and description, plus a career track: individual contributor, manager, or mixed.

Accuracy: 94% primary accuracy; 99.94% populated, the classifier never declines

Example: Senior

4

Employment Type

Classifies each posting's employment type from its title and description: full-time, part-time, contract, temporary, internship, per diem, and other arrangements.

Accuracy: 95% primary accuracy

Example: Full-time

5

Remote Work Status

Classifies each posting as Remote, Hybrid, or On-Site from the posting text, with in-office days per week where a hybrid posting states them.

Accuracy: 93% primary accuracy

Example: Hybrid

6

Salary Prediction

Where a posting states no pay, a Canaria model estimates annual USD minimum, average, and maximum. Salary benchmarks fuse three sources: posted pay, employee-reported pay, and BLS OES.

Scale: 42K+ benchmark cells, 95% CI per cell, 98.2% BLS-backed

Example: $95,000 – $125,000

7

Skills Extraction (NER)

Extracts 39,000+ skills, 3,300+ certifications, and 250+ soft skills using dictionary matching plus a title-aware relevance filter that removes off-role matches.

Coverage: At least one skill on 89.8% of delivered postings

Example: ["Python", "AWS", "Docker"]

8

NAICS Classification

NAICS-2022 industry classification, resolved per employer from company name and job text signals and carried onto job, company, and place records (a 6-digit industry code on 88.77% of postings and a sector on 90.65%). Enables industry-level labor market analysis.

Accuracy: 98% top-5, 89% top-1 across 1,012 codes

Example: 511210 (Software Publishers)

Classifier Accuracy

Each enrichment classifier publishes accuracy on a held-out evaluation set, plus a per-row confidence score so consumers can filter on quality. Class counts vary from 3 work-mode classes to about 100,000 canonical titles, and the numbers below should be read against that backdrop.

SOC Classification

BLS 2018, 867 6-digit codes

73%

top-1

93%

top-5

Measured on a 2,040-row held-out gold set. The classifier reads the full description as well as the title, which matters most for generic titles such as "Manager" or "Analyst."

NAICS Classification

NAICS 2022, 1,012 6-digit codes

89%

top-1

98%

top-5

Measured on a 1,010-row held-out gold set across 1,012 industry codes. NAICS classification uses company name plus description text and is the basis for industry-level aggregations across the dataset.

Job Attributes

3 LoRA heads on a shared BGE-m3 backbone

94%

seniority

95%

employment

93%

work mode

Primary-class accuracy on held-out gold sets of 138 to 274 rows per task. Seniority spans eight bands, employment type covers ten classes, and work mode classifies remote, hybrid, and on-site. None of the three abstains: every posting they score receives a value.

Title Normalization

Arctic Embed L v2.0 + LoRA + FAISS, about 100,000 canonical roles

69.94%

top-1

84.58%

top-5

Measured on an 11,848-row production-frequency benchmark, held out from training. Read it against the class count: 69.94% top-1 is a choice among about 100,000 canonical roles. On a stricter set never seen during fine-tuning, top-1 is 52.34% and top-5 is 74.20%. Confidence scores ship with every row for downstream filtering.

A confidence score ships per row (nlpSocConfidence, nlpSeniorityConfidence, etc.). Occupation, industry and normalized-title scores are calibrated to the probability that the value is correct; seniority, employment-type and work-mode scores are the model's raw score, meaningful for ranking and filtering rather than as a success rate.

SOC Classification: Accuracy and Coverage

Canaria assigns 6-digit Standard Occupational Classification (SOC) codes using both the job title and the full description text, because a generic title such as "Manager" or "Analyst" says little on its own. Accuracy is 93% top-5 and 73% top-1 across all 867 detailed occupation codes, measured on a held-out gold set. The taxonomy is aligned with the BLS 2018 SOC standard, enabling direct comparison with government labor statistics.

A SOC code ships on 99.9% of delivered postings, with a confidence score calibrated to the probability that the code is correct, so a threshold on it reads as an expected hit rate.

Known Classification Challenges

Failure modeExample
Ambiguous titles"Manager", "Associate", "Specialist" without description context
New-economy roles"Prompt Engineer" and "Growth Hacker" have no clean mapping in BLS taxonomy
Multi-function roles"Software Engineer / Data Scientist" requires a primary SOC assignment

Salary Prediction: Model and Coverage

An annual salary ships on 90.5% of delivered postings. 42.6% of those salaries come from the posting itself, either its text or the job board's salary field; the rest are estimates, from the job board or from Canaria's model. The salarySource column on every row says which, so stated and estimated pay can always be separated.

Where a posting states no pay, Canaria's model estimates minimum, average, and maximum annual USD. All salary values are annualized and converted to USD at the posting date's exchange rate; for stated pay, the period the employer used (hour, week, month, year) ships in salaryPayPeriod. Salary benchmarks fuse three sources (posted pay, employee-reported pay, and BLS OES) and report 42K+ cells, each with a 95% confidence interval.

Coverage and Benchmarks

MetricValueContext
Postings with an annual salary90.5%Stated or estimated, delivered Job Postings files
Share of those stated in the posting42.6%Posting text or the job board's salary field; the rest are estimates
Benchmark cells42K+42,002 occupation × state plus 867 national
Confidence interval per cell95%Reported alongside every benchmark estimate
BLS-backed cells98.2%Benchmark cells with a BLS OES leg

Accuracy Degradation Factors

Estimates are least reliable in the following conditions. Researchers should apply appropriate filters or widen confidence intervals when working with these subsets:

  • Very high compensation (above $300K/year)
  • Hourly and gig-economy work
  • International postings (salary benchmarks are US-only)
  • Older data vintages (salary norms shift over time)

Work Mode Detection: Remote, Hybrid, On-site

A classifier reads each posting and assigns Remote, Hybrid, or On-Site, with 93%primary accuracy on a held-out gold set. It never abstains, so a NULL means no classifier scored the posting, not an implicit on-site role. Its confidence score is the model's raw score: useful for ranking and filtering, not a success rate.

"Hybrid" is poorly standardized: it can mean one day a week in the office or four, depending on the employer. Where a hybrid posting states its in-office days, they ship in hybridDaysPerWeek, and an explicit five-days-in-office mandate is flagged separately.

MetricValueContext
Primary accuracy93%Held-out gold set
Postings with a work mode99.9%NULL only on postings no classifier scored
AbstentionNoneEvery scored posting receives Remote, Hybrid, or On-Site
Confidence scoreRawUseful for ranking and filtering, not a success rate

Named Entity Recognition

Skills extraction runs in two steps. First, a fast, dictionary-based keyword processor scans every job description with high recall to tag candidate entities. Second, a title-aware relevance filter reviews each role and skill pairing and removes off-role mentions, for example "Ruby on Rails" stamped on a delivery driver posting. The filter is tuned conservatively to keep legitimate skills: an independent multi-judge audit confirms about 99% of removed tags are genuine false positives, and fewer than 1 in 500 legitimate skills are affected. In production this concentrates the average from roughly 10 to roughly 8 skill tags per posting, focusing each record on the skills that actually define the role. Multi-valued attributes are represented as arrays; when no signal is present, fields return empty arrays rather than null.

~99%

of removed tags confirmed false positives in an independent multi-judge audit

<1 in 500

legitimate skills affected by the title-aware relevance filter

~10 → ~8

average skill tags per posting after off-role mentions are removed

Technical Skills

Programming languages, tools, frameworks

Soft Skills

Leadership, communication, problem-solving

Certifications

Professional licenses and credentials (PMP, AWS, CPA)

Qualifications

Education requirements, years of experience

Benefits

Health insurance, 401(k), PTO mentions

Contact Signals

Emails and phone numbers extracted from posting text

Work Requirements

Visa sponsorship indicators, citizenship or residency requirements, security clearance

Work Conditions

Travel requirements, shift work and shift type mentions, language requirements

Role Characteristics

Urgent hiring cues, manager or lead indicators, number of openings, team size, start date

Skills Taxonomy: Scale

Extraction matches against Canaria's 43,000+ entry skills and credentials taxonomy: 39,000+ skills, 3,300+ certifications, 220+ professional licenses, 250+ soft skills, and language requirements, reached through more than 120,000 surface forms (aliases, abbreviations, and spelling variants). At least one skill is extracted on 89.8% of delivered postings. The taxonomy powers extraction and is not sold on its own; the product built on it is Job Skills Data.

Semantic Deduplication

A central challenge in online job postings data is severe duplication, driven by reposting behavior, aggregator syndication, and minor text changes over time. Canaria resolves it in two stages. Exact matching on URL and the source's job key turns 10B+ observations into 1B+ unique postings. A second, semantic stage then groups the same job posted on several boards, or collected again over time, into one duplicate group. The semantic stage deletes nothing: isUniqueJobPosting marks each group's representative (the earliest seen), duplicateGroupID names the group, and representativeJobID joins a duplicate back to its representative. Summing isUniqueJobPosting gives unique volume and counting rows gives gross volume, so each analysis picks the grain it needs.

Exact Keys

The first pass removes repeat observations of the same posting by URL and the source's own job key. This is what turns the observation stream into unique postings.

Description Similarity

Embedding similarity on descriptions catches the same job posted again with reworded or reformatted text.

MinHash / Jaccard

Handles variations in company names like "Macy's," "Macys Inc," or "Macy's LLC."

Location

Postings are compared only when their location is resolved precisely enough to tell two places apart. A posting that cannot be tested is kept and counted as unique.

Time Window

A relisting first seen more than 60 days after its group's latest sighting starts a new unique posting; shorter gaps keep chaining into the same group.

Graph-based Transitive Matching

Captures transitive similarity (A ≈ B, B ≈ C → unify A, B, C) so a job seen on several boards resolves to one group.

Location Parsing

Raw location strings from job postings are parsed into structured geographic components and geocoded to latitude and longitude. A point is published only at zip or city precision: when only a state or country can be resolved, the coordinates are NULL rather than a coarse centroid. 97.8% of delivered postings carry a geocoded point, and 89.1% carry a metro area (OMB CBSA) code.

Measured Fill Rates

Fill rates are measured on the delivered Job Postings files (October 2026, catalog 8.4) and published per field in each product's data dictionary. They vary by source and by year, so check the schema for the fields an analysis depends on.

FieldPostings populated
Normalized title99.8%
SOC code99.9%
Seniority99.9%
Employment type99.9%
Work mode (remote / hybrid / on-site)99.9%
Normalized company name98.2%
Geocoded location (zip or city precision)97.8%
Metro area (CBSA)89.1%
Annual salary (stated or estimated)90.5%
At least one extracted skill89.8%

Company Enrichment

Canaria maintains a company database of 29M company profiles, 25.4M of them with firmographics (industry, employee size and count, headquarters, revenue band, and founding year) and 8.7M with job postings. Each posting carries a canonical normalizedCompanyName, the join key to the company files, on 98.2%of delivered postings. It is resolved from the company name on the posting; where a posting names no employer, it can be recovered from the posting's company profile link, or carried from its duplicate group when every named member names the same company.

Data Quality Monitoring

Canaria maintains comprehensive monitoring and quality assurance systems across all ingestion and processing stages. A pipeline component tracks field-level completeness for every scraped batch. Non-empty field counts are aggregated by source and time period, enabling detection of source degradation or scraping failures. Completeness metrics inform prioritization of enrichment efforts. Source URLs are monitored for consecutive failure streaks and flagged for manual review when potentially defunct. Monitoring dashboards visualize trends and trigger alerts for anomalies including queue depth spikes, error rate thresholds, and crawler failures.

Research Applications

The Canaria dataset enables empirical analysis across multiple labor economics domains.

Wage Dynamics & Compensation

  • Geographic wage differentials (metro vs rural, coastal vs inland)
  • Salary transparency effects (pre/post state disclosure laws)
  • Compensation trends by occupation, industry, and company size

Labor Demand & Skill Requirements

  • Skill demand evolution over time (e.g., AI/ML skill growth)
  • Education-occupation mismatch (degree requirements vs SOC norms)
  • Occupational mobility pathways via skill overlap analysis

Remote Work & Geographic Flexibility

  • Remote work adoption rates by occupation and industry
  • Remote work wage premia and discounts
  • Geographic concentration trends and migration patterns

Firm-level Hiring Behavior

  • Posting persistence: how long each posting stayed listed, from the opt-in open-duration module (a listing measure, not a record of when a hire was made)
  • Hiring velocity as a leading economic indicator
  • Expansion signals via new-role detection and headcount proxies

Want the full technical details?

Download the complete methodology document or request a sample dataset to evaluate the data firsthand.