Data Collection and Enrichment Methodology
How we collect, standardize, enrich, and deduplicate job market data at scale.
Platform Overview
| Observations | 10B+ scrape observations of job postings |
| Unique Job Postings | 1B+, unique by URL and source job key |
| Fields per Record | 100+ |
| Historical Coverage | 2022-present |
| Acquisition & Delivery | Collected daily to hourly per source; job postings delivered at least daily |
| Primary Sources | Indeed, LinkedIn Jobs, 200,000+ employer ATS portals (Greenhouse, Lever, Workday, iCIMS) |
| Geographic Scope | US-primary plus 240+ other countries and territories |
| Delivery Formats | Parquet (default) or CSV, via Amazon S3, Google Cloud Storage, Azure Blob Storage, Google Drive or SFTP |
Data Collection Philosophy
Canaria treats job postings as observable signals of employer labor demand, not as direct measures of hiring outcomes. A posted job reflects stated intent to hire; it does not confirm a hire occurred, a position was filled, or that a single headcount was added. Researchers should apply this distinction when using posting counts as economic indicators.
The platform focuses on making what is observable reliable, standardized, and analytically usable, while explicitly preserving uncertainty and avoiding inference beyond the data's scope. Raw job postings are ingested through a distributed scraping infrastructure spanning Indeed, LinkedIn Jobs, and 200,000+ employer career portals (Greenhouse, Lever, Workday, iCIMS, and others). Coverage is US-primary plus 240+ other countries and territories, with data beginning in 2022. Sources are collected daily to hourly (near-real-time acquisition); job postings are delivered at least daily, and company, salary, and skills products on the cadence each product states.
Each posting passes through a multi-stage enrichment pipeline combining deterministic parsing, machine learning classification, and job-level entity resolution. All raw, parsed, and enriched fields are retained for transparency and auditability. When a field is absent or unparseable, the record carries a NULL rather than a synthetic fill-in; the only model estimates that fill a gap are labeled as estimates, such as predicted salary, which the salarySource column marks on every row. Fill rates are measured on every delivered field. The field priority rule throughout the pipeline is: parsed structured data takes precedence over scraped text, which takes precedence over calculated or model-predicted values.
Pipeline Architecture
Every job posting passes through an 8-stage pipeline: ingestion, deduplication, NLP classification, location parsing, salary normalization, skills extraction, title normalization, and company enrichment.
Raw Scraped Data
10B+ observations of 1B+ unique postings (by URL and source job key)
Deduplication
Near-duplicates grouped across sources and over time; every posting kept and flagged
Location Parsing
City / state / country / zip / coordinates
Salary Normalization
Annual USD min / avg / max
Description Extraction
Skills, benefits, clearance, contact
ML Classification
SOC, seniority, employment, remote
Title Normalization
Standardized titles + confidence scores
Company Enrichment
29M companies, industry, size, HQ
Final Merged Delivery
1B+ postings with their duplicate-group flags, 100+ fields, all enrichment stages joined with field priority logic
Machine Learning Enrichment
Our enrichment pipeline is a microservice-based NLP system that transforms raw job text into structured, analytics-ready fields. Each model operates independently, allowing separate scaling and iteration.
Title Normalization
Canonicalizes noisy titles (abbreviations, typos) to standardized forms. Outputs confidence scores so researchers can filter or weight results by precision.
Accuracy: 84.58% top-5, 69.94% top-1 across about 100,000 roles
Example: SW Eng II → Software Engineer
SOC Classification
Assigns 6-digit Standard Occupational Classification codes using title + description context. Enables occupational analysis aligned with the BLS 2018 SOC taxonomy.
Accuracy: 93% top-5, 73% top-1 across 867 codes
Example: 15-1252 (Software Developers)
Seniority Detection
Classifies career level into eight bands (intern, entry, mid, supervisor, senior, staff/principal, director, executive) from the title and description, plus a career track: individual contributor, manager, or mixed.
Accuracy: 94% primary accuracy; 99.94% populated, the classifier never declines
Example: Senior
Employment Type
Classifies each posting's employment type from its title and description: full-time, part-time, contract, temporary, internship, per diem, and other arrangements.
Accuracy: 95% primary accuracy
Example: Full-time
Remote Work Status
Classifies each posting as Remote, Hybrid, or On-Site from the posting text, with in-office days per week where a hybrid posting states them.
Accuracy: 93% primary accuracy
Example: Hybrid
Salary Prediction
Where a posting states no pay, a Canaria model estimates annual USD minimum, average, and maximum. Salary benchmarks fuse three sources: posted pay, employee-reported pay, and BLS OES.
Scale: 42K+ benchmark cells, 95% CI per cell, 98.2% BLS-backed
Example: $95,000 – $125,000
Skills Extraction (NER)
Extracts 39,000+ skills, 3,300+ certifications, and 250+ soft skills using dictionary matching plus a title-aware relevance filter that removes off-role matches.
Coverage: At least one skill on 89.8% of delivered postings
Example: ["Python", "AWS", "Docker"]
NAICS Classification
NAICS-2022 industry classification, resolved per employer from company name and job text signals and carried onto job, company, and place records (a 6-digit industry code on 88.77% of postings and a sector on 90.65%). Enables industry-level labor market analysis.
Accuracy: 98% top-5, 89% top-1 across 1,012 codes
Example: 511210 (Software Publishers)
Classifier Accuracy
Each enrichment classifier publishes accuracy on a held-out evaluation set, plus a per-row confidence score so consumers can filter on quality. Class counts vary from 3 work-mode classes to about 100,000 canonical titles, and the numbers below should be read against that backdrop.
SOC Classification
BLS 2018, 867 6-digit codes
73%
top-1
93%
top-5
Measured on a 2,040-row held-out gold set. The classifier reads the full description as well as the title, which matters most for generic titles such as "Manager" or "Analyst."
NAICS Classification
NAICS 2022, 1,012 6-digit codes
89%
top-1
98%
top-5
Measured on a 1,010-row held-out gold set across 1,012 industry codes. NAICS classification uses company name plus description text and is the basis for industry-level aggregations across the dataset.
Job Attributes
3 LoRA heads on a shared BGE-m3 backbone
94%
seniority
95%
employment
93%
work mode
Primary-class accuracy on held-out gold sets of 138 to 274 rows per task. Seniority spans eight bands, employment type covers ten classes, and work mode classifies remote, hybrid, and on-site. None of the three abstains: every posting they score receives a value.
Title Normalization
Arctic Embed L v2.0 + LoRA + FAISS, about 100,000 canonical roles
69.94%
top-1
84.58%
top-5
Measured on an 11,848-row production-frequency benchmark, held out from training. Read it against the class count: 69.94% top-1 is a choice among about 100,000 canonical roles. On a stricter set never seen during fine-tuning, top-1 is 52.34% and top-5 is 74.20%. Confidence scores ship with every row for downstream filtering.
A confidence score ships per row (nlpSocConfidence, nlpSeniorityConfidence, etc.). Occupation, industry and normalized-title scores are calibrated to the probability that the value is correct; seniority, employment-type and work-mode scores are the model's raw score, meaningful for ranking and filtering rather than as a success rate.
SOC Classification: Accuracy and Coverage
Canaria assigns 6-digit Standard Occupational Classification (SOC) codes using both the job title and the full description text, because a generic title such as "Manager" or "Analyst" says little on its own. Accuracy is 93% top-5 and 73% top-1 across all 867 detailed occupation codes, measured on a held-out gold set. The taxonomy is aligned with the BLS 2018 SOC standard, enabling direct comparison with government labor statistics.
A SOC code ships on 99.9% of delivered postings, with a confidence score calibrated to the probability that the code is correct, so a threshold on it reads as an expected hit rate.
Known Classification Challenges
| Failure mode | Example |
|---|---|
| Ambiguous titles | "Manager", "Associate", "Specialist" without description context |
| New-economy roles | "Prompt Engineer" and "Growth Hacker" have no clean mapping in BLS taxonomy |
| Multi-function roles | "Software Engineer / Data Scientist" requires a primary SOC assignment |
Salary Prediction: Model and Coverage
An annual salary ships on 90.5% of delivered postings. 42.6% of those salaries come from the posting itself, either its text or the job board's salary field; the rest are estimates, from the job board or from Canaria's model. The salarySource column on every row says which, so stated and estimated pay can always be separated.
Where a posting states no pay, Canaria's model estimates minimum, average, and maximum annual USD. All salary values are annualized and converted to USD at the posting date's exchange rate; for stated pay, the period the employer used (hour, week, month, year) ships in salaryPayPeriod. Salary benchmarks fuse three sources (posted pay, employee-reported pay, and BLS OES) and report 42K+ cells, each with a 95% confidence interval.
Coverage and Benchmarks
| Metric | Value | Context |
|---|---|---|
| Postings with an annual salary | 90.5% | Stated or estimated, delivered Job Postings files |
| Share of those stated in the posting | 42.6% | Posting text or the job board's salary field; the rest are estimates |
| Benchmark cells | 42K+ | 42,002 occupation × state plus 867 national |
| Confidence interval per cell | 95% | Reported alongside every benchmark estimate |
| BLS-backed cells | 98.2% | Benchmark cells with a BLS OES leg |
Accuracy Degradation Factors
Estimates are least reliable in the following conditions. Researchers should apply appropriate filters or widen confidence intervals when working with these subsets:
- Very high compensation (above $300K/year)
- Hourly and gig-economy work
- International postings (salary benchmarks are US-only)
- Older data vintages (salary norms shift over time)
Work Mode Detection: Remote, Hybrid, On-site
A classifier reads each posting and assigns Remote, Hybrid, or On-Site, with 93%primary accuracy on a held-out gold set. It never abstains, so a NULL means no classifier scored the posting, not an implicit on-site role. Its confidence score is the model's raw score: useful for ranking and filtering, not a success rate.
"Hybrid" is poorly standardized: it can mean one day a week in the office or four, depending on the employer. Where a hybrid posting states its in-office days, they ship in hybridDaysPerWeek, and an explicit five-days-in-office mandate is flagged separately.
| Metric | Value | Context |
|---|---|---|
| Primary accuracy | 93% | Held-out gold set |
| Postings with a work mode | 99.9% | NULL only on postings no classifier scored |
| Abstention | None | Every scored posting receives Remote, Hybrid, or On-Site |
| Confidence score | Raw | Useful for ranking and filtering, not a success rate |
Named Entity Recognition
Skills extraction runs in two steps. First, a fast, dictionary-based keyword processor scans every job description with high recall to tag candidate entities. Second, a title-aware relevance filter reviews each role and skill pairing and removes off-role mentions, for example "Ruby on Rails" stamped on a delivery driver posting. The filter is tuned conservatively to keep legitimate skills: an independent multi-judge audit confirms about 99% of removed tags are genuine false positives, and fewer than 1 in 500 legitimate skills are affected. In production this concentrates the average from roughly 10 to roughly 8 skill tags per posting, focusing each record on the skills that actually define the role. Multi-valued attributes are represented as arrays; when no signal is present, fields return empty arrays rather than null.
~99%
of removed tags confirmed false positives in an independent multi-judge audit
<1 in 500
legitimate skills affected by the title-aware relevance filter
~10 → ~8
average skill tags per posting after off-role mentions are removed
Technical Skills
Programming languages, tools, frameworks
Soft Skills
Leadership, communication, problem-solving
Certifications
Professional licenses and credentials (PMP, AWS, CPA)
Qualifications
Education requirements, years of experience
Benefits
Health insurance, 401(k), PTO mentions
Contact Signals
Emails and phone numbers extracted from posting text
Work Requirements
Visa sponsorship indicators, citizenship or residency requirements, security clearance
Work Conditions
Travel requirements, shift work and shift type mentions, language requirements
Role Characteristics
Urgent hiring cues, manager or lead indicators, number of openings, team size, start date
Skills Taxonomy: Scale
Extraction matches against Canaria's 43,000+ entry skills and credentials taxonomy: 39,000+ skills, 3,300+ certifications, 220+ professional licenses, 250+ soft skills, and language requirements, reached through more than 120,000 surface forms (aliases, abbreviations, and spelling variants). At least one skill is extracted on 89.8% of delivered postings. The taxonomy powers extraction and is not sold on its own; the product built on it is Job Skills Data.
Semantic Deduplication
A central challenge in online job postings data is severe duplication, driven by reposting behavior, aggregator syndication, and minor text changes over time. Canaria resolves it in two stages. Exact matching on URL and the source's job key turns 10B+ observations into 1B+ unique postings. A second, semantic stage then groups the same job posted on several boards, or collected again over time, into one duplicate group. The semantic stage deletes nothing: isUniqueJobPosting marks each group's representative (the earliest seen), duplicateGroupID names the group, and representativeJobID joins a duplicate back to its representative. Summing isUniqueJobPosting gives unique volume and counting rows gives gross volume, so each analysis picks the grain it needs.
Exact Keys
The first pass removes repeat observations of the same posting by URL and the source's own job key. This is what turns the observation stream into unique postings.
Description Similarity
Embedding similarity on descriptions catches the same job posted again with reworded or reformatted text.
MinHash / Jaccard
Handles variations in company names like "Macy's," "Macys Inc," or "Macy's LLC."
Location
Postings are compared only when their location is resolved precisely enough to tell two places apart. A posting that cannot be tested is kept and counted as unique.
Time Window
A relisting first seen more than 60 days after its group's latest sighting starts a new unique posting; shorter gaps keep chaining into the same group.
Graph-based Transitive Matching
Captures transitive similarity (A ≈ B, B ≈ C → unify A, B, C) so a job seen on several boards resolves to one group.
Location Parsing
Raw location strings from job postings are parsed into structured geographic components and geocoded to latitude and longitude. A point is published only at zip or city precision: when only a state or country can be resolved, the coordinates are NULL rather than a coarse centroid. 97.8% of delivered postings carry a geocoded point, and 89.1% carry a metro area (OMB CBSA) code.
Measured Fill Rates
Fill rates are measured on the delivered Job Postings files (October 2026, catalog 8.4) and published per field in each product's data dictionary. They vary by source and by year, so check the schema for the fields an analysis depends on.
| Field | Postings populated |
|---|---|
| Normalized title | 99.8% |
| SOC code | 99.9% |
| Seniority | 99.9% |
| Employment type | 99.9% |
| Work mode (remote / hybrid / on-site) | 99.9% |
| Normalized company name | 98.2% |
| Geocoded location (zip or city precision) | 97.8% |
| Metro area (CBSA) | 89.1% |
| Annual salary (stated or estimated) | 90.5% |
| At least one extracted skill | 89.8% |
Company Enrichment
Canaria maintains a company database of 29M company profiles, 25.4M of them with firmographics (industry, employee size and count, headquarters, revenue band, and founding year) and 8.7M with job postings. Each posting carries a canonical normalizedCompanyName, the join key to the company files, on 98.2%of delivered postings. It is resolved from the company name on the posting; where a posting names no employer, it can be recovered from the posting's company profile link, or carried from its duplicate group when every named member names the same company.
Data Quality Monitoring
Canaria maintains comprehensive monitoring and quality assurance systems across all ingestion and processing stages. A pipeline component tracks field-level completeness for every scraped batch. Non-empty field counts are aggregated by source and time period, enabling detection of source degradation or scraping failures. Completeness metrics inform prioritization of enrichment efforts. Source URLs are monitored for consecutive failure streaks and flagged for manual review when potentially defunct. Monitoring dashboards visualize trends and trigger alerts for anomalies including queue depth spikes, error rate thresholds, and crawler failures.
Research Applications
The Canaria dataset enables empirical analysis across multiple labor economics domains.
Wage Dynamics & Compensation
- Geographic wage differentials (metro vs rural, coastal vs inland)
- Salary transparency effects (pre/post state disclosure laws)
- Compensation trends by occupation, industry, and company size
Labor Demand & Skill Requirements
- Skill demand evolution over time (e.g., AI/ML skill growth)
- Education-occupation mismatch (degree requirements vs SOC norms)
- Occupational mobility pathways via skill overlap analysis
Remote Work & Geographic Flexibility
- Remote work adoption rates by occupation and industry
- Remote work wage premia and discounts
- Geographic concentration trends and migration patterns
Firm-level Hiring Behavior
- Posting persistence: how long each posting stayed listed, from the opt-in open-duration module (a listing measure, not a record of when a hire was made)
- Hiring velocity as a leading economic indicator
- Expansion signals via new-role detection and headcount proxies
Explore the Data
The methodology above produces the numbers behind every free report on this site. Browse the full schema for all 100+ fields, review our data quality benchmarks, check the glossary for term definitions, or read engineering deep-dives on dedup and enrichment. Or jump straight into a topic:
Data Insights hub
All free labor market reports
Software Engineer Salaries
Salary ranges by seniority
AI & ML Job Market
Posting volume, skills, salaries
Most In-Demand Skills
Top skills by posting frequency
Remote Work Trends
Remote, hybrid, on-site share over time
Top Hiring Companies
Companies posting the most jobs
Hiring Trends by Industry
Monthly posting volume by industry
Want the full technical details?
Download the complete methodology document or request a sample dataset to evaluate the data firsthand.