NielsenIQ Product Insights and GS1 Company Matching¶
Research documentation for linking NielsenIQ (Kilts Center for Marketing) Retail Scanner Movement Data with GS1 Global Company Prefixes (GCP).
Interactive Data Explorer¶
Browse all 7,921 matched products with instant search, category filtering (GS1 Matched vs. Private Label), and sorting by 2010 revenue or volume:
Open Product Crosswalk (Full 7,921 Products)
Project Objectives¶
- Firm-Level Product Ownership: Map granular 12-digit Universal Product Codes (UPC) from retail scanner sales data to parent manufacturing firms using the GS1 Company Prefix registry.
- Scalable Architecture via DuckDB: Process tens of millions of weekly store transaction records in seconds using modern columnar storage (Parquet) and vectorized SQL execution (DuckDB) rather than uncompressed tabular formats.
- Academic Alignment: Implement the matching methodology established in leading finance and marketing literature:
- Fracassi, Previtero, and Sheen (The Journal of Finance, 2022)
- Servaes and Tamayo (Working Paper)
- Kilts Center Retail Scanner Dataset Manuals (2006-2020 and 2024 editions)
System Architecture¶
flowchart TD
A["Raw Nielsen Sales Movement Data<br/>(111.5M rows across 18 TSV files)"] -->|DuckDB Vectorized Distinct| B["Unique Sales UPCs<br/>(7,921 Products)"]
C["GS1 Company Prefix Registry<br/>(681,101 Prefixes)"] -->|Convert to Parquet / DuckDB| D["GS1 Prefix Lookup Table<br/>(Lengths 6 to 11 digits)"]
B --> E{"Longest-Prefix Matching Engine"}
D --> E
E -->|Exact GCP Match| F["Matched Branded Products<br/>(6,703 UPCs - 84.6%)"]
E -->|Prefix 8/9/02/04 Detection| G["Nielsen Private-Label Masked<br/>(854 UPCs - 10.8%)"]
E -->|Unregistered / Foreign| H["Unmatched Branded<br/>(364 UPCs - 4.6%)"]
F --> I["Final Crosswalk<br/>output/upc_firm_lookup.csv<br/>output/crosswalk.html"]
G --> I
H --> I
Documentation Contents¶
- Methodology and Matching: Barcode structures, GS1 prefix allocation tiers, and the longest-prefix algorithm.
- Data Dictionary: Comprehensive schemas for scanner, GS1, and crosswalk tables.
- Results and Empirical Findings: Summary statistics, market share rankings, and revenue distributions.
- Core Assumptions: Econometric and prefix-free assumptions underpinning the match.
- Limitations and Edge Cases: M&A brand transfers, prefix fallback risks, and foreign registrations.
- WRDS Integration: Protocols for connecting GS1 firms with Compustat, CRSP, and master product tables.
- Manuals and Academic Citations: Access to Kilts manuals, reference docs, and academic citations.