Skip to content

NielsenIQ Product Insights and GS1 Company Matching

Research documentation for linking NielsenIQ (Kilts Center for Marketing) Retail Scanner Movement Data with GS1 Global Company Prefixes (GCP).


Interactive Data Explorer

Browse all 7,921 matched products with instant search, category filtering (GS1 Matched vs. Private Label), and sorting by 2010 revenue or volume:

Open Product Crosswalk (Full 7,921 Products)


Project Objectives

  1. Firm-Level Product Ownership: Map granular 12-digit Universal Product Codes (UPC) from retail scanner sales data to parent manufacturing firms using the GS1 Company Prefix registry.
  2. Scalable Architecture via DuckDB: Process tens of millions of weekly store transaction records in seconds using modern columnar storage (Parquet) and vectorized SQL execution (DuckDB) rather than uncompressed tabular formats.
  3. Academic Alignment: Implement the matching methodology established in leading finance and marketing literature:
  4. Fracassi, Previtero, and Sheen (The Journal of Finance, 2022)
  5. Servaes and Tamayo (Working Paper)
  6. Kilts Center Retail Scanner Dataset Manuals (2006-2020 and 2024 editions)

System Architecture

flowchart TD
    A["Raw Nielsen Sales Movement Data<br/>(111.5M rows across 18 TSV files)"] -->|DuckDB Vectorized Distinct| B["Unique Sales UPCs<br/>(7,921 Products)"]
    C["GS1 Company Prefix Registry<br/>(681,101 Prefixes)"] -->|Convert to Parquet / DuckDB| D["GS1 Prefix Lookup Table<br/>(Lengths 6 to 11 digits)"]
    B --> E{"Longest-Prefix Matching Engine"}
    D --> E
    E -->|Exact GCP Match| F["Matched Branded Products<br/>(6,703 UPCs - 84.6%)"]
    E -->|Prefix 8/9/02/04 Detection| G["Nielsen Private-Label Masked<br/>(854 UPCs - 10.8%)"]
    E -->|Unregistered / Foreign| H["Unmatched Branded<br/>(364 UPCs - 4.6%)"]
    F --> I["Final Crosswalk<br/>output/upc_firm_lookup.csv<br/>output/crosswalk.html"]
    G --> I
    H --> I

Documentation Contents