Competitive Intelligence

Daymon International's category and sourcing teams make pricing and assortment decisions with retailers and suppliers. For that, they need to know what competing retailers charge and stock, across many markets, every day.

A system that collects prices and assortments from 80+ retailers in 27 countries on 6 continents every day. It cleans the data and matches products across retailers and over time by brand and attributes, so the same item can be compared. I built it from the first scraper in 2021 and still own it.

4M+ products tracked, with 5 years of history, delivered to dashboards every day. The dashboards were built in Tableau and are now moving to the company's in-house analytics platform.

Python and Scrapy for collection, Pandas for cleaning and matching, and SQL Server for storage. Proxy handling keeps collection working on sites with bot protection.

Python · Scrapy · bot-protection & proxy handling · Pandas · SQL Server