crawlPro
An asynchronous crawler that discovers product URLs across e-commerce sites, with per-domain rate limiting and retries.
crawlPro finds product pages across multiple e-commerce sites. It is built on aiohttp so requests never block each other, and it limits the request rate per domain so it stays polite.
Domain-specific URL patterns decide what counts as a product page. Responses are streamed, URLs are deduplicated in memory, compressed responses are handled, and failed requests retry with a fallback path. Everything is configured from a YAML file.