PriceDog

PriceDogBot

PriceDogBot is the crawler of pricedog.app, a price tracker and comparison service for shoppers. It is in a small test phase: we read a limited sample of public product pages, not whole shops.

How it identifies itself

User-Agent: PriceDogBot/1.0 (+https://pricedog.app/bot)

What we fetch, and why

Public product pages, one page at a time, to read the price, availability, product name, brand, product codes (EAN/MPN) and the page address, so visitors can compare prices and see price history. We do not download, host, store or link product photos, and we do not copy descriptions, reviews or any personal data, we do not log in, and we do not use a browser or any technique to get around blocks.

How politely

  • We read /robots.txt first and obey Disallow and Crawl-delay. If it cannot be read (server error, unreadable file), we do not crawl that site.
  • At least 5 seconds between two requests to a shop, and longer when the shop's robots.txt asks for it (10 seconds for some shops).
  • At most 300 product pages and 350 requests per run, at most 400 requests per day per shop, once a night. We do not crawl whole shops.
  • We stop at once on HTTP 403, 429 or 503 or an anti-bot page, and back off for hours.

Where your shop appears

In our comparison catalog at pricedog.app/katalog (not indexed by search engines, not in our sitemap): your shop's name, the product name, the price and availability we read, the time of the last check, and a "To the shop" link to your own product page. No photos, no descriptions, no ratings. We do not earn a commission on these links today; if that changes, the page says so next to the link.

Stručně česky

PriceDogBot čte jen veřejné produktové stránky, nejméně 5 sekund mezi dvěma požadavky, nejvýše 300 stránek a 400 požadavků denně na obchod, jednou za noc, a řídí se vaším robots.txt. Váš obchod se zobrazuje v katalogu pricedog.app/katalog (bez indexace vyhledávači): název obchodu, název produktu, cena, dostupnost a odkaz do vašeho obchodu, bez fotografií a popisů. Odmítnutí: pricedog.app/pro-eshopy nebo jan@pricedog.app; obchod přestaneme načítat a skryjeme jeho nabídky do 7 dnů od potvrzené žádosti.

Opt out

Shop owners can ask us not to crawl, list or link their shop at pricedog.app/pro-eshopy or by email to jan@pricedog.app. We stop crawling and hide the shop's offers within 7 days of a confirmed request, immediately when confirmed from a mailbox on the shop's own domain. Disallowing PriceDogBot in robots.txt also works.