What it does
Data Enricher helps webshops complete their product information. When a shop asks about a product (an EAN or a product name), we search the web and read a few public product pages to find the facts, such as specifications and dimensions. We do not copy your texts: the AI writes new texts from the facts.
How it behaves
- It follows robots.txt, including Disallow rules, Crawl-delay and Request-rate.
- It only reads public product pages that a search engine shows for one product, never whole websites, and it never logs in, fills in forms or places orders.
- It reads at most a few pages per site at a time, with a fixed limit per site per minute.
- A site that answers with 403, 429 or a bot check is left alone for hours, and after a few times for good.
- Products found earlier are taken from our own cache, so the same page is not read again and again.
How to recognise it
The crawler always says who it is and never poses as a browser.
- User-Agent:
DataEnricher/1.0 (+https://data-enricher.com/en/bot) - From:
info@mark-app.com - IP addresses:
188.245.8.38,2a01:4f8:c014:25c4::1 - Reverse DNS:
crawler.data-enricher.com
Limit or block it
To block the crawler for your whole site, add this to your robots.txt:
User-agent: DataEnricher
Disallow: /
Or ask for more time between two pages (in seconds):
User-agent: DataEnricher
Crawl-delay: 10
Questions or problems
Something not right, or do you want us to stay away from (part of) your site? Mail us and we will sort it out quickly: info@mark-app.com