VoyterraBot

VoyterraBot is the crawler for Voyterra, a travel guide for people with dietary constraints such as coeliac disease.

The bot reads the public pages of food venues. It visits only pages that public sources point to: OpenStreetMap entries, food blogs, and coeliac society directories. We read a venue's own page so our guide can describe the venue in its own words.

The bot obeys robots.txt. Before we use a site as a source, the bot reads its robots.txt and its terms page. Sometimes it must fetch the home page first, to find where the terms are. After that, each visit fetches one robots.txt file and one page. It follows no links. It does not come back for months.

To block it, add these lines to your robots.txt:

User-agent: VoyterraBot
Disallow: /