GovHubBot
GovHubBot is the crawler behind GovHub's bid inbox. It reads the public bid and solicitation listings that government agencies post on their own websites, so small businesses can find opportunities that fit them, and it links each listing back to the agency's page.
How to recognize it
Every request carries this user agent:
Mozilla/5.0 (compatible; GovHubBot/1.0; +https://www.govhub.online/bot/)
Its direct requests are also signed with
Web Bot Auth
(HTTP Message Signatures). Each one carries Signature-Agent,
Signature-Input and Signature headers, and the
public key that verifies them is published at
https://www.govhub.online/.well-known/http-message-signatures-directory. A request that claims to be
GovHubBot without a valid signature is not ours.
What it reads
- Public listing pages and feeds of open bids: state purchasing offices, cities, counties, school and special districts, transit and port authorities, and SAM.gov.
- Only pages anyone can open without signing in. It never logs in, never creates accounts, and uses only the search or paging controls a listing page itself offers to the public.
- The daily crawl records each listing's title, number, dates and a link to the agency's page, and links to bid documents rather than downloading them. When a GovHub user asks to analyze one listing, GovHub fetches that listing's documents once, with the same user agent.
How it behaves
- robots.txt first. It reads robots.txt before anything else on a site and follows the group for
GovHubBot, or the*group if there is none, includingCrawl-delay. - Once a day. It visits each listing page about once a day, mostly early in the morning, US Eastern time. Most sites see two or three requests a day: robots.txt and the listing page, plus a few pages for a long list.
- A refusal is final. If a site answers with an error, a challenge page or a captcha, GovHubBot records that and stops. It does not retry through proxies, switch user agents, or use a browser to get past the check.
- Terms are read by a person. Before a site is added, someone at GovHub reviews its robots.txt and terms of use. Where a site's terms rule out AI processing, its listings are stored without any.
- No model training. GovHub does not train AI models on the pages GovHubBot reads.
How to limit or stop it
To keep GovHubBot off your whole site, add this to your robots.txt:
User-agent: GovHubBot
Disallow: /
You can also disallow single paths, or set a Crawl-delay in
seconds. Changes take effect on its next daily visit. If you would rather
tell us directly, or want a listing corrected or removed, email
[email protected] and we will
act on it promptly.
Allowing it through bot protection
If your site sits behind bot protection that blocks GovHubBot and you
would like your bids listed, you can allow requests whose signature
verifies against the key directory above, or match on the
GovHubBot user agent. Email
[email protected] if you need
anything else from us to do that.