Workybara crawler

Workybara is a job search site. Its crawler collects public job postings from employer career pages and job boards so job seekers can search them in one place.

How to recognize it

Most requests from the crawler carry Workybara/1.0 in the User-Agent. Career pages and applicant tracking system pages are fetched with:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Workybara/1.0; +https://workybara.ai/bot) Chrome/128.0.0.0 Safari/537.36

Job board APIs and robots.txt are fetched with:

Mozilla/5.0 (compatible; Workybara/1.0; +https://workybara.ai/bot)

Pages that only load in a full browser are opened in headless Chrome, which sends a standard Chrome User-Agent with Workybara/1.0 added:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Workybara/1.0

Earlier versions of the crawler identified themselves as WorkybaraBot.

robots.txt

Career-site crawls check robots.txt and follow the group for Workybara, or the * group when there is none. The older WorkybaraBot token works too. To keep the crawler off a whole site:

User-agent: Workybara
Disallow: /

There are exceptions. Public job-listing pages of a small number of large employers are read even where robots.txt disallows them, and listings read through an applicant tracking system's public job API are not gated by the employer's robots.txt. Email is the reliable way to opt a site out completely.

Contact

To report a problem with the crawler or to have a site excluded, email contact@workybara.ai. Include the domain, and if you can, the time and IP address of a request you saw.