I built a GitHub Actions workflow and Shiny app that automates daily job scraping at midnight GMT (6PM CST), gathering postings from Glassdoor, LinkedIn, Indeed, ZipRecruiter, and Google Jobs based on predefined keywords. The script compiles these listings into a centralized table, appending new data while preserving historical results. Designed to save time, this solution streamlines the job search process by eliminating the need to manually check multiple job boards, providing an efficient and organized way to track relevant opportunities.
Every midnight GMT (6PM CST), Glassdoor, LinkedIn, Indeed, ZipRecruiter, and Google Jobs are searched for jobs in Winnipeg, MB, that were posted with the last 24 hours. Currently, the following search terms are used:
- Laboratory technician
- Research assistant
- Quality control analyst
- Biological technician
- Microbiology technician
- Data analyst
- Program coordinator
- Project manager
- R programming language
If you use this app and wish to have additional search terms included, please send me an email here.
At midnight GMT, a GitHub Actions workflow is triggered via a cron job, which runs the job_scraper.py script. The script scrapes job listings posted in the last 24 hours from various boards and appends results to the jobs_combined.csv file. The data is then read on load by the Shiny app which involves some data clean-up and displays all data in a table with various filtering options.
To avoid being blocked by each site, the script only runs once per day and uses proxies.