crawler-cluster Distributed, Fault-Tolerant Web Crawling. Multi-process, multiple workers Client process queues tasks in Redis. Worker nodes pull tasks from Redis, execute task, and store results in Redis. Client process pulls results from Redis. Pros: Worker nodes can run on any machine. Add or remove worker nodes at runtime without disrupting the system. Achieves fault-tolerance through process isolation and monitoring. Workers are ran as systemd services, where each service is the smallest possible processing unit (either a single browser with a […]

california-coronavirus-scrapers An experiment in open-sourcing the web scrapers that feed the Los Angeles Times’ California coronavirus tracker. The scrapers are written using Python and Jupyter notebooks, scheduled and run via GitHub Actions and then archived using git. Installation Clone the repository and install the Python dependencies. pipenv install Run all of the scraper commands. make Run one of the scraper commands. make -f vaccine-doses-on-hand/Makefile GitHub https://github.com/datadesk/california-coronavirus-scrapers