nsetscrapers

A quick-to-implement Python package focused on the Extract aspect of ETL. This project spawns from the extraction, transformation, and loading processes of COVID-19 SCRAPERS and EAGLE-I.

etl

This is the python package that includes modules for a standard ETL project. To install this package, run ./setup.py install to install system-wide, or use the user installation by running ./setup.py install --user, which is the preferred method.

System Environment Variables

This package is designed for use within a Docker-based ecosystem, but can be setup on any machine with the proper environmental variables defined. Defaults are included for those that are not set, but it is best practice to define your own in an .env file and reference that in your docker-compose.yaml file. Below is a table of environmental variables used by the `scrapers``` package: