Skip to content
This repository has been archived by the owner on Apr 22, 2024. It is now read-only.

Create job manager, enable append tables, add metadata tables (#14, #33) #62

Merged

Conversation

ssciolla
Copy link
Contributor

@ssciolla ssciolla commented Apr 6, 2020

This PR aims to enable multiple jobs (beyond the first, inventory.py) by creating a run_jobs.py as the Docker start point. Loading and management of environment variables has also been extracted to its own file (environ.py) so it could be loaded as needed by other parts of the application. run_jobs.py also records job history by updating two new database tables, job_run and data_source_status. Records should not be dropped from these tables, so I also introduced an "append table" concept, allowing some tables to be preserved by specifying them in env.json. Some additional details are provided below in the task list. This PR aims to resolve issues #14 and #33.

  • Move inventory.py and gql_queries.py into canvas and rename to course_inventory.
  • Enable what I'm calling "Append Tables" by updating db/db_creator.py.
  • Extract env.json loading from various places and create environ.py
  • Implement Job and JobManager classes in a new run_jobs.py file
  • Create job_run and data_source_status tables for storing job metadata
  • Add metadata validation and creation to run_jobs.py, including creation of ValidJobName and ValidDataSourceName enums.
  • Update course_inventory.py/inventory.py to use APPEND_TABLE_NAMES and environ.py, and to return data source dictionaries that job_run uses to create data_source_status database records.
  • Update README with a generalized overview, new variables, and a section describing how to implement a new job.

@ssciolla ssciolla added 📈 enhancement New feature or request config change introduced changes to the configuration file labels Apr 6, 2020
@ssciolla ssciolla marked this pull request as ready for review April 6, 2020 19:13
@ssciolla
Copy link
Contributor Author

ssciolla commented Apr 6, 2020

I've opened this for review. I will work on adding a section to the README providing instructions on implementing a new job.

run_jobs.py Show resolved Hide resolved
environ.py Outdated Show resolved Hide resolved
create_db.py Show resolved Hide resolved
environ.py Show resolved Hide resolved
Copy link
Member

@jonespm jonespm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good! We could improve the sleep loop and the append tables, but we could also fix those in a future issue too so approving.

Copy link
Member

@lsloan lsloan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've suggested some minor changes and asked questions, but overall I like it. Looking forward to using some of these features with the MiVideo code.

README.md Show resolved Hide resolved
course_inventory/inventory.py Show resolved Hide resolved
course_inventory/inventory.py Show resolved Hide resolved
db/migrations/0009.add_meta_tables.py Show resolved Hide resolved
@ssciolla ssciolla merged commit f2b15d7 into tl-its-umich-edu:master Apr 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
config change introduced changes to the configuration file 📈 enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants