This repository has been archived by the owner on Apr 22, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6
Create job manager, enable append tables, add metadata tables (#14, #33) #62
Merged
ssciolla
merged 25 commits into
tl-its-umich-edu:master
from
ssciolla:issue-33-create-job-manager
Apr 7, 2020
Merged
Create job manager, enable append tables, add metadata tables (#14, #33) #62
ssciolla
merged 25 commits into
tl-its-umich-edu:master
from
ssciolla:issue-33-create-job-manager
Apr 7, 2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to support metadata
ssciolla
added
📈 enhancement
New feature or request
config change
introduced changes to the configuration file
labels
Apr 6, 2020
I've opened this for review. I will work on adding a section to the README providing instructions on implementing a new job. |
ssciolla
commented
Apr 6, 2020
…mentation section
ssciolla
commented
Apr 7, 2020
jonespm
reviewed
Apr 7, 2020
jonespm
reviewed
Apr 7, 2020
jonespm
approved these changes
Apr 7, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks good! We could improve the sleep loop and the append tables, but we could also fix those in a future issue too so approving.
lsloan
approved these changes
Apr 7, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've suggested some minor changes and asked questions, but overall I like it. Looking forward to using some of these features with the MiVideo code.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to enable multiple jobs (beyond the first,
inventory.py
) by creating arun_jobs.py
as the Docker start point. Loading and management of environment variables has also been extracted to its own file (environ.py
) so it could be loaded as needed by other parts of the application.run_jobs.py
also records job history by updating two new database tables,job_run
anddata_source_status
. Records should not be dropped from these tables, so I also introduced an "append table" concept, allowing some tables to be preserved by specifying them inenv.json
. Some additional details are provided below in the task list. This PR aims to resolve issues #14 and #33.inventory.py
andgql_queries.py
intocanvas
and rename tocourse_inventory
.db/db_creator.py
.env.json
loading from various places and createenviron.py
Job
andJobManager
classes in a newrun_jobs.py
filejob_run
anddata_source_status
tables for storing job metadatarun_jobs.py
, including creation ofValidJobName
andValidDataSourceName
enums.course_inventory.py/inventory.py
to useAPPEND_TABLE_NAMES
andenviron.py
, and to return data source dictionaries thatjob_run
uses to createdata_source_status
database records.