Purpose of check_additional_stores?

[janosh]

i was expecting jobflow_remote.remote.data.check_additional_stores to return ["output_blobs"] if that collection doesn't exist yet but instead it returns a long list of keys Job keys. wondering if that is expected behavior and if so, what the idea behind this is.

from jobflow_remote import JobController
from jobflow_remote.jobs.state import JobState
from jobflow_remote.remote.data import check_additional_stores

job_controller = JobController.from_project_name(proj_name)
job = job_coll.find_one({Key.state: JobState.COMPLETED.value})
check_additional_stores(job, job_controller.jobstore)
>>> ['_id',
 'remote',
 'stored_data',
 'updated_on',
 'parents',
 'priority',
 'job',
 'run_dir',
 'end_time',
 'resources',
 'error',
 'previous_state',
 'lock_id',
 'worker',
 'lock_time',
 'created_on',
 'exec_config',
 'db_id',
 'start_time',
 'state']

[gpetretto]

Hi @janosh,
the purpose of the check_additional_stores is to allow the user to know at submission time if the jobstore has been properly configure in case the job requires an additional store. For example, in the case of VASP atomate2 jobs a data additional store should be present in the jobstore defined in the project configuration file. Without this check, if the data additional store was not defined, you would get an error at the end of the Job. In this way you the user is warned before the job is added to the DB.
So this function does not check the DB collections, but rather the configuration file.

So, for an atomate2 VASP Job, if the jobstore in the project configuration does not contain the data additional store the output of the function call would be ["data"]

The main reason why you are getting this weird output is that the function expects a Job instance or its dict representation, while you are passing the dictionary representation of a JobDoc, which is the object stored in the jobs collection. This includes both the serielized Job as well as all the running information:

jobflow-remote/src/jobflow_remote/jobs/data.py

Line 222 in 4870f02

class JobDoc(BaseModel):

.

In your case you should get the proper behaviour by modifying the call to the function like this:

check_additional_stores(job["job"], job_controller.jobstore)

[janosh]

thanks @gpetretto for the quick reply! maybe makes sense to link this discussion from the check_additional_stores doc string for future users. or copy over this paragraph:

the purpose of the check_additional_stores is to allow the user to know at submission time if the jobstore has been properly configure in case the job requires an additional store. For example, in the case of VASP atomate2 jobs a data additional store should be present in the jobstore defined in the project configuration file. Without this check, if the data additional store was not defined, you would get an error at the end of the Job. In this way you the user is warned before the job is added to the DB.
So this function does not check the DB collections, but rather the configuration file.