Support for various metadata store backends [need feedback and review] #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In current implementation, the ML metadata (MLMD) metadata store was tightly coupled with CMF. Our experiments and standalone proof-of-concept implementations have demonstrated that it is possible to use other platforms, such as MLflow to store pipeline metadata (though, not as efficiently as with MLMD store, but in certain cases when a team or organization has already setup the ML metadata management service, they can start tracking their pipeline metadata using that existing service). The idea is relatively straightforward - take MLMD related collection of functions and wrap them into a class (MlmdStore). Then functions that are used by `Cmf` become public interface for MlmdStore, and respectively, become API that Cmf requires from arbitrary metadata backend in the future. This commit moves existing MLMD functionality into `cmflib.metadata.mlmd_store` module.
sergey-serebryakov
requested review from
annmary-roy,
atripathy86,
martin-foltin and
suparna-bhattacharya
|
This PR should also address CMF running in the debug mode, where the backend can be a console , and we do a dry run and no hard changes like commits and branches . |
Fixing conflicts in `cmflib/cmf.py`.
sergey-serebryakov
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In current implementation, the ML metadata (MLMD) metadata store was tightly coupled with CMF. Our experiments and standalone proof-of-concept implementations have demonstrated that it is possible to use other platforms, such as MLflow to store pipeline metadata (though, not as efficiently as with MLMD store, but in certain cases when a team or organization has already setup the ML metadata management service, they can start tracking their pipeline metadata using that existing service).
The idea is relatively straightforward - take MLMD related collection of functions and wrap them into a class (MlmdStore). Then functions that are used by
Cmfbecome public interface for MlmdStore, and respectively, become API that Cmf requires from arbitrary metadata backend in the future.This commit moves existing MLMD functionality into
cmflib.metadata.mlmd_storemodule.