Set compatibility config to NONE for protobuf subjects #48
Conversation
|
@kishansairam9 is working on a generic solution. |
ravisingal
changed the title
Adding each proto is painful, there has been a meeting where laxman and team decided on having a cron job to set for all protobufs. No need to override each proto individually. I think this is the way to go forward, my old PR is abandoned |
compatibility-changer/Dockerfile
Outdated
| # update packages | ||
| RUN apt update && apt upgrade -y | ||
|
|
||
| RUN apt install -y curl && \ |
compatibility-changer/Dockerfile
Outdated
| # setup virtual environment | ||
| RUN python -m venv /opt/venv | ||
|
|
||
| # create directory and copy pinot-data-metrics directory |
| @@ -0,0 +1,6 @@ | |||
| certifi==2024.8.30 | |||
There was a problem hiding this comment.
do we need all of them explicitly? should we omit transitive dependencies and let pip figure it out?
| data: | ||
| application.conf: | | ||
| { | ||
| "defaultSchemaType": {{ .Values.compatibilityChanger.config.defaultSchemaType }} |
There was a problem hiding this comment.
schemaType -> compatibilityType
| logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') | ||
|
|
||
| # Read helm values | ||
| host = os.getenv('SCHEMA_REGISTRY_HOST', 'localhost') |
There was a problem hiding this comment.
make this configurable
| for subject in tqdm(subjects): | ||
| versions = get_subject_versions(subject) | ||
| if not versions: | ||
| continue |
There was a problem hiding this comment.
log a warn when no version available.
|
|
||
| def main(): | ||
| subjects = get_subjects() | ||
| logging.info(f"All subjects: {subjects}") |
There was a problem hiding this comment.
debug log or just log number of subjects
| response = requests.get(url) | ||
| return response.json() | ||
|
|
||
| def get_subject_version_details(subject, version): |
There was a problem hiding this comment.
get_subject_version_details -> get_schema_version
|
|
||
| # Select the first version | ||
| first_version = versions[0] | ||
| details = get_subject_version_details(subject, first_version) |
| charset-normalizer==3.3.2 | ||
| idna==3.10 | ||
| requests==2.32.3 | ||
| tqdm==4.66.5 |
There was a problem hiding this comment.
Comment from @ravisingal : Progress bar may not be required for cron.
.gitignore
Outdated
| .idea | ||
| *__pycache__* | ||
| *cc* |
There was a problem hiding this comment.
this looks too greedy and i think this can omit necessary files as well.
laxmanchekka
changed the title
|
@ravisingal : can you please check this once from helm, k8s and git actions perspective as well |
| hypertraceDocker { | ||
| defaultImage { | ||
| imageName.set("schema-compatibility") | ||
| } |
There was a problem hiding this comment.
add dockerFile.set(file("Dockerfile"))
|
|
||
| hypertraceDocker { | ||
| defaultImage { | ||
| imageName.set("schema-compatibility") |
Co-authored-by: Ravi Singal <[email protected]>
Co-authored-by: Ravi Singal <[email protected]>
Co-authored-by: Ravi Singal <[email protected]>
Co-authored-by: Ravi Singal <[email protected]>
Co-authored-by: Ravi Singal <[email protected]>
No description provided.