-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PythonMutator: propagate source locations #1783
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
The line protocol in the locations file looks fine.
def4744
to
43ce278
Compare
96a6cef
to
d9bf157
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you also merge main to make sure it passes with the latest linter settings?
// - resources.jobs.job_0.tasks[0].task_key is located at job_0.py:10:5 | ||
// | ||
// - resources.jobs.job_0.tasks[0].email_notifications is located at job_0.py:3:5, | ||
// because we use the location of the job as the most precise approximation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(indent mismatch between above and here)
The entries included (as an example of locations.json
) made me think it should match the data structure below. Please include a reference to [pythonLocationEntry]
to make it clear this is not the case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a reference, and I had to reformat doc slightly, or "go fmt" was always resulting into a bad indent
} | ||
|
||
newLocations = append(newLocations, location) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This logic is hard to parse in combination with the append above.
If I understand correctly, you want to want the output to:
- Have the Python location as the first element
- Filter out the locations with
generatedFileName
for the filename
It would be clearer if newLocations
was initialized by assignment and if the loop got a comment saying that it only appends locations that are relevant (i.e. not the ones with the virtual file path).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've extracted the code into a separate function, and added elaborate comment
_, err = paths.VisitJobPaths(generated, func(p dyn.Path, kind paths.PathKind, v dyn.Value) (dyn.Value, error) { | ||
putPythonLocation(locations, p, v.Location()) | ||
return v, nil | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can remove this condition once:
- Variable interpolation runs before running the Python mutators
- Path normalization (to make all relative paths relative to the sync root) runs before running the Python mutators
Then this mutator must make sure all paths are relative to the sync root.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you include a test to confirm that the value for a field that is a path has the right location?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a test, and included the note you wrote about into the code, so we don't forget when this code is safe to cleanup.
4ec0f52
to
a462c9f
Compare
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @kanterov !
Changes
Add a mechanism to load Python source locations in the Python mutator. Previously, locations pointed to generated YAML. Now, they point to Python sources instead. Python process outputs "locations.json" containing locations of bundle paths, examples:
Such locations form a tree, and we assign locations of the closest ancestor to each
dyn.Value
based on its path. For example,resources.jobs.job_0.tasks[0].task_key
is located atjob_0.py:10:5
andresources.jobs.job_0.tasks[0].email_notifications
is located atjob_0.py:3:5
, because we use the location of the job as the most precise approximation.This feature is only enabled if
experimental/python
is used.Note: for now, we don't update locations with relative paths, because it has a side effect in changing how these paths are resolved
Example
Tests
Unit tests and manually