Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): option to init via username password, add lint for smoke-test #9675

Merged
merged 22 commits into from
Jan 31, 2024

Conversation

anshbansal
Copy link
Collaborator

@anshbansal anshbansal commented Jan 19, 2024

Changes

  • Add option --use-password in datahub init to allow easier generation of tokens for admin for initial setup
  • Add some error handling to take care of some common mistakes when datahub init is run in configuring gms url leading to connection issues
  • Extract out helper method to get frontend session spread out in smoke tests
  • Extract out code for managing config file
  • Add linting to smoke-test directory
  • Ensure CI fails on lint failure in smoke-test

Follow up work todo in separate PR

  • Follow up work to ensure these 4 test cases/functionalities are working 0834a74
  • Ensure every folder in smoke-test has __init__ otherwise those tests won't work

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata smoke_test Contains changes related to smoke tests labels Jan 19, 2024
@anshbansal anshbansal changed the title feat(cli): option to get token via username pass in init feat(cli): option to init via username pass in init Jan 19, 2024
@anshbansal anshbansal changed the title feat(cli): option to init via username pass in init feat(cli): option to init via username password Jan 19, 2024
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly looks good, just a few comments on code style / docs

thanks for doing this :)

metadata-ingestion/src/datahub/cli/cli_utils.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/cli/cli_utils.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/cli/cli_utils.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/entrypoints.py Show resolved Hide resolved
metadata-ingestion/src/datahub/entrypoints.py Show resolved Hide resolved
@anshbansal anshbansal force-pushed the ab-add-option-get-token-password branch from bc69b11 to 75adfc9 Compare January 30, 2024 11:47
@anshbansal anshbansal marked this pull request as ready for review January 30, 2024 13:21
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly looks good

with:
distribution: "zulu"
java-version: 17
- name: Run lint on smoke test
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be in the setup job, since it'll block the rest of the docker builds

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is actually the intention. Lint should cause rest of CI to simply not run. This causes anyone making changes in smoke tests to fail fast. Otherwise CI will wait for docker builds which takes a while before failing.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I agree with that - this will add a few minutes of wall clock time to every docker-unified run, which is already the slowest part of our CI. Lint issues are going to be relatively rare in smoke test, so I don't think it makes sense to slow down our CI significantly to avoid a small number of wasted docker builds

Instead the lint should be a separate job so that the docker builds can run in parallel, and we can use something like this https://github.com/orgs/community/discussions/38361 to cancel the other workflows

from datahub.cli.env_utils import get_boolean_env_variable

__help__ = (
"For helper methods to contain manipulation of the config file in local system."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

module level docstrings should go at the top and be in """, don't need to use the __help__ thing

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to do this in follow up PR.

@anshbansal anshbansal merged commit 3f9490d into master Jan 31, 2024
59 checks passed
@anshbansal anshbansal deleted the ab-add-option-get-token-password branch January 31, 2024 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata smoke_test Contains changes related to smoke tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants