Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump running Drucker service configurations for backup #7

Merged
merged 40 commits into from
Aug 28, 2018

Conversation

keigohtr
Copy link
Member

@keigohtr keigohtr commented Aug 7, 2018

What is this PR for?

For backup.
If Kubernetes crushes completely, we will lost all existing Drucker service configurations.
For a service recovery, this PR dumps Drucker service configurations.

This PR includes

  • Add kube.datadir in settings.yml
  • Dump Kubernetes services under DIR_KUBE_CONFIG directory

What type of PR is it?

Feature

What is the issue?

N/A

How should this be tested?

Do "Sync" on Drucker-dashboard

@yoquankara
Copy link
Member

I have 2 comments:

  1. What were criteria for those selected configurations to be dumped? (I'm not familiar with k8s that much, so happy to be enlightened)

  2. Unit test for dump_drucker_on_kubernetes should be added if it is not so costly :-)

@keigohtr
Copy link
Member Author

keigohtr commented Aug 8, 2018

Thank you for the comments.

What were criteria for those selected configurations to be dumped?

There are 4 types of configuration this PR dumps. Deployment, Service and Ingress is the highest priority to dump since they are necessary for serving Drucker service. Since AutoScaler manages an auto-scaling policy for Drucker service, it is not necessary.

Unit test for dump_drucker_on_kubernetes should be added

Sure. As you know, @jkw552403 san are creating unit test. I will add the test after merging his commit. (So this PR will be merged after merging his commit)

@yoquankara
Copy link
Member

Thanks for your explanation! So I got that with those 4 fundamental configurations we can recreate service in case of emergency.

Looking forward to unit test.

….com/drucker/drucker-dashboard into feature/dump-kubernetes-yaml"

This reverts commit 0e60cac, reversing
changes made to 0acfe39.
@keigohtr
Copy link
Member Author

@yoquankara
I have rebased to the latest master and add unit-test here. Please check it again!

:return:
"""
mobj = db.session.query(Model).filter(
Model.application_id==application_id,Model.model_id==model_id).one()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think exception NoResultFound should be handled or logged in case none is found.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exceptions are caught by api.errorhandler.
So we can handle and log them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I still prefer explicit error handling at the right place because it is under our control and enable us to fix quickly with more specific log, instead of stack trace in default error handler.

If we decide to follow above practice, review of whole code base is necessary though. Maybe it should be a TODO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloud you create Issue about this?
Error handling is one of the most important things. I agree with your opinion!

At this time, this PR is not for refactoring. Thank you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#31 created.

Model.application_id==application_id,Model.model_id==model_id).one()
model_path = mobj.model_path
sobj = db.session.query(Service).filter(
Service.application_id == application_id, Service.service_id==service_id).one()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

sobj.model_id = model_id
db.session.flush()

aobj = db.session.query(Application).filter(Application.application_id == application_id).one()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@@ -747,6 +763,145 @@ def create_or_update_drucker_on_kubernetes(
body=v2_beta1_horizontal_pod_autoscaler
)
"""
return service_name

def switch_drucker_service_model_assignment(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if unit test for this method could also be added

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think so.
But this PR is not for modifying the unit test. We will create another one for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's do so.

Copy link
Member

@yoquankara yoquankara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@keigohtr keigohtr merged commit ce0fdb5 into master Aug 28, 2018
@keigohtr keigohtr deleted the feature/dump-kubernetes-yaml branch August 28, 2018 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants