Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Elastic stack from 7.17 to 8 #216

Merged
merged 25 commits into from
Jul 30, 2024
Merged

Conversation

makelicious
Copy link
Collaborator

@makelicious makelicious commented Jul 17, 2024

In order to fix the issue opencrvs/opencrvs-core#7149 we needed to upgrade to next major version of elastic stack

Support matrix: https://www.elastic.co/support/matrix
Stack upgrade guide: https://www.elastic.co/guide/en/elastic-stack/8.14/upgrading-elastic-stack.html#prepare-to-upgrade

Changes

Needs core changes from opencrvs/opencrvs-core#7389

@makelicious
Copy link
Collaborator Author

makelicious commented Jul 17, 2024

On upgrading:

Support matrix: https://www.elastic.co/support/matrix
Stack upgrade guide: https://www.elastic.co/guide/en/elastic-stack/8.14/upgrading-elastic-stack.html#prepare-to-upgrade

Upgrade Elastic stack from 7.17.x to 8.14.2

  1. Log in to kibana environment as superuser (elastic by default)
  2. Take backup
  3. Go to 'Upgrade assistant' (https://kibana.your-environment-opencrvs.org/app/management/stack/upgrade_assistant/overview)
Screenshot 2024-07-11 at 14 45 58 4. When upgrading from 1.5. you should see following issues: Screenshot 2024-07-11 at 14 58 36 Screenshot 2024-07-11 at 14 58 44

At least critical issues should be fixed before moving forward. This PR addresses them.

On upgrade order:

  • Each of the components should be on 7.17.x. on before upgrading. (apm-server is not)

  • Elastalert 2.3.0 does not support elasticsearch 8. It needs to be upgraded.

  • Verify we can upgrade an environment with one go or create a plan to divide it in steps

  • Verify upgrade works for new environments

@Zangetsu101
Copy link
Collaborator

So for existing systems we can't just do a new deploy and expect everything to "just work" is that it? Instead we need to do it step by step, for example apm-server would first need to be on 7.17.x and the critical issues need to be resolved.

How about doing it in 2 phases? For example we could do a 1.5.x & 1.6.x release with the apm-server upgrade & critical issues resolved and then on 1.7.0 upgrade elasticsearch to 8?

@makelicious
Copy link
Collaborator Author

I tested this against existing environment and the upgrade with one go seemed to work. Thanks to docker swarm, I think. There might be some error state for a minute until every container has updated, unless we can define the order in which the containers are updated, or they can be swapped 'all at once'.

I'm now filling dev environment with test data and check for regression next

@makelicious makelicious marked this pull request as ready for review July 30, 2024 05:38
@makelicious makelicious merged commit 05fbd63 into develop Jul 30, 2024
1 check passed
@makelicious makelicious deleted the chore/upgrade-to-elastic-8 branch July 30, 2024 06:59
@makelicious makelicious restored the chore/upgrade-to-elastic-8 branch July 30, 2024 07:00
@makelicious makelicious added this to the v1.6.0 milestone Jul 30, 2024
makelicious added a commit that referenced this pull request Jul 30, 2024
🍒 Merge changes from PR #216 to release-v1.6.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants