Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Amazon linux 2 #21

Merged
merged 6 commits into from
Dec 22, 2021
Merged

Support Amazon linux 2 #21

merged 6 commits into from
Dec 22, 2021

Conversation

saravanan30erd
Copy link
Collaborator

@saravanan30erd saravanan30erd commented Dec 8, 2021 •
edited
Loading

Issues Resolved

#17

@peterzhuamazon Tested the playbook with latest amazon linux 2, its compatible for almost all tasks except one which is fixed now.

Logs

PLAY [opensearch installation & configuration] ************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************
ok: [os3]
ok: [os5]
ok: [os4]
ok: [os2]
ok: [os1]

TASK [centos7/opensearch : hostname] **********************************************************************************************************************************
changed: [os3]
changed: [os5]
changed: [os2]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Install the libselinux-python for Amazon Linux 2] ******************************************************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os1]
changed: [os2]

TASK [centos7/opensearch : Disable the selinux] ***********************************************************************************************************************
ok: [os3]
ok: [os4]
ok: [os2]
ok: [os5]
ok: [os1]

TASK [centos7/opensearch : Hosts | populate inventory into hosts file] ************************************************************************************************
changed: [os3]
changed: [os4]
changed: [os5]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Set vm.max_map_count in sysctl.conf] *******************************************************************************************************
changed: [os3]
changed: [os5]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Set open files limit in sysctl.conf] *******************************************************************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : OpenSearch Install | Download opensearch 1.2.0] ********************************************************************************************
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]
changed: [os5]

TASK [centos7/opensearch : OpenSearch Install | Create opensearch user] ***********************************************************************************************
changed: [os5]
changed: [os3]
changed: [os1]
changed: [os4]
changed: [os2]

TASK [centos7/opensearch : OpenSearch Install | Create home directory] ************************************************************************************************
changed: [os3]
changed: [os5]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : OpenSearch Install | Extract the tar file] *************************************************************************************************
[WARNING]: Consider using the unarchive module rather than running tar. If you need to use command because unarchive is insufficient you can add warn=False to this
command task or set command_warnings=False in ansible.cfg to get rid of this message.

changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : OpenSearch Install | Copy Configuration File] **********************************************************************************************
changed: [os3]
changed: [os5]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : OpenSearch Install | Copy jvm.options File for Instance] ***********************************************************************************
changed: [os3]
changed: [os5]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : OpenSearch Install | create systemd service] ***********************************************************************************************
changed: [os3]
changed: [os5]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Create local temporary directory for certificates generation] **********************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Download certificates generation tool] *********************************************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Extract the certificates generation tool] ******************************************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Make the executable file] **********************************************************************************
ok: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Prepare the certificates generation template file] *********************************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Generate the node & admin certificates in local] ***********************************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Security Plugin configuration | Copy the node & admin certificates to opensearch nodes] ****************************************************
changed: [os3] => (item=root-ca.pem)
changed: [os5] => (item=root-ca.pem)
changed: [os4] => (item=root-ca.pem)
changed: [os3] => (item=root-ca.key)
changed: [os1] => (item=root-ca.pem)
changed: [os5] => (item=root-ca.key)
changed: [os2] => (item=root-ca.pem)
changed: [os3] => (item=os3.key)
changed: [os5] => (item=os5.key)
changed: [os4] => (item=root-ca.key)
changed: [os3] => (item=os3.pem)
changed: [os2] => (item=root-ca.key)
changed: [os1] => (item=root-ca.key)
changed: [os5] => (item=os5.pem)
changed: [os3] => (item=os3_http.key)
changed: [os4] => (item=os4.key)
changed: [os5] => (item=os5_http.key)
changed: [os3] => (item=os3_http.pem)
changed: [os2] => (item=os2.key)
changed: [os1] => (item=os1.key)
changed: [os5] => (item=os5_http.pem)
changed: [os3] => (item=admin.key)
changed: [os4] => (item=os4.pem)
changed: [os5] => (item=admin.key)
changed: [os2] => (item=os2.pem)
changed: [os3] => (item=admin.pem)
changed: [os1] => (item=os1.pem)
changed: [os5] => (item=admin.pem)
changed: [os4] => (item=os4_http.key)
changed: [os2] => (item=os2_http.key)
changed: [os4] => (item=os4_http.pem)
changed: [os1] => (item=os1_http.key)
changed: [os2] => (item=os2_http.pem)
changed: [os4] => (item=admin.key)
changed: [os1] => (item=os1_http.pem)
changed: [os2] => (item=admin.key)
changed: [os4] => (item=admin.pem)
changed: [os1] => (item=admin.key)
changed: [os2] => (item=admin.pem)
changed: [os1] => (item=admin.pem)

TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 1 to cluster] *********************************************************
changed: [os5]
changed: [os3]
changed: [os2]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 2 to cluster] *********************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Prepare the opensearch security configuration file] ********************************************************
[WARNING]: Consider using the replace, lineinfile or template module rather than running sed. If you need to use command because replace, lineinfile or template is
insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.

changed: [os3]
changed: [os5]
changed: [os2]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Set the file ownerships] ***********************************************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Set the folder permission] *********************************************************************************
changed: [os5]
changed: [os3]
changed: [os2]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Restart opensearch with security configuration] ************************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Pause for 3 seconds to provide sometime for OpenSearch start] ******************************************************************************
Pausing for 3 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Copy the opensearch security internal users template] ******************************************************
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Set the Admin user password] *******************************************************************************
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Set the kibanaserver user pasword] *************************************************************************
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch] ****************************************************
changed: [os1]

TASK [centos7/opensearch : Security Plugin configuration | Cleanup local temporary directory] *************************************************************************
changed: [os1 -> localhost]

TASK [centos7/opensearch : Remove cluster.initial_master_nodes setting from configuration] **************************************************************************
changed: [os3]
changed: [os5]
changed: [os2]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Make sure opensearch is started] ***********************************************************************************************************
ok: [os3]
ok: [os5]
ok: [os4]
ok: [os2]
ok: [os1]

TASK [centos7/opensearch : Get all the installed ES plugins] **********************************************************************************************************
changed: [os5]
changed: [os3]
changed: [os4]
changed: [os2]
changed: [os1]

TASK [centos7/opensearch : Show all the installed ES plugins] *********************************************************************************************************
ok: [os1] => {
"msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql"
}
ok: [os2] => {
"msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql"
}
ok: [os3] => {
"msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql"
}
ok: [os4] => {
"msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql"
}
ok: [os5] => {
"msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql"
}

TASK [centos7/opensearch : Wait for opensearch to startup] ************************************************************************************************************
ok: [os5]
ok: [os3]
ok: [os4]
ok: [os2]
ok: [os1]

TASK [centos7/opensearch : Check the opensearch status] ***************************************************************************************************************
[WARNING]: Consider using the get_url or uri module rather than running curl. If you need to use command because get_url or uri is insufficient you can add
warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.

changed: [os3]
changed: [os2]
changed: [os5]
changed: [os4]
changed: [os1]

TASK [centos7/opensearch : Show the opensearch status] ****************************************************************************************************************
ok: [os1] => {
"failed_when_result": false,
"msg": {
"active_primary_shards": 2,
"active_shards": 7,
"active_shards_percent_as_number": 100.0,
"cluster_name": "development-cluster",
"delayed_unassigned_shards": 0,
"discovered_master": true,
"initializing_shards": 0,
"number_of_data_nodes": 5,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 5,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
}
ok: [os2] => {
"failed_when_result": false,
"msg": {
"active_primary_shards": 2,
"active_shards": 7,
"active_shards_percent_as_number": 100.0,
"cluster_name": "development-cluster",
"delayed_unassigned_shards": 0,
"discovered_master": true,
"initializing_shards": 0,
"number_of_data_nodes": 5,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 5,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
}
ok: [os3] => {
"failed_when_result": false,
"msg": {
"active_primary_shards": 2,
"active_shards": 7,
"active_shards_percent_as_number": 100.0,
"cluster_name": "development-cluster",
"delayed_unassigned_shards": 0,
"discovered_master": true,
"initializing_shards": 0,
"number_of_data_nodes": 5,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 5,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
}
ok: [os4] => {
"failed_when_result": false,
"msg": {
"active_primary_shards": 2,
"active_shards": 7,
"active_shards_percent_as_number": 100.0,
"cluster_name": "development-cluster",
"delayed_unassigned_shards": 0,
"discovered_master": true,
"initializing_shards": 0,
"number_of_data_nodes": 5,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 5,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
}
ok: [os5] => {
"failed_when_result": false,
"msg": {
"active_primary_shards": 2,
"active_shards": 7,
"active_shards_percent_as_number": 100.0,
"cluster_name": "development-cluster",
"delayed_unassigned_shards": 0,
"discovered_master": true,
"initializing_shards": 0,
"number_of_data_nodes": 5,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 5,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
}

TASK [centos7/opensearch : Verify the roles of opensearch cluster nodes] **********************************************************************************************
changed: [os1]

TASK [centos7/opensearch : Show the roles of opensearch cluster nodes] ************************************************************************************************
ok: [os1] => {
"msg": "ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name\n10.0.10.96 22 96 14 0.23 0.15 0.06 di - os5\n10.0.10.254 17 95 14 0.28 0.19 0.08 di - os4\n10.0.10.243 20 95 25 0.40 0.25 0.10 dm - os2\n10.0.10.153 14 93 25 0.68 0.32 0.13 dm - os1\n10.0.10.79 6 96 15 0.23 0.16 0.07 dm * os3"
}

PLAY [opensearch dashboards installation & configuration] *************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************
ok: [dashboards1]

TASK [centos7/dashboards : hostname] **********************************************************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Install the libselinux-python for Amazon Linux 2] ******************************************************************************************
ok: [dashboards1]

TASK [centos7/dashboards : Disable the selinux] ***********************************************************************************************************************
ok: [dashboards1]

TASK [centos7/dashboards : Hosts | populate inventory into hosts file] ************************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Set open files limit in sysctl.conf] *******************************************************************************************************
ok: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | Download opensearch dashbaord 1.2.0] **********************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | Create opensearch user] ***********************************************************************************************
ok: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | Create home directory] ************************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | Extract the tar file] *************************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | Copy Configuration File] **********************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Dashboards Install | create systemd service] ***********************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Make sure opensearch dashboards is started] ************************************************************************************************
changed: [dashboards1]

TASK [centos7/dashboards : Get all the installed dashboards plugins] **************************************************************************************************
[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo

changed: [dashboards1]

TASK [centos7/dashboards : Show all the installed dashboards plugins] *************************************************************************************************
ok: [dashboards1] => {
"msg": "[email protected]\[email protected]\[email protected]\[email protected]\[email protected]\[email protected]\[email protected]\[email protected]"
}

PLAY RECAP ************************************************************************************************************************************************************
dashboards1 : ok=15 changed=9 unreachable=0 failed=0
os1 : ok=42 changed=33 unreachable=0 failed=0
os2 : ok=28 changed=22 unreachable=0 failed=0
os3 : ok=28 changed=22 unreachable=0 failed=0
os4 : ok=28 changed=22 unreachable=0 failed=0
os5 : ok=28 changed=22 unreachable=0 failed=0

@saravanan30erd
add changes to support amz linux
7595c06 
Signed-off-by: saravanan palanisamy <[email protected]>
peterzhuamazon
peterzhuamazon reviewed Dec 9, 2021
View reviewed changes
Copy link
Member

@peterzhuamazon peterzhuamazon left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the issues I commented there are more things I want to add:

  1. The folder name is still centos7, can we change to something else or remove the folder?
  2. Can you also update the README to include AL2? As of now only centos7 is mentioned.
  3. I recently noticed that a lot of people still using Oracle Linux which is also RedHat based, if that can be test and support at some time it would be great.

Overall great.

Thanks.

opensearch.yml Show resolved Hide resolved
roles/centos7/dashboards/tasks/main.yml Outdated Show resolved Hide resolved
@peterzhuamazon peterzhuamazon linked an issue Dec 9, 2021 that may be closed by this pull request
[Enhancement]Test runs on Amazon Linux 2 Machines #17
Closed
@saravanan30erd
change folder name
93184df 
Signed-off-by: saravanan palanisamy <[email protected]>
@saravanan30erd
Copy link
Collaborator Author

@peterzhuamazon

  1. The folder name is still centos7, can we change to something else or remove the folder?

Changed to linux.

  1. Can you also update the README to include AL2? As of now only centos7 is mentioned.

Done

  1. I recently noticed that a lot of people still using Oracle Linux which is also RedHat based, if that can be test and support at some time it would be great.

Sure, created an issue #22

@peterzhuamazon
Copy link
Member

Seems like there are some conflicts after my merge of #19.
I will test this once the conflicts are resolved.

Thanks.

@saravanan30erd
Copy link
Collaborator Author

@peterzhuamazon merge conflicts are resolved. Please continue with test.

@peterzhuamazon
Copy link
Member

@saravanan30erd we just released 1.2.2 and lets hold this for a bit so we can safely cut 1.2.2 tag.
Will merge this next week when I get into the office.
Same for Ubuntu.

Thanks.

@peterzhuamazon peterzhuamazon mentioned this pull request Dec 16, 2021
Merged
@saravanan30erd
Copy link
Collaborator Author

@peterzhuamazon We need to merge this PR before any other PR's. Because this one have major change (folder changed from centos7 into linux) so it keep creating merge conflicts whenever new PR is merged and thus creating duplication work.

Could you please prioritize this one?
May be we can have a call today to close this and ubuntu PR.

@peterzhuamazon
Copy link
Member

Hi @saravanan30erd I am releasing 1.2.3 now so very little time on my side.
Can we have a discussion for this later?
Meanwhile, @TheAlgo if you can take a look it would be great.

Thanks.

@peterzhuamazon
Copy link
Member

Update: I will update this PR today with more comments of testing.
Able to find a small time window to do some tests.
Sorry about delays.

@peterzhuamazon
Copy link
Member 

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'selinux'
fatal: [os2]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (libselinux-python) on os2's Python /usr/bin/python3.7. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

@saravanan30erd
Copy link
Collaborator Author

@peterzhuamazon but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter may be its using wrong interpreter python 3 because in my case, its used python2.7 and I didn't face this issue. There was lot of issues due to different python versions since centos7/RHEL7 uses python2.7 only and AL2 have both python2.7 & 3, ubuntu only having python 3. so we need to stick with some stable ansible version which supports all. Can we have a call to check this?

@peterzhuamazon
Copy link
Member

@peterzhuamazon but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter may be its using wrong interpreter python 3 because in my case, its used python2.7 and I didn't face this issue. There was lot of issues due to different python versions since centos7/RHEL7 uses python2.7 only and AL2 have both python2.7 & 3, ubuntu only having python 3. so we need to stick with some stable ansible version which supports all. Can we have a call to check this?

Reply email.

@saravanan30erd
remove selinux step for AM2
cef5af9 
Signed-off-by: saravanan palanisamy <[email protected]>
@saravanan30erd
update readme
46bdef4 
Signed-off-by: saravanan palanisamy <[email protected]>
@peterzhuamazon
Copy link
Member

Verified AL2:

IP  28 27 0 0.00 0.06 0.03 di - os5
IP  8 36 2 0.01 0.12 0.07 dm * os1
IP 26 27 0 0.00 0.08 0.05 di - os4
IP 27 27 0 0.00 0.06 0.03 dm - os2
IP 25 27 0 0.02 0.07 0.03 dm - os3

$ curl https://IP:9200/_cluster/health?pretty -u admin:admin --insecure
{
  "cluster_name" : "development-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 5,
  "discovered_master" : true,
  "active_primary_shards" : 3,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

peterzhuamazon
peterzhuamazon approved these changes Dec 22, 2021
View reviewed changes
Copy link
Member

@peterzhuamazon peterzhuamazon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@peterzhuamazon peterzhuamazon merged commit d98c62d into opensearch-project:main Dec 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bbarani bbarani bbarani approved these changes

@peterzhuamazon peterzhuamazon peterzhuamazon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement]Test runs on Amazon Linux 2 Machines
3 participants
@saravanan30erd @peterzhuamazon @bbarani