[BUG][Security Plugin Configuration] securityadmin.sh execution fails

[rodolfovillordo]

Describe the bug
New securityadmin.sh execution fails on task Security Plugin configuration | Initialize the opensearch security index in opensearch if copy_custom_security_configs is False.

To Reproduce
Steps to reproduce the behavior:

1. checkout the latest version from this repository
2. Apply the fix for [BUG][Security Plugin Configuration] #80
3. Apply the fix for [BUG][Security Plugin Configuration] local action requesting unnecessary privilege escalation #82 or workaround local become request
4. Apply the fix for [BUG][Security Plugin Configuration] Unecessary user left on internal_users.yml #86
5. Execute the playbook as instructed on the README:

$ ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789" --become 

7. See error

{
  "changed": true,
  "cmd": "bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig -nhnv -icl -h 172.31.83.42
",
  "delta": "0:00:02.979015",
  "end": "2022-07-19 17:57:47.499792",
  "msg": "non-zero return code",
  "rc": 255,
  "start": "2022-07-19 17:57:44.520777",
  "stderr": "",
  "stderr_lines": [],
  "stdout": "**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 172.31.83.42:9200 ... done
Connected as \"CN=admin.example.com,OU=Ops,O=example.com\\\\, Inc.,DC=example.com\"
OpenSearch Version: 2.1.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: development-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)
Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml (legacy mode)
   SUCC: Configuration for 'internalusers' created or updated
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
ERR: cannot upload configuration, see errors above",
  "stdout_lines": [
    "**************************************************************************",
    "** This tool will be deprecated in the next major release of OpenSearch **",
    "** https://github.com/opensearch-project/security/issues/1755           **",
    "**************************************************************************",
    "Security Admin v7",
    "Will connect to 172.31.83.42:9200 ... done",
    "Connected as \"CN=admin.example.com,OU=Ops,O=example.com\\\\, Inc.,DC=example.com\"",
    "OpenSearch Version: 2.1.0",
    "Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...",
    "Clustername: development-cluster",
    "Clusterstate: GREEN",
    "Number of nodes: 1",
    "Number of data nodes: 1",
    ".opendistro_security index already exists, so we do not need to create one.",
    "Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!",
    "Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)",
    "Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml (legacy mode)",
    "   SUCC: Configuration for 'internalusers' created or updated",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)",
    "ERR: cannot upload configuration, see errors above"
  ]
}

in case of multi-node deployment the service will not start: #83 (comment)

Host/Environment (please complete the following information):

• Ansible Version: 2.12.6
• Playbook Version: 2.1.0

[rodolfovillordo]

On multi-node deployment the behavior is slightly different from single node. the OpenSearch service does not start:

TASK [linux/opensearch : Wait for opensearch to startup] *************************************************************************************************************************************
fatal: [os4]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.31.94.224:9200"}
fatal: [os5]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.31.86.160:9200"}
fatal: [os1]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.31.91.122:9200"}
fatal: [os3]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.31.94.117:9200"}
fatal: [os2]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.31.85.137:9200"}

PLAY RECAP ***********************************************************************************************************************************************************************************
os1                        : ok=22   changed=5    unreachable=0    failed=1    skipped=31   rescued=0    ignored=0
os2                        : ok=18   changed=5    unreachable=0    failed=1    skipped=18   rescued=0    ignored=0
os3                        : ok=18   changed=5    unreachable=0    failed=1    skipped=18   rescued=0    ignored=0
os4                        : ok=18   changed=5    unreachable=0    failed=1    skipped=18   rescued=0    ignored=0
os5                        : ok=18   changed=5    unreachable=0    failed=1    skipped=18   rescued=0    ignored=02

On service logs it says that the likely root cause is the certificate filepath was not set.

Jul 20 20:22:33 os1 systemd[1]: Started opensearch.
Jul 20 20:22:35 os1 opensearch[3261]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 20 20:22:35 os1 opensearch[3261]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.1.0.jar)
Jul 20 20:22:35 os1 opensearch[3261]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jul 20 20:22:35 os1 opensearch[3261]: WARNING: System::setSecurityManager will be removed in a future release
Jul 20 20:22:36 os1 opensearch[3261]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 20 20:22:36 os1 opensearch[3261]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.1.0.jar)
Jul 20 20:22:36 os1 opensearch[3261]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jul 20 20:22:36 os1 opensearch[3261]: WARNING: System::setSecurityManager will be removed in a future release
Jul 20 20:22:38 os1 opensearch[3261]: uncaught exception in thread [main]
Jul 20 20:22:38 os1 opensearch[3261]: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Jul 20 20:22:38 os1 opensearch[3261]: Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
Jul 20 20:22:38 os1 opensearch[3261]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419)
Jul 20 20:22:38 os1 opensearch[3261]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255)
Jul 20 20:22:38 os1 opensearch[3261]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:176)
Jul 20 20:22:38 os1 opensearch[3261]:         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
Jul 20 20:22:38 os1 opensearch[3261]:         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:262)
Jul 20 20:22:38 os1 opensearch[3261]:         at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)