• OpenSearch Continuous Integration
• Getting Started
• Deployment
  • CI Deployment
  • Dev Deployment
  • Executing Optional Tasks
    • SSL Configuration
    • Setup OpenId Connect (OIDC) via Federate
    • Restricting Server Access
    • Data Retention
    • Add environment variable
    • Assume role
    • Mac agents
    • Use Production Agents
  • Troubleshooting
    • Main Node
  • Useful commands
  • Architecture Overview
• Contributing
• Getting Help
• Code of Conduct
• Security
• License
• Copyright

OpenSearch Continuous Integration

OpenSearch Continuous Integration is an open source CI system for OpenSearch and its plugins.

Getting Started

• Requires NPM to be installed
• Install project dependencies using npm install from this project directory
• Configure aws credentials
• Deploy stacks with npm run cdk deploy

Deployment

Important: Please ensure to use strong passwords for you jenkins instance in order to keep it secure.

CI Deployment

1. Create another cdk project and depend on this package
2. Import the config / ci stacks alongside the other resources

   new CIConfigStack(app, 'CI-Config-Beta', {});
   new CIStack(app, 'CI-Beta', ciSettings, {});

3. Update the ciSettings according to the environment needs such as SSL or strict deployment, see CIStackProps for details.
4. Update the assetsSettings according to the environment needs such as SSL or strict deployment, see deployAwsAssetProps for details.
5. Deploy using the CI system of your choice.

Dev Deployment

1. Setup your local machine to credentials to deploy to the AWS Account

2. Deploy the bootstrap stack by running the following command that sets up required resources to create the stacks. More info

   npm run cdk bootstrap -- -c useSsl=false -c runWithOidc=false -c serverAccessType=ipv4 -c restrictServerAccessTo=10.10.10.10/32

3. Deploy the ci-config-stack using the following (takes ~1 minute to deploy) -

   npm run cdk deploy OpenSearch-CI-Config-Dev -- -c useSsl=false -c runWithOidc=false -c serverAccessType=ipv4 -c restrictServerAccessTo=10.10.10.10/32

4. Locate the secret manager arns in the ci-config-stack outputs for CASC_RELOAD_TOKEN and update the secret value (see docs) with the password you want to use to reload jenkins configuration. Do not enclose it in quotes

$aws secretsmanager put-secret-value \
--secret-id MyCASCreloadTokenSecretARN \
--secret-string CascReloadToken

5. Optional Configure the elements of the config stack for SSL configuration

6. Optional Configure the elements setting up oidc via federate

7. Deploy the ci-stack, takes ~10 minutes to deploy (parameter values depend on step 2 and step 3)

   npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c serverAccessType=ipv4 -c restrictServerAccessTo=10.10.10.10/32

8. Fetch the key-pair id of AgentNodeKeyPair and locate actual value in SSM Parameter Store, it will of the format /ec2/keypair/{key_pair_id}. Add the actual value in Secrets Manager to secret named jenkins-agent-node-key-pair. This will allow jenkins manager node to be able to connect to agent nodes.

9. When OIDC is disabled, this set up will enforce the user to secure jenkins by adding first admin user on deployment. Create admin user and password, fill in all other details like name and email id to start using jenkins.

10. Go to the OpenSearch-CI-Dev.JenkinsExternalLoadBalancerDns url returned by CDK output to access the jenkins host.

11. If you want to destroy the stack make sure you delete the agent nodes manually (via jenkins UI or AWS console) so that shared resources (like vpc, security groups, etc) can be deleted.

Executing Optional Tasks

Construct Props

Name	Type	Description
useSsl	boolean	Should the Jenkins use https
runWithOidc	boolean	Should an OIDC provider be installed on Jenkins
restrictServerAccessTo	Ipeer	Restrict jenkins server access
ignoreResourcesFailures	boolean	Additional verification during deployment and resource startup
adminUsers	string[]	List of users with admin access during initial deployment
additionalCommands	string	Additional logic that needs to be run on Master Node. The value has to be path to a file
dataRetention	boolean	Do you want to retain jenkins jobs and build history
agentAssumeRole	string	IAM role ARN to be assumed by jenkins agent nodes
envVarsFilePath	string	Path to file containing env variables in the form of key value pairs
macAgent	boolean	Add mac agents to jenkins
useProdAgents	boolean	Should jenkins server use production agents
enableViews	boolean	Adds Build, Test, Release and Misc views to Jenkins Dashboard . Defaults to false

SSL Configuration

1. Locate the secret manager arns in the ci-config-stack outputs
2. Update the secret value (see docs) for the certContentsSecret with the certificate contents

$aws secretsmanager put-secret-value \
--secret-id MyTestDatabaseSecret_or_ARN \
--secret-string file://mycreds.json_or_value

1. Update the secret value (see docs) for the privateKeySecret with the certificate private key
2. Upload the certificate to IAM see docs
3. Update the secret value for the certificateArnSecret with the certificate arn generated by IAM
4. Update the secret value for redirectUrlSecret with a dummy or valid redirect URL. eg: https://dummyJenkinsUrl.com
5. Run with parameter using one of the following (refer this for value of runWithOidc)
   1. npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=true -c runWithOidc=true or,
   2. cdk deploy OpenSearch-CI-Dev -c useSsl=true -c runWithOidc=true
6. Continue with next steps

Setup OpenId Connect (OIDC) via Federate

1. Locate the secret manager arns in the ci-config-stack outputs
2. Update the secret value (see docs) for the OIDCClientIdSecret with the credentials as json as follows:
   1. JSON format

    {
        "clientId": "example_id",
        "clientSecret": "example_password",
        "wellKnownOpenIDConfigurationUrl": "https://www.example.com",
        "tokenServerUrl": "https://example.com/token",
        "authorizationServerUrl": "https://example.com/authorize",
        "userInfoServerUrl": "https://example.com/userinfo"
    }
   

   1. Command Eg: see docs

    $aws secretsmanager put-secret-value \
    --secret-id MyTestDatabaseSecret_or_ARN \
    --secret-string file://mycreds.json_or_value
   

3. Add additional adminUsers for role based authentication according to your needs, see CIStackProps for details.
4. Run with parameter with one of the following (refer this for value of useSsL) -
   1. npm run cdk deploy OpenSearch-CI-Dev -- -c runWithOidc=true -c useSsl=true or,
   2. cdk deploy OpenSearch-CI-Dev -c runWithOidc=true -c useSsl=true
5. Continue with next steps

Restricting Server Access

You need to restrict access to your jenkins endpoint (load balancer). Here's how:

1. Using command line as below:

npm run cdk synth OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c serverAccessType=ipv4 -c restrictServerAccessTo=10.10.10.10/32

Below values are allowed:

serverAccessType	restrictServerAccessTo
ipv4	all (0.0.0.0/0) or any ipv4 CIDR (eg: 10.10.10.10/32)
ipv6	all (::/0) or any ipv6 CIDR (eg: 2001:0db8:85a3:0000:0000:8a2e:0370:7334)
prefixList	Prefix List id (eg: ab-12345)
securityGroupId	A security group ID (eg: sg-123456789)

2. You can also update the restrictServerAccessTo property in ciSettings to your desired Ipeer. See CIStackProps for details.

Example:

const stack = new CIStack(app, 'MyStack', { restrictServerAccessTo: Peer.ipv4('10.0.0.0/24') });

Ignore resources failure

Update the ignoreResourcesFailures property in ciSettings. This parameter is used to ignore resources failure if the option is available. Right now that is only applicable while for EC2 init scripts.

Data Retention

Change in any EC2 config (specially init config) leads to replacement of EC2. The jenkins configuration is managed via code using configuration as code plugin. More details. See inital jenkins.yaml If you want to retain all the jobs and its build history,

1. Update the dataRetention property in ciSettings to true (defaults to false) see CIStackProps for details. This will create an EFS (Elastic File System) and mount it on /var/lib/jenkins which will retain all jobs and its build history.

Add environment variables

Users can add global level environment variables using configuration as code as follows:

Update the envVarsFilePath property in ciSettings to the yaml file path containing all environment variables in the form of key:value pair. See CIStackProps for details.

Example: See env.txt

envVarsFilePath = 'test/data/env.yaml'

Assume role

The Created jenkins agent role can assume cross account role by passing agentAssumeRole parameter Example:

npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c agentAssumeRole=arn:aws:iam::522XXX13897:role/sample-assume-role

NOTE: The assume role has to be pre-created for the agents to assume. Once CDK stack is deployed with -c agentAssumeRole flag, make sure this flag is passed for next CDK operations to make sure this created policy that assumes cross-account role is not removed.

Mac agents

Prerequisite

To deploy mac agents, as a prerequisites make sure the backend AWS account has dedicated hosts setup done with instance family as mac1 and instance type as mac1.metal. For More details check the getting-started guide.

Configuration

To configure ec2 Mac agent setup run the stack with -c macAgent=true. Example:

npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c macAgent=true

Windows agents

Prerequisite

Make sure there is an existing Windows AMI with necessary requirements, see packer directory for more information and AMI build commands.

Runnning additional commands

In cases where you need to run additional logic/commands, such as adding a cron to emit ssl cert expiry metric, you can pass the commands as a script using additionalCommands context parameter. Below sample will write the python script to $HOME/hello-world path on jenkins master node and then execute it once the jenkins master node has been brought up.

cat << EOF > $HOME/hello-world && chmod 755 $HOME/hello-world && $HOME/hello-world
#!/usr/bin/env python3
def print_hello():
    print("Hello World")


if __name__ == "__main__":
    print_hello()
EOF

To use above example, you need to write the contents of the script to a file, say example.txt and pass the path of example.txt to additionalCommands paramter.
Usage:

npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c additionalCommands='./example.txt'

Use Production Agents

Please note that if you have decided to use the provided production jenkins agents then please make sure that you are deploying the stack in US-EAST-1 region as the AMIs used are only publicly available in US-EAST-1 region. If you want to deploy the stack in another region then please make sure you copy the public AMIs used from us-east-1 region to your region of choice and update the new ami-id in agent-nodes.ts file accordingly.

If you do not copy the AMI in required region and update the code then the desired jenkins agents will not spin up.

If you do not specify this flag or use false then jenkins server will spin up two ec2 agents with minimal config, AL2 AMD64 and ARM64, they will be using the latest ami available.

Enable Views

Views on Jenkins dashboard allows us to classify jobs into different sections. By enabling views, all the jobs will be classified into 4 categories namely Build, Test, Release and Misc.

• Build - Any job containing word build in it.
• Test - Any job containing word test in it.
• Release - Any job containing word release in it.
• Misc - All the jobs that belong to neither of the above categories.

The All is the default view and contains all the jobs.

Troubleshooting

Main Node

Useful links

• Log are found in Cloud Watch Logs
• Need to access the host, ssh via Session Manager in EC2 Console
• Instance instance isn't coming up, get the system log in EC2 Console

Useful commands

• npm run build compile typescript to js, run lint, run tests
• npm run watch watch for changes and compile
• npm run cdk deploy deploy this stack to your default AWS account/region
• npm run cdk diff compare deployed stack with current state
• npm run cdk synth emits the synthesized CloudFormation template

Architecture Overview

Built using AWS Cloud Development Kit the configuration of the CI systems will be available for replication in your own accounts. The Jenkins instance will be hardened and publically visible, connected to GitHub to make build notifications easy for everyone to see.

Contributing

See developer guide and how to contribute to this project.

Getting Help

If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.

For more information, see project website and documentation. If you need help and are unsure where to open an issue, try forums.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

Security

If you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to [email protected]. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.