Skip to content

Latest commit

 

History

History
146 lines (109 loc) · 12.3 KB

README.md

File metadata and controls

146 lines (109 loc) · 12.3 KB

Module - Scale runners

This module creates resources required to run the GitHub action runner on AWS EC2 spot instances. The life cycle of the runners on AWS is managed by two lambda functions. One function will handle scaling up, the other scaling down.

Overview

Action runners on EC2

The action runners are created via a launch template, on launch template only the subnet needs to be provided. During launch the installation is handled via a user data script. The configuration is fetched from SSM parameter store.

Lambda scale up

The scale up lambda is triggered by events on a SQS queue. Events on this queued are delayed, which will will give the workflow some time to start running on available runners. For each event the lambda will check the workflow is still queued and no other limits are reached. In that case the lambda will create a new EC2 instance. The lambda only needs to know which launch template to use and which subnets are available. From the available subnets a random one will be chosen. Once the instance is created the event is assumed as handled, and we assume the workflow wil start at some moment once the created instance is ready.

Lambda scale down

The scale down lambda is triggered via a CloudWatch event. The event is triggered by a cron expression defined in the variable scale_down_schedule_expression (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html). For scaling down GitHub does not provide a good API yet, therefore we run the scaling down based on this event every x minutes. Each time the lambda is triggered it tries to remove all runners older than x minutes (configurable) managed in this deployment. In case the runner can be removed from GitHub, which means it is not executing a workflow, the lambda will terminate the EC2 instance.

Usages

Usage examples are available in the root module. By default the root module will assume local zip files containing the lambda distribution are available. See the download lambda module for more information.

Lambda Function

The Lambda function is written in TypeScript and requires Node 12.x and yarn. Sources are located in [./lambdas/runners]. Two lambda functions share the same sources, there is one entry point for scaleDown and another one for scaleUp.

Install

cd lambdas/runners
yarn install

Test

Test are implemented with Jest, calls to AWS and GitHub are mocked.

yarn run test

Package

To compile all TypeScript/JavaScript sources in a single file ncc is used.

yarn run dist

Requirements

No requirements.

Providers

Name Version
aws n/a

Inputs

Name Description Type Default Required
ami_filter List of maps used to create the AMI filter for the action runner AMI. map(list(string)) 
{
  "name": [
    "amzn2-ami-hvm-2.*-x86_64-ebs"
  ]
}
 no
ami_owners The list of owners used to select the AMI of action runner instances. list(string) 
[
  "amazon"
]
 no
aws_region AWS region. string n/a yes
block_device_mappings The EC2 instance block device configuration. Takes the following keys: device_name, delete_on_termination, volume_type, volume_size, encrypted, iops map(string) {} no
cloudwatch_config (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. string null no
create_service_linked_role_spot (optional) create the serviced linked role for spot instances that is required by the scale-up lambda. bool false no
enable_cloudwatch_agent Enabling the cloudwatch agent on the ec2 runner instances, the runner contains default config. Configuration can be overridden via cloudwatch_config. bool true no
enable_organization_runners n/a bool n/a yes
enable_ssm_on_runners Enable to allow access the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. bool n/a yes
encryption KMS key to encrypted lambda environment secrets. Either provide a key and encrypt set to true. Or set the key to null and encrypt to false. 
object({
    kms_key_id = string
    encrypt    = bool
  })
 n/a yes
environment A name that identifies the environment, used as prefix and for tagging. string n/a yes
ghes_url GitHub Enterprise Server URL. DO NOT SET IF USING PUBLIC GITHUB string null no
github_app GitHub app parameters, see your github app. Ensure the key is the base64-encoded .pem file (the output of base64 app.private-key.pem, not the content of private-key.pem). 
object({
    key_base64    = string
    id            = string
    client_id     = string
    client_secret = string
  })
 n/a yes
idle_config List of time period that can be defined as cron expression to keep a minimum amount of runners active instead of scaling down to 0. By defining this list you can ensure that in time periods that match the cron expression within 5 seconds a runner is kept idle. 
list(object({
    cron      = string
    timeZone  = string
    idleCount = number
  }))
 [] no
instance_profile_path The path that will be added to the instance_profile, if not set the environment name will be used. string null no
instance_type Default instance type for the action runner. string "m5.large" no
key_name Key pair name string null no
lambda_s3_bucket S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. any null no
lambda_security_group_ids List of subnets in which the lambda will be launched, the subnets needs to be subnets in the vpc_id. list(string) [] no
lambda_subnet_ids List of subnets in which the lambda will be launched, the subnets needs to be subnets in the vpc_id. list(string) [] no
lambda_timeout_scale_down Time out for the scale down lambda in seconds. number 60 no
lambda_timeout_scale_up Time out for the scale up lambda in seconds. number 60 no
lambda_zip File location of the lambda zip file. string null no
logging_retention_in_days Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. number 180 no
market_options Market options for the action runner instances. string "spot" no
minimum_running_time_in_minutes The time an ec2 action runner should be running at minimum before terminated if non busy. number 5 no
overrides This maps provides the possibility to override some defaults. The following attributes are supported: name_sg overwrite the Name tag for all security groups created by this module. name_runner_agent_instance override the Name tag for the ec2 instance defined in the auto launch configuration. name_docker_machine_runners override the Name tag spot instances created by the runner agent. map(string) 
{
  "name_runner": "",
  "name_sg": ""
}
 no
role_path The path that will be added to the role, if not set the environment name will be used. string null no
role_permissions_boundary Permissions boundary that will be added to the created role for the lambda. string null no
runner_additional_security_group_ids (optional) List of additional security groups IDs to apply to the runner list(string) [] no
runner_architecture The platform architecture of the runner instance_type. string "x64" no
runner_as_root Run the action runner under the root user. bool false no
runner_extra_labels Extra labels for the runners (GitHub). Separate each label by a comma string "" no
runner_group_name Name of the runner group. string "Default" no
runner_iam_role_managed_policy_arns Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role list(string) [] no
runner_log_files (optional) List of logfiles to send to cloudwatch, will only be used if enable_cloudwatch_agent is set to true. Object description: log_group_name: Name of the log group, prefix_log_group: If true, the log group name will be prefixed with /github-self-hosted-runners/<var.environment>, file_path: path to the log file, log_stream_name: name of the log stream. 
list(object({
    log_group_name   = string
    prefix_log_group = bool
    file_path        = string
    log_stream_name  = string
  }))
 
[
  {
    "file_path": "/var/log/messages",
    "log_group_name": "messages",
    "log_stream_name": "{instance_id}",
    "prefix_log_group": true
  },
  {
    "file_path": "/var/log/user-data.log",
    "log_group_name": "user_data",
    "log_stream_name": "{instance_id}",
    "prefix_log_group": true
  },
  {
    "file_path": "/home/ec2-user/actions-runner/diag/Runner**.log",
    "log_group_name": "runner",
    "log_stream_name": "{instance_id}",
    "prefix_log_group": true
  }
]
 no
runners_lambda_s3_key S3 key for runners lambda function. Required if using S3 bucket to specify lambdas. any null no
runners_lambda_s3_object_version S3 object version for runners lambda function. Useful if S3 versioning is enabled on source bucket. any null no
runners_maximum_count The maximum number of runners that will be created. number 3 no
s3_bucket_runner_binaries n/a 
object({
    arn = string
  })
 n/a yes
s3_location_runner_binaries S3 location of runner distribution. string n/a yes
scale_down_schedule_expression Scheduler expression to check every x for scale down. string "cron(*/5 * * * ? *)" no
sqs_build_queue SQS queue to consume accepted build events. 
object({
    arn = string
  })
 n/a yes
subnet_ids List of subnets in which the action runners will be launched, the subnets needs to be subnets in the vpc_id. list(string) n/a yes
tags Map of tags that will be added to created resources. By default resources will be tagged with name and environment. map(string) {} no
userdata_post_install User-data script snippet to insert after GitHub acton runner install string "" no
userdata_pre_install User-data script snippet to insert before GitHub acton runner install string "" no
userdata_template Alternative user-data template, replacing the default template. By providing your own user_data you have to take care of installing all required software, including the action runner. Variables userdata_pre/post_install are ignored. string null no
vpc_id The VPC for the security groups. string n/a yes

Outputs

Name Description
lambda_scale_down n/a
lambda_scale_up n/a
launch_template n/a
role_runner n/a
role_scale_down n/a
role_scale_up n/a

Philips Forest

This module is part of the Philips Forest.


                                                     ___                   _
                                                    / __\__  _ __ ___  ___| |_
                                                   / _\/ _ \| '__/ _ \/ __| __|
                                                  / / | (_) | | |  __/\__ \ |_
                                                  \/   \___/|_|  \___||___/\__|

                                                                 Infrastructure

Talk to the forestkeepers in the forest-channel on Slack.

Slack