The GCP-WEKA Deployment Terraform module simplifies the creation of WEKA deployments on the Google Cloud Platform (GCP). It allows you to efficiently manage resources such as launch templates, cloud functions, workflows, and schedulers. Using the Terraform module establishes a process that automatically launches instances based on the specified cluster size.
Scope: This README describes the Terraform module’s configuration files. For the introduction and deployment workflows, refer to WEKA installation on GCP in WEKA documentation.
When deploying WEKA on GCP, you have two options for network configuration:
-
Use an existing network:
If you choose this option, WEKA uses your existing network resources. These resources include Virtual Private Clouds (VPCs), subnets, security groups (firewalls), private DNS zones, and VPC access connectors. Ensure that you provide the necessary network parameters when using an existing network. -
Automatically create network resources:
Alternatively, WEKA can create the required network resources for you. This includes setting up VPCs, subnets, security groups, private DNS zones, and VPC access connectors.
Refer to the examples for guidance.
Example of using an existing network:
vpcs_name = ["vpc-0","vpc-1","vpc-2","vpc-3"]
subnets_name = ["subnet-0","subnet-1","subnet-2","subnet-3"]
private_dns_name = "existing.private.net."
private_zone_name = "existing-private-zone"
vpc_connector_name = "existing-connector"module "weka_deployment" {
source = "weka/weka/gcp"
version = "3.0.2"
cluster_name = "myCluster"
project_id = "myProject"
vpcs_name = ["weka-vpc-0", "weka-vpc-1", "weka-vpc-2", "weka-vpc-3"]
region = "europe-west1"
subnets_name = ["weka-subnet-0","weka-subnet-1","weka-subnet-2","weka-subnet-3"]
zone = "europe-west1-b"
cluster_size = 7
nvmes_number = 2
vpc_connector = "weka-connector"
sa_email = "[email protected]"
get_weka_io_token = "GET_WEKA_IO_TOKEN"
private_dns_zone = "weka-private-zone"
private_dns_name = "weka.private.net."
}You can deploy the network on the host project and the cluster on the service project.
To set up the deployment, provide the following variable:
network_project_id = NETWORK_PROJECT_IDIn GCP, external IP addresses are always public and can be assigned to instances. These addresses allow communication with resources outside the Virtual Private Cloud (VPC) network.
Note: Using external IP addresses may incur additional charges.
To enable public IP assignment, set:
assign_public_ip = trueCloud NAT (Network Address Translation) on GCP allows instances within a private network to access the internet without requiring external IP addresses, enhancing security by keeping instances private while enabling outbound connectivity.
To enable Cloud NAT, set:
create_nat_gateway = trueWEKA supports tiering to buckets. To configure tiering, add the following variables:
tiering_enable_obs_integration = true
tiering_obs_name = "..."
tiering_enable_ssd_percent = 20WEKA enables automatic client creation and mounting. Specify the number of clients you need (default is 0).
For example, to create two clients, add the following:
clients_number = 2
You can also define the client instance type and the number of allocated cores with the following variables:
client_instance_type = "c2-standard-8"
client_frontend_cores = DESIRED_NUMTo mount the clients in UDP mode, add the following:
clients_use_dpdk = falseWEKA supports the creation of NFS protocol gateways that automatically mount to the cluster. Specify the number of instances (default is 0).
Example:
nfs_protocol_gateways_number = 2
Additional optional variables include:
nfs_protocol_gateway_machine_type = "c2-standard-8"
nfs_protocol_gateway_disk_size = 48
nfs_protocol_gateway_fe_cores_num = 1
nfs_setup_protocol = trueWEKA supports the creation of S3 protocol gateways that automatically mount to the cluster. Specify the number of instances (default is 0).
Example:
s3_protocol_gateways_number = 1
Additional optional variables include:
s3_protocol_gateway_machine_type = "c2-standard-8"
s3_protocol_gateway_disk_size = 48
s3_protocol_gateway_fe_cores_num = 1
s3_setup_protocol = trueWEKA supports the creation of SMB protocol gateways that automatically mount to the cluster. A minimum of three instances is required (default is 0).
Example:
smb_protocol_gateways_number = 3
Additional optional variables include:
smb_protocol_gateway_machine_type = "c2-standard-8"
smb_protocol_gateway_disk_size = 48
smb_protocol_gateway_fe_cores_num = 1
smb_setup_protocol = true
smb_cluster_name = ""
smb_domain_name = ""Join an SMB cluster in the Active Directory
To join an SMB cluster in the Active Directory, run this command manually:
weka smb domain join <smb_domain_username> <smb_domain_password> [--server smb_server_name].
Shared VPC (Virtual Private Cloud) lets you connect resources from multiple projects to a common VPC network. It’s a way to share network resources securely and efficiently. The host project defines the network and service projects attached to it, allowing eligible resources to use the shared network.
To enable the use of Shared VPC, provide the following variables:
shared_vpcs = [".."]
host_project = HOST_PROJECT_ID
host_shared_range = [".."]
shared_vpc_project_id = SHARED_VPC_PROJECT_ID
To enable the project as a host project, provide the following variable:
enable_shared_vpc_host_project = true
To enable VPC network peering between the host project and the service project, provide the following variable:
set_shared_vpc_peering = true| Name | Version |
|---|---|
| terraform | >=1.3.1 |
| archive | ~>2.4.0 |
| >=4.38.0 | |
| local | ~>2.4.0 |
| time | ~>0.9.1 |
| tls | ~>4.0.4 |
| Name | Version |
|---|---|
| archive | ~>2.4.0 |
| >=4.38.0 | |
| local | ~>2.4.0 |
| time | ~>0.9.1 |
| tls | ~>4.0.4 |
| Name | Source | Version |
|---|---|---|
| clients | ./modules/clients | n/a |
| network | ./modules/network | n/a |
| nfs_protocol_gateways | ./modules/protocol_gateways | n/a |
| peering | ./modules/vpc_peering | n/a |
| s3_protocol_gateways | ./modules/protocol_gateways | n/a |
| service_account | ./modules/service_account | n/a |
| shared_vpc_peering | ./modules/shared_vpcs | n/a |
| smb_protocol_gateways | ./modules/protocol_gateways | n/a |
| worker_pool | ./modules/worker_pool | n/a |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| allow_ssh_cidrs | List of CIDR blocks allowed for SSH access (port 22). If empty, SSH access is restricted to all sources (not recommended for production). Example: Allow access from specific IP addresses: allow_ssh_cidrs = ["10.0.0.1/32", "192.168.1.0/24"] | list(string) |
[] |
no |
| allow_weka_api_cidrs | List of CIDR blocks allowed for WEKA API access (port 14000). By default, no connections are allowed to port 14000. Specifying CIDRs here restricts access to the WEKA API on its backends and load balancer (if it exists and shares the security group) to the listed sources only. All ports (including 14000) are allowed within the VPC by default. | list(string) |
[] |
no |
| assign_public_ip | Controls public IP assignment for deployed instances (backends, clients, and gateways). | string |
"auto" |
no |
| boot_disk_type | The boot disk type. | string |
"pd-standard" |
no |
| client_frontend_cores | The number of frontend cores allocated to client instances. This value corresponds to the number of NICs attached to each instance because each WEKA core requires its dedicated NIC. | number |
1 |
no |
| client_instance_type | The client virtual machine type (SKU) to deploy. | string |
"c2-standard-8" |
no |
| client_nic_type | The type of virtual network interface (vNIC). Valid values include GVNIC and VIRTIO_NET. | string |
null |
no |
| client_source_image_id | Client Source image ID is set to Rocky 8.9. For the list of all supported Weka Client OSs please refer to: https://docs.weka.io/planning-and-installation/prerequisites-and-compatibility#operating-system | string |
"rocky-linux-8-v20240515" |
no |
| clients_number | The number of client virtual machines to deploy. | number |
0 |
no |
| clients_use_dpdk | Enables mounting WEKA clients in DPDK mode. | bool |
true |
no |
| cloud_functions_region_map | Defines a mapping between regions lacking Cloud Functions functionality and alternative regions. It ensures Cloud Functions availability by redirecting workflows to supported regions when necessary. | map(string) |
{ |
no |
| cloud_run_dns_zone_name | The name of existing private DNS zone for the domain run.app (it provides GCP hosting services). | string |
"" |
no |
| cloud_scheduler_region_map | Defines a mapping between regions lacking Cloud Scheduler functionality and alternative regions. It ensures Cloud Scheduler functionality by redirecting workflows to supported regions when necessary. | map(string) |
{ |
no |
| cluster_name | Cluster name prefix for all resources. | string |
n/a | yes |
| cluster_size | The number of instances deployed for your WEKA cluster. | number |
n/a | yes |
| containers_config_map | Defines a mapping of WEKA processes, NICs, and memory specifications for machine types. | map(object({ |
{ |
no |
| create_cloudscheduler_sa | Enables creation of a Cloud Scheduler service account. Set this to false to reuse an existing service account for Cloud Scheduler jobs. | bool |
true |
no |
| create_nat_gateway | Specifies whether to create a NAT gateway when no public IP is assigned to the backend, allowing internet access. | bool |
false |
no |
| create_worker_pool | Determines whether to create a worker pool. Set to true if a worker pool is needed. | bool |
false |
no |
| debug_down_backends_removal_timeout | Timeout duration for removing non-functional backends. Specify the timeout period in time units: ns, us (or µs), ms, s, m, h. This parameter is critical for managing the removal of non-operational backend resources. Consult with the WEKA Success Team before making any changes. | string |
"3h" |
no |
| default_disk_name | The default disk name. | string |
"wekaio-volume" |
no |
| default_disk_size | The default disk size. | number |
48 |
no |
| dns_zone_project_id | Project ID for the DNS zone. If omitted, it uses network project ID or falls back to project ID. | string |
"" |
no |
| enable_shared_vpc_host_project | Specifies whether the created project functions as a Shared VPC host project. If true, ensure the shared_vpc variable remains disabled (set to false). | bool |
true |
no |
| endpoint_apis_internal_ip_address | The private IP address for all-apis endpoint. | string |
"10.0.1.5" |
no |
| endpoint_vpcsc_internal_ip_address | The private IP address for VPC service connection endpoint. | string |
"10.0.1.6" |
no |
| get_weka_io_token | WEKA software download token. Obtain a valid token from https://get.weka.io/ to download and install the WEKA software during deployment. | string |
"" |
no |
| googleapis_dns_zone_name | The name of existing private DNS zone for domain googleapis.com. | string |
"" |
no |
| host_project | The ID of the project that acts as a shared VPC host project. | string |
"" |
no |
| host_shared_range | List of host ranges to allow security groups. | list(string) |
[] |
no |
| hotspare | A hot spare is the system's ability to withstand the loss of a defined number of failure domains, rebuild data completely, and maintain original net capacity. | number |
1 |
no |
| install_cluster_dpdk | Specifies whether to install the WEKA cluster with DPDK. | bool |
true |
no |
| install_weka_url | The URL to WEKA installation script or tar object. Examples: URL to installation script: https://[email protected]/dist/v1/install/4.3.1/4.3.1. URL to tar object: https://[email protected]/dist/v1/pkg/weka-4.3.1.tar. URL to tar object in a cloud bucket: gs://weka-installation/weka-4.2.12.87.tar. (Replace TOKEN with your valid WEKA download token). | string |
"" |
no |
| lb_allow_global_access | Specifies whether to allow access to the load balancer from all regions. | bool |
false |
no |
| machine_type | The machine type for the WEKA backend instance. | string |
"c2-standard-8" |
no |
| mtu_size | The Maximum Transmission Unit (MTU) size is the largest packet size that can be transmitted over a network. | number |
8896 |
no |
| network_project_id | The project ID for the network. | string |
"" |
no |
| nfs_interface_group_name | Interface group name. | string |
"weka-ig" |
no |
| nfs_protocol_gateway_disk_size | The default disk size for NFS protocol gateways. | number |
48 |
no |
| nfs_protocol_gateway_fe_cores_num | The number of frontend cores on each NFS protocol gateway machine. | number |
1 |
no |
| nfs_protocol_gateway_machine_type | The virtual machine type (SKU) for the NFS protocol gateways to deploy. | string |
"c2-standard-8" |
no |
| nfs_protocol_gateway_secondary_ips_per_nic | The number of secondary IPs per single NIC per NFS protocol gateway virtual machine. | number |
0 |
no |
| nfs_protocol_gateways_number | The number of NFS protocol gateway virtual machines to deploy. | number |
0 |
no |
| nfs_setup_protocol | Specifies whether to configure the NFS protocol. | bool |
false |
no |
| nic_number | Number of NICs per host. | number |
-1 |
no |
| nic_type | The type of vNIC. Possible values: GVNIC, VIRTIO_NET. | string |
null |
no |
| nvmes_number | Number of NVMe disks to attach to each host. | number |
2 |
no |
| prefix | Prefix for all resources (maximum 15 characters). | string |
"weka" |
no |
| private_dns_name | Private DNS name. | string |
"" |
no |
| private_zone_name | Private zone name. | string |
"" |
no |
| project_id | Project id | string |
n/a | yes |
| protection_level | The protection level, referring to the cluster data, indicates the number of additional protection blocks per stripe, either 2 or 4. | number |
2 |
no |
| proxy_url | The URL for the WEKA Home proxy. | string |
"" |
no |
| psc_subnet_cidr | The CIDR range for the private service connection subnet. | string |
"10.9.0.0/28" |
no |
| region | GCP region, a broader geographic area within GCP that houses your resources. It encompasses multiple zones. | string |
n/a | yes |
| s3_protocol_gateway_disk_size | The default disk size for S3 protocol gateways. | number |
375 |
no |
| s3_protocol_gateway_fe_cores_num | The number of frontend cores on each S3 protocol gateway machine. | number |
1 |
no |
| s3_protocol_gateway_machine_type | The virtual machine type (SKU) for deploying S3 protocol gateways. | string |
"c2-standard-8" |
no |
| s3_protocol_gateways_number | The Number of virtual machines to deploy as S3 protocol gateways. | number |
0 |
no |
| s3_setup_protocol | Specifies whether to configure S3 protocol cluster. | bool |
false |
no |
| sa_email | Email address of an existing service account to be used. Leave blank to create a new service account during deployment. | string |
"" |
no |
| set_dedicated_fe_container | Creates a cluster with dedicated frontend containers. | bool |
true |
no |
| set_peering | Specifies whether to apply peering connection between subnets. | bool |
true |
no |
| set_shared_vpc_peering | Enables peering for shared VPC. | bool |
true |
no |
| shared_vpc_project_id | The project ID for the shared VPC. | string |
"" |
no |
| shared_vpcs | list of shared vpc names. | list(string) |
[] |
no |
| smb_cluster_name | The name of the SMB cluster. | string |
"Weka-SMB" |
no |
| smb_domain_name | The domain to join the SMB cluster. | string |
"" |
no |
| smb_protocol_gateway_disk_size | The default disk size for SMB protocol gateways. | number |
375 |
no |
| smb_protocol_gateway_fe_cores_num | The number of frontend cores on each SMB protocol gateway machine. | number |
1 |
no |
| smb_protocol_gateway_machine_type | The virtual machine type (SKU) for deploying SMB protocol gateways. | string |
"c2-standard-8" |
no |
| smb_protocol_gateway_secondary_ips_per_nic | Number of secondary IPs per NIC per SMB protocol gateway virtual machine. | number |
3 |
no |
| smb_protocol_gateways_number | The number of virtual machines to deploy as SMB protocol gateways. | number |
0 |
no |
| smb_setup_protocol | Specifies whether to configure SMB protocol cluster. | bool |
false |
no |
| smbw_enabled | Enables SMBW protocol. Allocate extra capacity for SMB-W cluster before creating the cluster. | bool |
true |
no |
| source_image_id | Source image for deployment (default: rocky-linux-8-v20240515). While other distributions may be compatible, only Rocky Linux 8 is officially tested with this Terraform module. | string |
"rocky-linux-8-v20240515" |
no |
| ssh_public_key | The SSH public key to pass to VMs. | string |
null |
no |
| state_bucket_name | The name of a bucket used for state storage in the cloud. | string |
"" |
no |
| stripe_width | The stripe width is the number of blocks sharing a common protection set, which ranges from 3 to 16. By default, stripe_width = cluster_size - protection_level - 1. The default value -1 means the stripe size is calculated automatically; otherwise, the specified value is used. | number |
-1 |
no |
| subnet_autocreate_as_private | Creates a private subnet using NAT gateway to route traffic. The default is a public network. Applicable only when subnet_ids is empty. | bool |
false |
no |
| subnets_name | List of names (0, 4, or 7) for subnets defined in the subnets_range variable. | list(string) |
[] |
no |
| subnets_range | List of subnet CIDRs (0, 4, or 7) for cluster creation. 0: No subnets (for single-node testing). 4: Common setup for production (spread across AZs for redundancy). 7: Less common deployments with specific needs. | list(string) |
[ |
no |
| tiering_enable_obs_integration | Controls integration with object stores in the WEKA cluster for tiering. Set to true to enable this integration. | bool |
false |
no |
| tiering_enable_ssd_percent | When the OBS integration setting is enabled, this parameter sets the percentage of the filesystem capacity that resides on the SSD. For example, if this parameter is set to 20 (percent) and the total available SSD capacity is 20GB, the total capacity is 100 GB. | number |
20 |
no |
| tiering_obs_name | The name of the OBS cloud storage used for tiering. | string |
"" |
no |
| tiering_obs_start_demote | Target tiering cue (in seconds) before starting upload data to OBS (turning it into read cache). Default is 10 seconds. | number |
10 |
no |
| tiering_obs_target_ssd_retention | Target retention period (in seconds) before tiering to OBS (how long data will stay in SSD). Default is 86400 seconds (24 hours). | number |
86400 |
no |
| traces_per_ionode | The number of traces generated per ionode. Traces represent low-level events generated by WEKA processes and are used for support. | number |
10 |
no |
| vm_username | The username provided as part of the output for automated use of Terraform. Replace with the user for SSH connection in case of custom image and automated use of outputs. | string |
"weka" |
no |
| vpc_connector_egress_settings | Egress settings for the VPC connector. Possible values: ALL_TRAFFIC, PRIVATE_RANGES_ONLY. | string |
"PRIVATE_RANGES_ONLY" |
no |
| vpc_connector_id | ID of an existing VPC connector for serverless VPC access in the format: projects//locations//connectors/. Leave blank to create a new VPC connector during deployment. | string |
"" |
no |
| vpc_connector_range | VPC connector CIDR block for serverless VPC access. | string |
"10.8.0.0/28" |
no |
| vpcs_name | Names of VPC networks to associate with the resource. Depending on your configuration, you can specify 0, 4, or 7 VPC networks. | list(string) |
[] |
no |
| vpcs_range_to_peer_to_deployment_vpc | The list of VPC ranges to peer in CIDR format. | list(string) |
[] |
no |
| vpcs_to_peer_to_deployment_vpc | The list of VPC names to peer. | list(string) |
[] |
no |
| weka_home_url | The URL for WEKA Home. | string |
"" |
no |
| weka_tar_bucket_name | The bucket name for the WEKA software tar file. | string |
"" |
no |
| weka_tar_project_id | The project ID for the WEKA software tar file. | string |
"" |
no |
| weka_version | WEKA version | string |
"" |
no |
| worker_address_prefix_length | The prefix length for IP addresses, expressed in the worker_pool_address_cidr. For example, use 24 for a /24 subnet or 16 for a /16 subnet. The maximum value is 24. | string |
"16" |
no |
| worker_disk_size | The size of the disk attached to the worker node in GB. | number |
100 |
no |
| worker_machine_type | The machine type for a worker. | string |
"e2-standard-4" |
no |
| worker_pool_address_cidr | The address range for worker machines within a Cloud Build Private Pool. It follows CIDR notation excluding the prefix length. | string |
"10.37.0.0" |
no |
| worker_pool_id | The unique identifier for the worker pool. The worker pool must belong to the same project and region. If left empty, the default worker pool is used. | string |
"" |
no |
| workflow_map_region | Defines a mapping between regions lacking Cloud Workflows functionality and alternative regions. It ensures Cloud Workflows functionality by redirecting workflows to supported regions when necessary. | map(string) |
{ |
no |
| yum_repo_server | URL of a Yum repository server for package installation. Leave blank to use the default repositories. | string |
"" |
no |
| zone | GCP zone, which is a deployment area within a region, providing physical separation for your resources. | string |
n/a | yes |
| Name | Description |
|---|---|
| backend_lb_ip | The backend load balancer ip address. |
| client_ips | If 'assign_public_ip' is set to true, it will output clients public ips, otherwise private ips. |
| cluster_helper_commands | Useful commands and script to interact with weka cluster |
| cluster_name | The cluster name |
| functions_url | Functions url and body for api request |
| get_cluster_status_uri | URL of status function |
| lb_url | URL of LB |
| nfs_protocol_gateways_ips | Ips of NFS protocol gateways |
| private_ssh_key | private_ssh_key: If 'ssh_public_key' is set to null, it will output the private ssh key location. |
| project_id | Project ID |
| resize_cluster_uri | URL of resize function |
| s3_protocol_gateways_ips | Ips of S3 protocol gateways |
| smb_protocol_gateways_ips | Ips of SMB protocol gateways |
| terminate_cluster_uri | URL of terminate function |
| vm_username | Provided as part of output for automated use of terraform, ssh user to weka cluster vm |
| weka_cluster_admin_password_secret_id | Secret id of weka cluster admin password |