csi: document volume expansion (#18573)

and show Capacity in `volume status` command.

command/volume_status_csi.go

@@ -10,6 +10,7 @@ import (
 	"sort"
 	"strings"
 
+	"github.com/dustin/go-humanize"
 	"github.com/hashicorp/nomad/api"
 )
 
@@ -218,6 +219,7 @@ func (c *VolumeStatusCommand) formatBasic(vol *api.CSIVolume) (string, error) {
 		fmt.Sprintf("Plugin ID|%s", vol.PluginID),
 		fmt.Sprintf("Provider|%s", vol.Provider),
 		fmt.Sprintf("Version|%s", vol.ProviderVersion),
+		fmt.Sprintf("Capacity|%s", humanize.IBytes(uint64(vol.Capacity))),
 		fmt.Sprintf("Schedulable|%t", vol.Schedulable),
 		fmt.Sprintf("Controllers Healthy|%d", vol.ControllersHealthy),
 		fmt.Sprintf("Controllers Expected|%d", vol.ControllersExpected),

website/content/api-docs/volumes.mdx

@@ -278,8 +278,10 @@ $ curl \
 ## Register Volume
 
 This endpoint registers an external volume with Nomad. The volume must exist
-in the external storage provider (see [Create Volume] below). It is an error
-to register an already registered volume.
+in the external storage provider (see [Create Volume] below).
+
+Making the same request again with a higher `RequestedCapacityMin` value
+may trigger a [Volume Expansion][].
 
 | Method | Path                        | Produces           |
 | ------ | --------------------------- | ------------------ |
@@ -344,9 +346,12 @@ $ curl \
 ## Create Volume
 
 This endpoint creates a volume in an external storage provider and registers
-it with Nomad. It is an error to create an already-registered volume. Only CSI
-plugins that implement the [Controller][csi_plugins_internals] interface with
-the `CREATE_DELETE_VOLUME` capability support this endpoint.
+it with Nomad. Only CSI plugins that implement the
+[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME`
+capability support this endpoint.
+
+Making the same request again with a higher `RequestedCapacityMin` value
+may trigger a [Volume Expansion][].
 
 | Method | Path                               | Produces           |
 | ------ | ---------------------------------- | ------------------ |
@@ -781,3 +786,4 @@ $ curl \
 [csi_plugin]: /nomad/docs/job-specification/csi_plugin
 [csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins
 [Create Volume]: #create-volume
+[Volume Expansion]: /nomad/docs/other-specifications/volume#volume-expansion

website/content/docs/commands/volume/register.mdx

@@ -69,9 +69,10 @@ Specification][volume_specification] page.
 
 Not all fields can be updated after the volume is registered:
 
-* The `capacity_min` and `capacity_max` fields can be updated, so long as the
-  volumes existing capacity fits within that range. The actual capacity of the
-  volume is unchanged.
+* The `capacity_min` and `capacity_max` fields can be updated, and may increase
+  the volume size if the CSI plugin supports it. Expansion may or may not be
+  possible while the volume is in use, again depending on the plugin.
+  Reducing volume capacity is not allowed per the CSI spec.
 * The `capability` blocks can be added or removed, but only if the capability is
   not currently in use by a mounted volume.
 * The `mount_options` block can be updated if the volume is not in use.

website/content/docs/commands/volume/status.mdx

@@ -85,6 +85,7 @@ External ID          = vol-23452345
 Plugin ID            = ebs-prod
 Provider             = aws.ebs
 Version              = 1.0.1
+Capacity             = 100 GiB
 Schedulable          = true
 Controllers Healthy  = 1
 Controllers Expected = 1
@@ -107,6 +108,7 @@ External ID          = vol-23452345
 Plugin ID            = ebs-prod
 Provider             = aws.ebs
 Version              = 1.0.1
+Capacity             = 100 GiB
 Schedulable          = true
 Controllers Healthy  = 1
 Controllers Expected = 1

website/content/docs/other-specifications/volume/index.mdx

@@ -106,7 +106,9 @@ parameters {
   the exact behavior is up to the storage provider. If you want to specify an
   exact size, you should set `capacity_min` and `capacity_max` to the same
   value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
-  be supported by all storage providers. Only allowed on **volume creation**.
+  be supported by all storage providers. Increasing this value and re-issuing
+  `volume create` or `volume register` may expand the volume, if the CSI plugin
+  supports it.
 
 - `capacity_max` `(string: <optional>)` - Option for requesting a maximum
   capacity, in bytes. The capacity of a volume may be the physical size of a
@@ -115,7 +117,7 @@ parameters {
   the exact behavior is up to the storage provider. If you want to specify an
   exact size, you should set `capacity_min` and `capacity_max` to the same
   value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
-  be supported by all storage providers. Only allowed on **volume creation**.
+  be supported by all storage providers.
 
 - `capability` <code>([Capability][capability]: &lt;required&gt;)</code> -
   Option for validating the capability of a volume.
@@ -150,9 +152,8 @@ Several fields are set automatically by the plugin when `volume create` or
 `volume register` commands are successful and you should not set their values
 if they are not supported by the operation.
 
-You should not set the [`snapshot_id`](#snapshot_id), [`clone_id`](#clone_id),
-[`capacity_min`](#capacity_min), or [`capacity_max`](#capacity_max) fields on
-**volume registration**.
+You should not set the [`snapshot_id`](#snapshot_id), or [`clone_id`](#clone_id)
+fields on **volume registration**.
 
 And you should not set the [`external_id`](#external_id) or
 [`context`](#context) fields on **volume creation**.
@@ -162,9 +163,10 @@ And you should not set the [`external_id`](#external_id) or
 The `volume register` command allows updating a volume definition. But not all
 fields can be updated after the volume is registered:
 
-* The `capacity_min` and `capacity_max` fields can be updated, so long as the
-  volumes existing capacity fits within that range. The actual capacity of the
-  volume is unchanged.
+* The `capacity_min` and `capacity_max` fields can be updated, and may increase
+  the volume size if the CSI plugin supports it. Expansion may or may not be
+  possible while the volume is in use, again depending on the plugin.
+  Reducing volume capacity is not allowed per the CSI spec.
 * The `capability` blocks can be added or removed, but only if the capability is
   not currently in use by a mounted volume.
 * The `mount_options` block can be updated if the volume is not in use.
@@ -173,6 +175,20 @@ fields can be updated after the volume is registered:
   provided by the CSI plugin, and should not be updated unless recommended by
   the CSI plugin's documentation.
 
+## Volume Expansion
+
+CSI Volumes may be expanded (increased in size) if the CSI controller plugin
+(and node plugin, if required) has the `EXPAND_VOLUME` capability.
+
+To trigger a volume expansion, increase [`capacity_min`](#capacity_min)
+above the current real capacity of the volume (as seen with the
+[`volume status`][] command), and re-issue either [`volume create`][]
+or [`volume register`][].
+
+Nomad will reconcile the requested capacity by issuing expand volume requests
+to the controller plugin, and if required by the controller, also to the
+node plugins for each allocation that has a claim on the volume.
+
 ## Examples
 
 ### Volume registration
@@ -221,6 +237,73 @@ context {
 }
 ```
 
+
+### Example Volume Expansion
+
+Either [`volume create`][] or [`volume register`][] commands can trigger
+an expand to occur, after the volume has already been created or registered.
+
+Example using `volume create`, with a `volume.hcl` file:
+
+```hcl
+id        = "ebs_prod_db1"
+name      = "database"
+type      = "csi"
+plugin_id = "ebs-prod"
+
+capacity_min = "50gib"
+capacity_max = "50gib"
+
+# ... etc ...
+```
+
+Create the volume:
+
+```shell-session
+$ nomad volume create volume.hcl
+Created external volume vol-asdf1234 with ID ebs_prod_db1
+```
+
+See its current capacity with [`volume status`][]:
+
+```shell-session
+$ nomad volume status ebs_prod_db1 | grep Capacity
+Capacity             = 50 GiB
+```
+
+Update `volume.hcl`:
+
+```hcl
+id        = "ebs_prod_db1"
+name      = "database"
+type      = "csi"
+plugin_id = "ebs-prod"
+# all of the above must remain the same
+
+capacity_min = "100gib" # double
+capacity_max = "100gib" # increased to match
+
+# ... etc ...
+```
+
+Run `volume create` again:
+
+```shell-session
+$ nomad volume create volume.hcl
+Created external volume vol-asdf1234 with ID ebs_prod_db1
+```
+
+Check the volume capacity again:
+
+```shell-session
+$ nomad volume status ebs_prod_db1 | grep Capacity
+Capacity             = 100 GiB
+```
+
+If you encounter errors that are not clear from the command output,
+the CSI plugin allocation logs and/or Nomad leader server logs may be
+helpful.
+
 [api_volume_create]: /nomad/api-docs/volumes#create-volume
 [api_volume_register]: /nomad/api-docs/volumes#register-volume
 [capability]: /nomad/docs/other-specifications/volume/capability
@@ -230,3 +313,4 @@ context {
 [topology_request]: /nomad/docs/other-specifications/volume/topology_request
 [`volume create`]: /nomad/docs/commands/volume/create
 [`volume register`]: /nomad/docs/commands/volume/register
+[`volume status`]: /nomad/docs/commands/volume/status