Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added new documentation for HPE COSI Driver. #225

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+725 −4
Open

Added new documentation for HPE COSI Driver. #225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions docs/cosi_driver/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Overview

The HPE COSI Driver for Kubernetes is deployed by using industry standard means, a Helm chart.

[TOC]

## Delivery Vehicles

Helm is currently the only supported delivery vehicle for the deployment of the HPE COSI Driver.

### Helm

[Helm](https://helm.sh) is the package manager for Kubernetes. Software is being delivered in a format designated as a "chart". Helm is a [standalone CLI](https://helm.sh/docs/intro/install/) that interacts with the Kubernetes API server using your `KUBECONFIG` file.

The official [Helm chart](https://github.com/hpe-storage/co-deployments/tree/master/helm) for the HPE COSI Driver for Kubernetes is hosted on [Artifact Hub](https://artifacthub.io/packages/helm/hpe-storage/hpe-cosi-driver). The chart supports Helm 3 from version 1.0.0 of the HPE COSI Driver. In an effort to avoid duplicate documentation, please see the chart for instructions on how to deploy the COSI driver using Helm.

- Go to the chart on [Artifact Hub](https://artifacthub.io/packages/helm/hpe-storage/hpe-cosi-driver).

## Add an HPE Storage Backend

Once the COSI driver is deployed, you must create a `Secret` with the following details before you can use the [COSI API resources](using.md).

### Secret Parameters

All parameters are mandatory and described below.

| Parameter | Description |
| ------------------- | ------------|
| accessKey | The access key of the S3 user with bucket creation, bucket-tagging and deletion permissions.
| secretKey | The secret key for the S3 user who has bucket creation, bucket-tagging and deletion permissions.
| endpoint | The S3 frontend network DNS subdomains address of the backend object storage system; that is, an HPE Alletra Storage MP X10000 system.
| glcpUserClientId | The HPE Green Lake API client ID.
| glcpUserSecretKey | The HPE Green Lake API client secret.
| dsccZone | The fully qualified domain name (FQDN) of the HPE Data Services Cloud Console (DSCC) zone.
| clusterSerialNumber | The backend storage system cluster serial number.

!!! note
The Kubernetes compute nodes where the HPE COSI Driver is allowed to run need to be able to access the DSCC zone specified.

Example `Secret` manifest named "hpe-object-backend.yaml":

```yaml fct_label="HPE COSI Driver v1.0.0"
apiVersion: v1
kind: Secret
metadata:
name: hpe-object-backend
namespace: default
stringData:
accessKey: testuser
secretKey: testkey
endpoint: http://192.168.1.100:8080
glcpUserClientId: 00000000-0000-0000-0000-000000000000
glcpUserSecretKey: 00000000000000000000000000000000
dsccZone: common.cloud.hpe.com
clusterSerialNumber: 0000000000
```

Create the `Secret`.

```text
kubectl create -f hpe-object-backend.yaml
```

!!! tip "See Also"
The COSI source code repository contains a parameterized script that can assist in creating a correctly formatted `Secret`. See [github.com/hpe-storage/cosi-driver/scripts/cosi_secret](https://github.com/hpe-storage/cosi-driver/tree/master/scripts/cosi_secret) for more details.

### Creating the S3 user, GLCP user and locating the S3 endpoint, DSCC zone and cluster serial number

1. To create the S3 user:
* Follow the steps in the HPE documentation to [create an access policy](https://support.hpe.com/hpesc/docDisplay?docId=sd00004219en_us&page=objstr_access_policies_create_dscc.html).
- Choose _All Buckets_.
- Add custom actions for the policy: `CreateBucket`, `DeleteBucket`, `PutBucketTagging`.
* To create the user, refer to the [HPE documentation](https://support.hpe.com/hpesc/docDisplay?docId=sd00004219en_us&page=objstr_users_create_dscc.html) for this purpose and select the access policy created in the previous step.
- Save the user name and password. These will be used as the S3 access key and S3 secret key respectively in the COSI secret.
2. To create the HPE Green Lake API client ID and secret, refer to the following [HPE documentation](https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/).
3. To locate the S3 endpoint:
* Log into Data Services Cloud Console.
* Select _Data Ops Manager_.
* From the menu on the left, select _Systems_. From the list click on the name of the system you want to use for COSI operations.
* Click on the _Networking_ tab. Under the _Frontend Network_ section, save the value of the _Network DNS Subdomains_ field.
* The S3 endpoint can be constructed from the _Network DNS Subdomains_ value by using the format: `http://<Network DNS Subdomains>`.
4. To locate the DSCC zone and cluster serial number of the HPE Alletra Storage MP X10000 system, refer to the [HPE documentation](https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&page=GUID-616CE4D4-C31A-4BFE-8F41-887C2B0B9046.html).

!!! tip
In a real world scenario it's more practical to name the `Secret` something that makes sense for the organization. It could be the hostname of the backend or the role it carries; i.e., "hpe-alletra-sanjose-prod".

## Next Steps

Next you need to create [a BucketClass](using.md#configure_a_bucketclass).
165 changes: 165 additions & 0 deletions docs/cosi_driver/diagnostics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Introduction

It's recommended to familiarize yourself with inspecting workloads on Kubernetes. This [cheat sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods) is very useful, and you should have it readily available.

## Sanity Checks

Once the COSI driver has been deployed using Helm, the list of pods shown below is representative of what a healthy system would look like after installation. If any of the workload deployments lists anything but `Running`, proceed to inspect the logs for the problematic workload.

```text
kubectl get pods
NAME READY STATUS RESTARTS AGE
hpe-cosi-provisioner-559b458788-5jc56 2/2 Running 0 8d
objectstorage-controller-7dff56f8fc-r5tdq 1/1 Running 0 8d
```

Once the COSI API objects `BucketClaim`, `Bucket` and `BucketAccess` have been created in the cluster, use the `kubectl describe` command to check the status and events, which will have information useful to help diagnose any issue. If any of the resources' statuses shows anything but `true` or if there are any warning events, inspect the logs of the COSI driver and sidecar.

```text
kubectl describe bucketclaim my-first-bucketclaim
Name: my-first-bucketclaim
Namespace: default
Labels: <none>
Annotations: <none>
API Version: objectstorage.k8s.io/v1alpha1
Kind: BucketClaim
Metadata:
Creation Timestamp: 2024-09-17T04:34:55Z
Finalizers:
cosi.objectstorage.k8s.io/bucketclaim-protection
Generation: 1
Resource Version: 110670
UID: 99dde004-4f8d-4a20-900b-e5d61e3facb9
Spec:
Bucket Class Name: hpe-standard-object
Protocols:
s3
Status:
Bucket Name: bc199dde004-4f8d-4a20-900b-e5d61e3facb9
Bucket Ready: true
Events: <none>
```

```text
kubectl describe bucket bc199dde004-4f8d-4a20-900b-e5d61e3facb9
Name: bc199dde004-4f8d-4a20-900b-e5d61e3facb9
Namespace:
Labels: <none>
Annotations: <none>
API Version: objectstorage.k8s.io/v1alpha1
Kind: Bucket
Metadata:
Creation Timestamp: 2024-09-17T04:34:55Z
Finalizers:
cosi.objectstorage.k8s.io/bucket-protection
cosi.objectstorage.k8s.io/bucketaccess-bucket-protection
Generation: 1
Resource Version: 111014
UID: e7fb69f6-c20e-4abf-a522-f1ae765b3b6a
Spec:
Bucket Claim:
Name: my-first-bucketclaim
Namespace: default
UID: 99dde004-4f8d-4a20-900b-e5d61e3facb9
Bucket Class Name: hpe-standard-object
Deletion Policy: Delete
Driver Name: cosi.hpe.com
Parameters:
Bucket Tags: mytag=myvalue, mytag2=, mytag3=myvalue3,
Cosi User Secret Name: hpe-object-backend
Cosi User Secret Namespace: default
Protocols:
s3
Status:
Bucket ID: bc199dde004-4f8d-4a20-900b-e5d61e3facb9
Bucket Ready: true
Events: <none>
```

```text
kubectl describe bucketaccess hpe-standard-access
Name: hpe-standard-access
Namespace: default
Labels: <none>
Annotations: <none>
API Version: objectstorage.k8s.io/v1alpha1
Kind: BucketAccess
Metadata:
Creation Timestamp: 2024-09-17T04:38:00Z
Finalizers:
cosi.objectstorage.k8s.io/bucketaccess-protection
Generation: 1
Resource Version: 111017
UID: 37fd517f-27ac-45fa-b369-57e645967366
Spec:
Bucket Access Class Name: hpe-standard-access
Bucket Claim Name: my-first-bucket-claim
Credentials Secret Name: my-first-access-secret
Protocol: s3
Status:
Access Granted: true
Account ID: ba-37fd517f-27ac-45fa-b369-57e645967366
Events: <none>
```

## Logging

Log files associated with the HPE COSI Driver posts data to the standard output stream and can be accessed using options under [kubectl logs](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/).

If the logs need to be retained for long term, use a standard logging solution for Kubernetes, such as Fluentd. Alternatively, it's advised to use the log collector script, at regular intervals to preserve logs.

### COSI Driver Logs

```text
kubectl logs -f deploy/hpe-objectstorage-provisioner -c hpe-cosi-driver
```

### COSI Sidecar Logs

```text
kubectl logs -f deploy/hpe-objectstorage-provisioner -c hpe-cosi-provisioner-sidecar
```

### COSI Controller Logs

```text
kubectl logs -f deploy/objectstorage-controller
```

### Log Level of Sidecar

You can control the log level for the COSI Sidecar using the `.containers.sideCar.verbosityLevel` field in [`values.yaml`](https://github.com/hpe-storage/cosi-driver/tree/master/helm/values.yaml) of the Helm chart. The values are generally small positive integers.

```yaml
containers:
sideCar:
# verbosityLevel specifies the verbosity of the logs that will be printed by the sidecar container
# Small postive integer values are generally recommended.
# Ref.: https://pkg.go.dev/k8s.io/klog/v2#V
verbosityLevel: 5
```

### Log Collector

The log collector script `hpe-logcollector.sh` can be used to collect the logs from any node that has `kubectl` access to the cluster. Please see the script's associated [documentation](https://github.com/hpe-storage/cosi-driver/tree/master/scripts/log_collection) for more details on usage and troubleshooting.

Download the script and provide execute permissions:

```text
wget https://raw.githubusercontent.com/hpe-storage/cosi-driver/main/scripts/log_collection/hpe-logcollector.sh
chmod +x hpe-logcollector.sh
```

Usage:

```text
./hpe-logcollector.sh -h
Collect HPE storage diagnostic logs using kubectl.

Usage:
hpe-logcollector.sh [-h|--help] [-n|--namespace NAMESPACE]
Options:
-h|--help Print this usage text
-n|--namespace NAMESPACE Collect logs from HPE COSI Driver Deployment in Namespace
NAMESPACE (default: default)
```
Binary file added docs/cosi_driver/img/cosi_driver_architecture-1.0.0.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
84 changes: 84 additions & 0 deletions docs/cosi_driver/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Introduction

A Container Object Storage Interface ([COSI](https://github.com/kubernetes-sigs/container-object-storage-interface-spec)) Driver for HPE Alletra Storage MP X10000. The HPE COSI Driver for Kubernetes allows you to use the HPE Object Storage Provider (OSP) to perform bucket management operations on storage resources. The COSI architecture allows you to integrate HPE Alletra Storage MP X10000 Object Storage with a COSI-compatible containerized application running on the Kubernetes cluster. The COSI driver follows the gRPC [specification](https://github.com/kubernetes-sigs/container-object-storage-interface/blob/main/proto/cosi.proto) provided by Kubernetes.

![HPE COSI Driver for Kubernetes architecture](img/cosi_driver_architecture-1.0.0.png)

!!! tip
The HPE COSI Driver for Kubernetes is vendor-specific and works only with the HPE Alletra Storage MP X10000 OSP.

## Table of Contents

[TOC]

## Features and Capabilities

Below is the official table for COSI features that HPE has officially tested and validated against the [platform matrix](#compatibility_and_support).

| Feature | K8s maturity | Since K8s version | HPE COSI Driver |
|----------------------------------------|--------------|-------------------|-----------------|
| Bucket Creation | Alpha | 1.25 | 1.0.0 |
| Bucket Deletion | Alpha | 1.25 | 1.0.0 |
| Bucket Tagging | Alpha | 1.25 | 1.0.0 |
| Granting Bucket Access | Alpha | 1.25 | 1.0.0 |
| Revoking Bucket Access | Alpha | 1.25 | 1.0.0 |

Refer to the [official table](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) of feature gates in the Kubernetes docs to determine the availability of alpha features. File any issues, questions or feature requests [here](https://github.com/hpe-storage/cosi-driver/issues). You may also join the HPE Slack community to chat with people close to this project on the `#Alletra` and `#Kubernetes` channels. Sign up at [slack.hpedev.io](https://slack.hpedev.io/) and log in at [hpedev.slack.com](https://hpedev.slack.com).

!!! tip
Familiarize yourself with the basic requirements below for running the COSI driver on your Kubernetes cluster before you install it. HPE strongly recommends you install the COSI driver with a [Helm chart](deployment.md#helm).

## Compatibility and Support

HPE has tested the following combinations and included them as part of the official support services for the first COSI driver release.

<a name="latest_release"></a>
### HPE COSI Driver for Kubernetes v1.0.0

Release highlights:

* Support for Kubernetes v1.25 to v1.31.
* Implementation of bucket creation, configuration (bucket tagging), lifecycle and access management.
* A log collector script that can be used to collect logs from any node.

<table>
<tr>
<th>Kubernetes</th>
<td>v1.25-v1.31</td>
</tr>
<tr>
<th>Helm Chart</th>
<td><a href="https://artifacthub.io/packages/helm/hpe-storage/hpe-cosi-driver/1.0.0">v1.0.0</a> on ArtifactHub</td>
</tr>
<tr>
<th>Platforms</th>
<td>
HPE Alletra Storage MP X10000
</td>
</tr>
<tr>
<th>HPE Alletra Storage MP X10000 OS</th>
<td>
R1
</td>
</tr>
<tr>
<th>Protocols</th>
<td>S3</td>
</tr>
<tr>
<th>Release&nbsp;notes</th>
<td><a href="https://github.com/hpe-storage/cosi-driver/tree/master/release-notes/v1.0.0.md">v1.0.0</a> on GitHub</td>
</tr>
</table>

### Release Archive

HPE does not currently have any archived releases of the HPE COSI Driver.

## Known Limitations

* The HPE COSI driver can be deployed only in the `default` namespace due to a [bug](https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar/issues/140) in creating events in the COSI API objects when deployed in non-default namespaces.
* Creating `BucketClaim` or `BucketAccess` objects in parallel can cause failures in the COSI driver. A [bug](https://github.com/kubernetes-sigs/container-object-storage-interface-api/issues/101) has been filed to address this issue.
* A warning event is created in the `Bucket` or `BucketAccess` resources when an error occurs, and has a life-span of one hour. During this period, if the error is resolved the Status will show `Bucket Ready: true` or `Access Granted: true` in the `Bucket` or `BucketAccess` respectively, but the warning event will persist till an hour lapses. A [bug](https://github.com/kubernetes-sigs/container-object-storage-interface-api/issues/103) has been raised to resolve this ambiguity.
* Recreation of `BucketClaim` or `BucketAccess` objects doesn't work intermittently, as gRPC request is not sent to the COSI driver. This [pull request](https://github.com/kubernetes-retired/container-object-storage-interface-api/pull/86) will address the issue.
Loading