generative-ai-cdk-constructs: VectorIndex delete fails

[waltmayf]

Describe the bug

When you delete a bedrock knowledge base backed by an AOSS cluster, the delete fails. with an authorization exception.

Expected Behavior

The cloudformation stack should delete successfully

Current Behavior

Deleting the stack currently causes the errors below:

Error in cloudformatino during stack deletion:

...
amplify-xxxxxx |  62 | 12:53:54 PM | DELETE_IN_PROGRESS   | AWS::OpenSearchServerless::Collection | MaintKBKBVectorsVectorCollection327A668A 
amplify-xxxxx |  63 | 12:54:00 PM | DELETE_COMPLETE      | AWS::OpenSearchServerless::Collection | MaintKBKBVectorsVectorCollection327A668A 
amplify-axxxxx |  63 | 12:54:01 PM | DELETE_IN_PROGRESS   | AWS::OpenSearchServerless::SecurityPolicy | MaintKBKBVectorsEncryptionPolicy675BF4F1 
amplify-xxxxx |  63 | 12:54:01 PM | DELETE_IN_PROGRESS   | AWS::OpenSearchServerless::SecurityPolicy | MaintKBKBVectorsNetworkPolicy96A8D1C0 
amplify-xxxxx |  64 | 12:54:02 PM | DELETE_COMPLETE      | AWS::OpenSearchServerless::SecurityPolicy | MaintKBKBVectorsNetworkPolicy96A8D1C0 
amplify-xxxxx |  65 | 12:54:02 PM | DELETE_COMPLETE      | AWS::OpenSearchServerless::SecurityPolicy | MaintKBKBVectorsEncryptionPolicy675BF4F1 
amplify-xxxxx |  72 | 12:58:33 PM | DELETE_FAILED        | AWS::CloudFormation::CustomResource | MaintKBKBIndexVectorIndexBFEA7945 Received response status [FAILED] from custom resource. Message returned: Error: RetryError[<Future at 0x7f0ee6977a40 state=finished raised AuthorizationException>]

Logs: /aws/lambda/amplify-xxxx-OpenSearchIndexCRProvide-M183aNHQjKYl

    at invokeUserFunction (/var/task/framework.js:2:6)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async onEvent (/var/task/framework.js:1:369)

Error in OpenSearchIndexCRProvide lambda cloudwatch logs

INIT_START Runtime Version: python:3.12.v38 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:xxxxx

START RequestId: b7b4dc0e-27f6-4c38-9661-c530007a4015 Version: $LATEST

[WARNING]	2025-01-14T16:55:10.881Z	b7b4dc0e-27f6-4c38-9661-c530007a4015	PUT https://xxxxxx.us-east-1.aoss.amazonaws.com:443/bedrock-knowledge-base-default-index?wait_for_active_shards=all [status:403 request:0.075s]

[ERROR]	2025-01-14T16:55:10.881Z	b7b4dc0e-27f6-4c38-9661-c530007a4015	Error creating index bedrock-knowledge-base-default-index

[ERROR]	2025-01-14T16:55:10.882Z	b7b4dc0e-27f6-4c38-9661-c530007a4015	AuthorizationException(403, 'security_exception', 'OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]')
Traceback (most recent call last):
  File "/var/task/custom_resources/opensearch_index.py", line 218, in handle_create
    create_index(client, index_name, mapping, setting)
  File "/var/task/custom_resources/opensearch_index.py", line 186, in create_index
    client.indices.create(
  File "/var/task/opensearchpy/client/utils.py", line 176, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/opensearchpy/client/indices.py", line 244, in create
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/opensearchpy/transport.py", line 457, in perform_request
    raise e
  File "/var/task/opensearchpy/transport.py", line 418, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/opensearchpy/connection/http_requests.py", line 238, in perform_request
    self._raise_error(
  File "/var/task/opensearchpy/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]')

Reproduction Steps

1. Create a knowledge base using @cdklabs/generative-ai-cdk-constructs

import { bedrock as cdkLabsBedrock } from '@cdklabs/generative-ai-cdk-constructs';

const maintenanceKnowledgeBase = new cdkLabsBedrock.KnowledgeBase(scope, `MaintenanceKB`, {
        embeddingsModel: cdkLabsBedrock.BedrockFoundationModel.TITAN_EMBED_TEXT_V2_1024,
        instruction: `You are a helpful question answering assistant. You answer user questions factually and honestly related to industrial facility maintenance and operations`,
        description: 'Maintenance Knowledge Base',
    });

2. Delete the stack

Possible Solution

The aoss SecurityPolicy deletes successfully before the custom resource fails. Maybe putting a dependency so that the custom resource will delete before the SecurityPolicy will allow the custom resource to delete successfully? You could also just ignore this error on deletion.

Additional Information/Context

No response

CDK CLI Version

2.173.1 (build 4eac959)

Framework Version

@cdklabs/[email protected]

Node.js Version

v20.18.1

OS

Mac

Language

Typescript

Language Version

TypeScript (5.5.3)

Region experiencing the issue

us-east-1

Code modification

No

Other information

No response

Service quota

• I have reviewed the service quotas for this construct

[krokoko]

Hi @waltmayf , thank you for reporting this issue. I will try to reproduce it on my end and update the ticket here.

[krokoko]

Hi @waltmayf , I am not able to reproduce the issue. Created a new stack in TS using the latest version of the library (0.1.289):

export class BedrockAgentStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const maintenanceKnowledgeBase = new bedrock.KnowledgeBase(this, `MaintenanceKB`, {
      embeddingsModel: bedrock.BedrockFoundationModel.TITAN_EMBED_TEXT_V2_1024,
      instruction: `You are a helpful question answering assistant. You answer user questions factually and honestly related to industrial facility maintenance and operations`,
      description: 'Maintenance Knowledge Base',
  });
  }
}

Creation and destruction are working as expected:

Follow up:

• are you able to consistently reproduce this error ? If yes, is there a sample stack you could share with us to help reproducing and fixing the issue ?

[krokoko]

Thanks Walt for providing the stack. @dineshSajwan will have a look