Skip to content

Commit

Permalink
Bigtable plugin filter added
Browse files Browse the repository at this point in the history 
Signed-off-by: Pankaj Kumar <[email protected]>
  • Loading branch information
@pankajkumaribm
pankajkumaribm committed Nov 7, 2024
1 parent b3383e6 commit d25d77a
Showing 1 changed file with 75 additions and 79 deletions.
154 changes: 75 additions & 79 deletions filter-plugin/logstash-filter-pubsub-bigtable-guardium/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,90 +7,89 @@
* Guardium Data Protection: 12.1 and above
* Guardium Insights: 3.3 and above

This is a [Logstash](https://github.com/elastic/logstash) filter plug-in for the universal connector that is featured in IBM Security Guardium. It parses a GCP (Google Cloud Platform) event logs into a [Guardium record](https://github.com/IBM/universal-connectors/blob/main/common/src/main/java/com/ibm/guardium/universalconnector/commons/structures/Record.java) instance (which is a standard structure made out of several parts). The information is then sent over to Guardium. Guardium records include the accessor (the person who tried to access the data), the session, data, and exceptions. If there are no errors, the data contains details about the query "construct". The construct details the main action (verb) and collections (objects) involved. As of now, the BigTable plug-in only supports Guardium Data Protection.
This is a [Logstash](https://github.com/elastic/logstash) filter plug-in for the universal connector that is featured in IBM Security Guardium. It parses GCP (Google Cloud Platform) event logs into a [Guardium record](https://github.com/IBM/universal-connectors/blob/main/common/src/main/java/com/ibm/guardium/universalconnector/commons/structures/Record.java) instance (which is a standard structure made out of several parts). The information is then sent over to Guardium. Guardium records include the accessor (the person who tried to access the data), the session, data, and exceptions. If there are no errors, the data contains details about the query "construct". The construct details the main action (verb) and collections (objects) involved. The Bigtable Logstash filter plug-in supports Guardium Data Protection and Guardium Insights.

### Note
This plug-in contains a runtime dependency of Logstash Google PubSub input plug-in (version ~> 1.2.1, i.e. at least 1.2.1).

This version is compliant with Guardium Data Protection v12.1 and above. Please refer to the [input plug-in repository](https://github.com/IBM/universal-connectors/tree/main/input-plugin/logstash-input-google-pubsub) for more information.
This version is compliant with Guardium Data Protection v12.1 and above. For more information, refer to [input plug-in repository](https://github.com/IBM/universal-connectors/tree/main/input-plugin/logstash-input-google-pubsub).

## Configuring BigTable on GCP
### BigTable Setup
1. [Prerequisites](https://cloud.google.com/bigtable/docs/quickstarts/quickstart-cloud-console)
2. BigTable is automatically enabled in new projects. To activate BigTable in an existing project, enable the BigTable API. Please refer to [Enable the BigTable API](https://console.cloud.google.com/marketplace/product/google/bigtable.googleapis.com) for more information.
3. Create a BigTable instance that stores the data. Please refer to [Create a Dataset](https://cloud.google.com/bigtable/docs/creating-instance) for more information.
4. Create a table. Please refer to [Create Table](https://cloud.google.com/bigtable/docs/samples/bigtable-hw-create-table) for more information.
5. Query table data. Please refer to [Query Table Data](https://cloud.google.com/bigquery/docs/external-data-bigtable) for more information.
2. BigTable is automatically enabled in new projects. To activate BigTable in an existing project, refer to [Enable the BigTable API](https://console.cloud.google.com/marketplace/product/google/bigtable.googleapis.com).
3. Create a BigTable instance that stores the data, refer to [Create a Dataset](https://cloud.google.com/bigtable/docs/creating-instance).
4. Create a table. For more information, refer to [Create Table](https://cloud.google.com/bigtable/docs/samples/bigtable-hw-create-table).
5. Query table data. For more information, refer to [Query Table Data](https://cloud.google.com/bigquery/docs/external-data-bigtable).

### Permissions and Roles Details

##### Grant Permission Required for log view/ download

User can view and download the generated logs.
The following Identity and Access Management roles are required to view and download logs:
You can view and download the generated logs.
You need the following Identity and Access Management (IAM) roles to view and download the generated logs:

* To view logs:
roles/logging.viewer (Logs Viewer)
roles/logging.privateLogViewer (Private Logs Viewer)
* To download logs:
roles/logging.admin (Logging Admin)
roles/logging.viewAccessor (Logs View Accessor)
* View logs:
* roles/logging.viewer (Logs Viewer)
* roles/logging.privateLogViewer (Private Logs Viewer)
* Download logs:
* roles/logging.admin (Logging Admin)
* roles/logging.viewAccessor (Logs View Accessor)

### Create a topic in Pub/Sub
* Go to the Pub/Sub topics page in the Cloud Console. 
### Creating a topic in Pub/Sub
* Go to the Pub/Sub topics page in the Cloud Console.
* Click ```Create a topic```
* In the Topic ID field, provide a unique topic name, for example, MyTopic.
* In the Topic ID field, provide a unique topic name, for example, ```MyTopic```.
* Click ```Create Topic```.

### Create a subscription in Pub/Sub
* Display the menu for the topic created in the previous step and click ```New subscription```.
* Type a name for the subscription, such as ```MySub```.
* Leave the delivery type as Pull.
* Click ```Create```
### Creating a subscription in Pub/Sub
* Display the menu for the topic created in the previous step and click```New subscription```.
* Type a name for the subscription, such as```MySub```.
* Leave the delivery type as```Pull```.
* Click```Create```

### Create a log sink in Pub/Sub
### Creating a log sink in Pub/Sub
* In the Cloud Console, go to the ```Logging > Log Router``` page.
* Click ```Create sink```.
* In the Sink details panel, enter the following details:
* ```Sink name```: Provide an identifier for the sink. Note that after you create the sink you cannot rename it. However, you can delete a sink and create a new one.
* ```Sink description (optional)```: Describe the purpose or use case for the sink.
* In the ```Sink destination``` panel, select the Cloud ```Pub/Sub topic``` as sink service and select the topic created in previous steps.
* Choose logs to include in the sink in the Build inclusion filter panel.
* You can filter the logs by log name, resource, and severity.
Multi-region
* In cases of multiple regions, you need to do the same set of configurations per each region.
* ```Sink name```: Provide an identifier for the sink. Note that after you create the sink you cannot rename it. However, you can delete a sink and create a new one.
* ```Sink description (optional)```: Describe the purpose or use case for the sink.
* In the ```Sink destination``` panel, select the Cloud ```Pub/Sub topic``` as sink service and select the topic created in previous steps.
* Choose logs to include in the sink in the Build inclusion filter panel. You can filter the logs by log name, resource, and severity.
* Multi-region
* In cases of multiple regions, you need to do the same set of configurations per each region.
Based on the region, different configuration files will be used for the input plug-in

#### Set destination (TOPIC & SUBSCRIPTION) permissions
#### Setting permissions for the destination (TOPIC & SUBSCRIPTION)

To set permissions for the log sink to route to its destination, do the following:
* Obtain the sink's writer identity—an email address—from the new sink.
1. Go to the ```Log Router``` page, and select ```menu```  > ```View sink details```.
2. The writer identity appears in the Sink details panel.
* Obtain the sink's writer identity—an email address—from the new sink.
* Go to the```Log Router```page, and select```menu``` >```View sink details```.
* The writer identity appears in the Sink details panel.
* If you have owner access to the destination:
1. Add the sink's writer identity to topic >>>>
- Navigate to the Topic created in the earlier steps
- Click on SHOW INFO panel
- Click on ADD PRINCIPAL
- Paste writer identity in the New Principals
- Give it the Pub/Sub Publisher role and subscriber role
- Navigate to the Topic created in the earlier steps.
- Click SHOW INFO panel.
- Click ADD PRINCIPAL.
- Paste writer identity in the New Principals.
- Give it the Pub/Sub Publisher role and subscriber role.

2. Add the sink's writer identity to subscription >>>>
- Navigate to the Subscription
- Click on SHOW INFO panel
- Click on ADD PRINCIPAL
- Paste writer identity in the New Principals
- Give it the subscriber role

### Create service account credentials
* Go to the Service accounts section of the IAM & Admin console.
* Select ```project``` and click ```Create Service Account```.
* Enter a Service account name, such as Bigtable-pubsub.
* Click ```Create```.
* The owner role is required for the service account. Select the owner role from the drop-down menu
* Click ```Continue```. You do not need to grant users access to this service account.
* Click ```Create Key```. The key is used by the Logstash input plug-in configuration file.
* Select JSON and click ```Create```.
- Navigate to the Subscription.
- Click SHOW INFO panel.
- Click ADD PRINCIPAL.
- Paste writer identity in the New Principals.
- Give it the subscriber role.

### Creating service account credentials
1. Go to the Service accounts section of the IAM & Admin console.
2. Select ```project``` and click ```Create Service Account```.
3. Enter a Service account name, such as Bigtable-pubsub.
4. Click ```Create```.
5. The owner role is required for the service account. Select the owner role from the drop-down list.
6. Click ```Continue```. You do not need to grant users access to this service account.
7. Click ```Create Key```. The key is used by the Logstash input plug-in configuration file.
8. Select JSON and click ```Create```.

### Inclusion Filter
Edit the Sink via *Logs Router > Sink Inclusion Filter*:
Expand All @@ -102,32 +101,29 @@ protoPayload.serviceName="bigtableadmin.googleapis.com" OR protoPayload.serviceN

## Viewing the Audit logs

The inclusion filter mentioned above will be used to view the Audit logs in the GCP Logs Explorer.
The inclusion filter mentioned above is used to view the Audit logs in the GCP Logs Explorer.

### Supported audit logs
1. BigTableAudit - `ACTIVITY`, `DATA_ACCESS` logs
2. BigTable Log - `CREATEINSTANCE`, `DELETEINSTANCE`, `UPDATEINSTANCE`, `CREATECLUSTER`, `DELETECLUSTER`, `UPDATECLUSTER`, `CREATETABLE`, `DELETETABLE`, `MODIFYCOLUMNFAMILIES`, `EXECUTEQUERY`, `LISTCLUSTERS`, `LISTINSTANCES`.

## 4. Limitations
1. If no information regarding certain fields is available in the logs, those fields will not be mapped.
2. Exception object will be prepared based on severity of the logs.
3. The data model size is limited to 10 GB per table. If you have a 100 GB reservation per project per location, BigTable BI Engine limits the reservation per table to 10 GB. The rest of the available reservation is used for other tables in the project.
4. BigTable cannot read the data in parallel if you use gzip compression. Loading compressed JSON data into BigTable is slower than loading uncompressed data.
5. You cannot include both compressed and uncompressed files in the same load job.
6. JSON data must be newline delimited. Each JSON object must be on a separate line in the file.
7. The maximum size for a gzip file is 4 GB.
8. Log messages have a size limit of 100K bytes
9. The Audit/Data access log doesn't contain a server IP. The default value is set 0.0.0.0 and can be in IPV4 or IPV6 format.
10. The following important fields cannot be mapped, as there is no information regarding these fields in the logs:
* BigTableAudit - `ACTIVITY`, `DATA_ACCESS` logs
* BigTable Log - `CREATEINSTANCE`, `DELETEINSTANCE`, `UPDATEINSTANCE`, `CREATECLUSTER`, `DELETECLUSTER`, `UPDATECLUSTER`, `CREATETABLE`, `DELETETABLE`, `MODIFYCOLUMNFAMILIES`, `EXECUTEQUERY`, `LISTCLUSTERS`, `LISTINSTANCES`.

## Limitations
* Exception object is prepared based on severity of the logs.
* The data model size is limited to 10 GB per table. If you have a 100 GB reservation per project per location, BigTable BI Engine limits the reservation per table to 10 GB. The rest of the available reservation is used for other tables in the project.
* BigTable cannot read the data in parallel if you use gzip compression. Loading compressed JSON data into BigTable is slower than loading uncompressed data.
* You cannot include both compressed and uncompressed files in the same load job.
* JSON data must be newline delimited. Each JSON object must be on a separate line in the file.
* The maximum size for a gzip file is 4 GB.
* Log messages have a size limit of 100K bytes.
* The Audit/Data access log doesn't contain a server IP. The default value is set to 0.0.0.0 and can be in IPV4 or IPV6 format.
* The following important fields cannot be mapped, as there is no information regarding these fields in the logs:
- Source program
- OS User
- Client HostName
11. `serverHostName` pattern for BigTable GCP :
`project-id_bigtableadmin.googleapis.com` | `project-id_bigtable.googleapis.com`.
12. While using GCP, excluding the web UI, duplicate entries may appear in both the reports and audit logs.
13. Bigtable uses two different service names (`bigtable.googleapis.com` & `bigtableadmin.googleapis.com`) depending on the tasks being performed. This will result in two distinct S-TAP Host entries.
14. Multiple sessions from the same Bigtable instance may result in multiple S-TAP entries.
15. The BigTable audit log doesn’t include login failed logs, so these will not appear in the guardium LOGIN_FAILED report.
- Client HostName
* While using GCP, duplicate entries may appear in both the reports and audit logs.
* Bigtable uses two different service names (`bigtable.googleapis.com` & `bigtableadmin.googleapis.com`) depending on the tasks being performed. This results in two distinct S-TAP host entries.
* Multiple sessions from the same Bigtable instance may result in multiple S-TAP entries.
* The BigTable audit log doesn’t include login failed logs. So, these logs do not appear in the guardium LOGIN_FAILED report.

## Configuring the BigTable filter in Guardium
The Guardium universal connector is the Guardium entry point for native audit/data_access logs. The Guardium universal connector identifies and parses the received events, and converts them to a standard Guardium format. The output of the Guardium universal connector is forwarded to the Guardium sniffer on the collector, for policy and auditing enforcements. Configure Guardium to read the native audit/data_access logs by customizing the BigTable template.
Expand All @@ -138,15 +134,15 @@ The Guardium universal connector is the Guardium entry point for native audit/da
* Download the [logstash-filter-big_table_guardium_filter](gdp-pubsub-bigtable-package/logstash-filter-big_table_guardium_filter.zip) plug-in.

### Procedure
1. On the collector, go to Setup > Tools and Views > Configure Universal Connector.
1. On the collector, go to ```Setup``` > ```Tools and Views``` > ```Configure Universal Connector```.
2. Enable the universal connector if it is disabled.
3. Click Upload File and select the offline [logstash-filter-big_table_guardium_filter](gdp-pubsub-bigtable-package/logstash-filter-big_table_guardium_filter.zip) plug-in. After it is uploaded, click ```OK```.
3. Click ```Upload File``` and select the offline [logstash-filter-big_table_guardium_filter](gdp-pubsub-bigtable-package/logstash-filter-big_table_guardium_filter.zip) plug-in. After it is uploaded, click ```OK```.
4. Click ```Upload File``` and select the key.json file. After it is uploaded, click ```OK```.
5. Click the Plus sign to open the Connector Configuration dialog box.
6. Type a name in the Connector name field.
7. Update the input section to add the details from the [pubsub_big_table.conf](gdp-pubsub-bigtable-package/pubsub_big_table.conf) file's input part, omitting the keyword "input{" at the beginning and its corresponding "}" at the end.
8. Update the filter section to add the details from the [pubsub_big_table.conf](gdp-pubsub-bigtable-package/pubsub_big_table.conf) file's filter part, omitting the keyword "filter{" at the beginning and its corresponding "}" at the end.
9. The 'type' fields should match in the input and filter configuration sections. This field should be unique for every individual connector added.
10. Click ```Save```. Guardium validates the new connector and displays it in the Configure Universal Connector page.
11. After the offline plug-in is installed and the configuration is uploaded and saved in the Guardium machine, restart the Universal Connector using the Disable/Enable button.
11. After the offline plug-in is installed and the configuration is uploaded and saved in the Guardium machine, restart the Universal Connector using the ```Disable/Enable``` button.

0 comments on commit d25d77a

Please sign in to comment.