Skip to content

Latest commit

 

History

History
18 lines (10 loc) · 1.74 KB

README.md

File metadata and controls

18 lines (10 loc) · 1.74 KB

bq-steampipe

The goal of this is to create a way of being able to query Google Cloud information in BigQuery through Steampipe. Steampipe is an open source tool that allows you to query your deployed cloud resources using SQL. Their webpage gives a lot more information on what it does and how it works, but the gist of it is that it allows you to do things like select * from gcp_compute_instance across all your cloud resources.

Deployed Assets

Architecture Diagram

At its core, it's just three main things. The Steampipe service is a command line tool, that will be deployed in a Google Compute Engine virtual machine.

Steampipe will be running in service mode, which will expose a Postgres endpoint. This will stand up a CloudSQL for Postgres instance which will connect to the Steampipe service using the foreign data wrapper.

Google BigQuery will then use the external data source feature to connect to Cloud SQL.

More details on how these different components is in the following digram.

Data Connection Diagram

Go here for a writeup detailing how this all works, and a step by step guide on how to deploy this.