Skip to content

Connecting steampipe.io to Google BigQuery via CloudSQL for PostgreSQL.

License

Notifications You must be signed in to change notification settings

gray-host/steampipe

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

58 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

bq-steampipe

The goal of this is to create a way of being able to query Google Cloud information in BigQuery through Steampipe. Steampipe is an open source tool that allows you to query your deployed cloud resources using SQL. Their webpage gives a lot more information on what it does and how it works, but the gist of it is that it allows you to do things like select * from gcp_compute_instance across all your cloud resources.

Deployed Assets

Architecture Diagram

At its core, it's just three main things. The Steampipe service is a command line tool, that will be deployed in a Google Compute Engine virtual machine.

Steampipe will be running in service mode, which will expose a Postgres endpoint. This will stand up a CloudSQL for Postgres instance which will connect to the Steampipe service using the foreign data wrapper.

Google BigQuery will then use the external data source feature to connect to Cloud SQL.

More details on how these different components is in the following digram.

Data Connection Diagram

Go here for a writeup detailing how this all works, and a step by step guide on how to deploy this.

About

Connecting steampipe.io to Google BigQuery via CloudSQL for PostgreSQL.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HCL 93.8%
  • Shell 6.2%