Google BigQuery

What is Google BigQuery?

Google BigQuery is a fully managed and serverless data warehouse cloud-based solution available on the Google Cloud Platform. It offers rapid SQL queries and interactive analysis of massive datasets in a matter of seconds.

Because of its columnar storage paradigm and tree architecture model, data scanning, querying and aggregating results is now much faster and significantly easier. BigQuery serverless design is highly scalable thanks with fast deployment cycle and on-demand pricing.

Big Query utilizes batch ingest, enabling fast loading of thousands of data points without high amount of existing computational resources. Furthermore, the real-time ingest mechanism allows on-demand queries and analytics by loading up to 100,000 rows of data for instant access.

Connecting a Google BigQuery Destination

Prerequisites

  1. An existing Google Cloud Platform (GCP) project with the following setup:
    • Enabled billing for the GCP project.
    • An existing Google BigQuery instance in the GCP project
  2. Permissions in the GCP project that allow you to create Identity Access Management (IAM) service accounts.

 

Step 1: Enable Google BigQuery API

  1. GCP web console -> API & Services -> Library

Big Query API and Service Library

 

  1. Search for BigQuery API -> click Enable

 

Search for Big Query

Enable BigQuery API

 

Step 2: Authenticate with a service account

It is recommended to use target-bigquery with a service account.

Create a service account credentials as following:

  1. API & Services -> Credentials -> Create Credentials -> Service account

 

Create Service account

  1. Under Service account details, enter Service account name. Click Create

BigQuery Service Account Name

 

 

  1. Under Grant this service account access to the project, select BigQuery Data Editor and BigQuery Job User as the minimal set of permissions. Select Done
  • BigQuery Data Editor permission allows to the service account to access (and change) the data.
  • BigQuery Job User permission allows the service account to actually run a load or select job.

 

BigQuery Service account Access

 

 

  1. On the API & Services Credentials screen, click on the service account you just created.

 

BigQuery Select Service Account

 

  1. Click ADD KEY -> Create new key -> JSON key. Download the service account credential JSON file.

 

BigQuery Add Key

 

BigQuery JSON Key

 

 

  1. Name the file client_secrets.json. You can place the file where target-bigquery will be executed or provide a path to the service account json file.

  2. Set a GOOGLE_APPLICATION_CREDENTIALS environment variable on the machine, where the value is the fully qualified path to client_secrets.json file:

 

Source: https://github.com/adswerve/target-bigquery