What is Google BigQuery?
Google BigQuery is a fully managed and serverless data warehouse cloud-based solution available on the Google Cloud Platform. It offers rapid SQL queries and interactive analysis of massive datasets in a matter of seconds.
Because of its columnar storage paradigm and tree architecture model, data scanning, querying and aggregating results is now much faster and significantly easier. BigQuery serverless design is highly scalable thanks with fast deployment cycle and on-demand pricing.
Big Query utilizes batch ingest, enabling fast loading of thousands of data points without high amount of existing computational resources. Furthermore, the real-time ingest mechanism allows on-demand queries and analytics by loading up to 100,000 rows of data for instant access.
Connecting a Google BigQuery Destination
- An existing Google Cloud Platform (GCP) project with the following setup:
- Enabled billing for the GCP project.
- An existing Google BigQuery instance in the GCP project
- Permissions in the GCP project that allow you to create Identity Access Management (IAM) service accounts.
Step 1: Enable Google BigQuery API
- GCP web console -> API & Services -> Library
- Search for BigQuery API -> click Enable
Step 2: Authenticate with a service account
It is recommended to use
target-bigquery with a service account.
Create a service account credentials as following:
- API & Services -> Credentials -> Create Credentials -> Service account
- Under Service account details, enter Service account name. Click Create
- Under Grant this service account access to the project, select BigQuery Data Editor and BigQuery Job User as the minimal set of permissions. Select Done
- BigQuery Data Editor permission allows to the service account to access (and change) the data.
- BigQuery Job User permission allows the service account to actually run a load or select job.
- On the API & Services Credentials screen, click on the service account you just created.
- Click ADD KEY -> Create new key -> JSON key. Download the service account credential JSON file.
Name the file client_secrets.json. You can place the file where
target-bigquerywill be executed or provide a path to the service account json file.
Set a GOOGLE_APPLICATION_CREDENTIALS environment variable on the machine, where the value is the fully qualified path to client_secrets.json file:
- Creating an environment variable on a Windows 10 machine
- Creating an environment variable on a Mac machine