Big Data Processing

How to store connections?

for connection create a path with connection name and put the json with proper key and value for connection. for example, default bigquery connection. it would look like this.

How to store variables?

for variables. in the json key would alway be value. as shows below mount point: airflow path: dev-composer/variables/raw_project_name { "value": "raw_project_id" } Leave a Comment

What Are Partition Columns?

In BigQuery, tables can be partitioned based on particular column. Partitioning involves dividing a large table into smaller, more manageable pieces or partitions. Each partition contains a subset of the data and is stored separately. The key idea behind partitioning is to allow BigQuery to scan and process only the partitions that are relevant to a query, rather than scanning the entire table.

Why Are Partition Columns Important?

1. Query Performance Partitioning a table based on a meaningful column, such as a timestamp or date, can significantly speed up query execution. When you run a query that includes a filter condition on the partitioning column, BigQuery can prune irrelevant partitions, scanning only the data that meets the criteria. This reduces query execution time, especially when dealing with large datasets.