将数据存储在S3中,并使用BigQuery Omni来查询数据。
代码示例:
1.创建存储Bucket:
from google.cloud import storage
storage_client = storage.Client()
bucket_name = ""
bucket = storage_client.create_bucket(bucket_name)
print("Bucket {} created.".format(bucket.name))
2.将数据上传到S3:
from google.cloud import storage
storage_client = storage.Client()
bucket_name = ""
bucket = storage_client.bucket(bucket_name)
file_name = ""
destination_blob_name = ""
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(file_name)
print("File {} uploaded to {}.".format(file_name, destination_blob_name))
3.使用BigQuery Omni来查询数据:
from google.cloud import bigquery
project_id = ""
location = "us"
instance_id = ""
client = bigquery.Client(project=project_id)
table_id = ""
query_string = "SELECT * FROM `{0}.{1}.{2}` LIMIT 100".format(project_id, instance_id, table_id)
df = client.query(query_string).to_dataframe()
print(df.head())