在Apache Beam 2.39.0的DataFlow runner中使用BigQuery Storage API时,可能会遇到权限拒绝的问题。这是因为默认情况下,DataFlow runner使用Google Cloud Storage API而不是BigQuery Storage API,因此需要进行以下更改:
org.apache.beam
beam-sdks-java-io-google-cloud-platform
2.39.0
PCollection rows = pipeline.apply(BigQueryIO.readTableRows()
.from("my-project:my_dataset.my_table")
.usingStandardSql()
.withMethod(Method.DIRECT_READ)
.withStorageReadOptions(new StorageReadOptions()
.withFormatOptions(new AvroOptions())
.withSelectedFields(Arrays.asList("field1", "field2"))))
请注意,使用BigQuery Storage API需要在Google Cloud Console中启用BigQuery Storage API并授权相应的IAM权限。
上一篇:ApacheBeam2.37.0移除了GoogleCloudBigQuery依赖项
下一篇:ApacheBeam:IllegalStateException-Valueonlyavailableatruntimeafterupgradingtobeam2.41.0