升级AWS Glue版本。Spline Lineage支持AWS Glue 5.0及以上版本。如果您的AWS Glue版本低于5.0,您需要升级AWS Glue。
配置Glue作业启用Spline Lineage。在AWS Glue控制台中,将“Spline Lineage跟踪”选项设置为“启用”,以启用Spline Lineage功能。您还需要在您的Glue作业定义中包括以下选项:
import com.hotels.beans.BeanUtils;
import com.hotels.beans.BeanUtilsBean;
import com.hotels.transformer.TransformContext;
import com.hotels.transformer.logger.LogAppender;
import com.hotels.transformer.logger.TransformerLoggerFactory;
public class MyJobClass {
private static final com.hotels.transformer.logger.TransformerLogger log = TransformerLoggerFactory.getLogger(MyJobClass.class);
public void execute() {
log.info("Start executing...");
// Your processing code here
log.info("Processing finished.");
}
}
确认Glue作业已生成Spline事件。您可以通过AWS Glue控制台查看Spline事件,以确认Glue作业是否已成功生成Spline事件。如果您发现作业没有生成Spline事件,则需要检查您的Glue作业定义,确保已正确配置Spline Lineage选项和日志记录选项。
检查您的Spline Lineage设置是否正确。如果您已经升级了AWS Glue并且已启用Spline Lineage选项,但仍然无法使用Spline Lineage,请检查您的Spline Lineage设置是否正确。您可以通过以下代码示例来确认您的Spline Lineage设置是否正确:
import com.hotels.spline.api.client.SplineClient;
import com.hotels.spline.api.client.SplineClientConfiguration;
String baseUrl = "http://localhost:8080/api"; // or a host and port where Spline UI is running
SplineClient splineClient = new SplineClientConfiguration(baseUrl).newClient();
String jobName = "my-glue-job";
List eventIds = splineClient.getEventsByJobNames(Collections.singleton(jobName)).stream().map(event -> event.getId()).collect(Collectors.toList());
if (eventIds.isEmpty()) {
System.out.println(String.format("Job %s has not produced any events", jobName));
} else {
System.out.println(String.format("Job %s has produced %d events", jobName, eventIds.size()));
}