可以使用其他支持镜像sha256哈希值的工具来代替amazon EMR的spark-submit。例如,可以使用docker命令行或kubernetes来提交任务。
代码示例:
docker run -v /usr/local/spark:/usr/local/spark \
-v /path/to/application:/application \
my-spark-app \
/usr/local/spark/bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://localhost:7077 \
--deploy-mode cluster \
--executor-memory 512m \
/application/SparkPi.jar \
10
apiVersion: v1
kind: Pod
metadata:
name: spark-pi
spec:
containers:
- name: spark
image: my-spark-app
command: ["/bin/sh", "-c"]
args: ["/usr/local/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://localhost:7077 --deploy-mode client --executor-memory 512m /application/SparkPi.jar 10"]
volumeMounts:
- name: spark-local
mountPath: /usr/local/spark
- name: application
mountPath: /application
volumes:
- name: spark-local
hostPath:
path: /usr/local/spark
- name: application
hostPath:
path: /path/to/application
上一篇:AmazonEMR-EmrManagedMasterSecurityGroup上缺少端口9443的入站规则。
下一篇:AmazonEMRwithFlink使用旧版commons-math3中的Percentile类,导致NoSuchMethodError错误。