在EMR中使用Spark时,可能会出现以下报错信息:“java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path”。这是因为缺少Snappy本地库的原因。可以通过以下步骤解决:
#!/bin/bash
SNAPPY_DL_URL=https://github.com/google/snappy/releases/download/1.1.7.5/snappy-1.1.7.5.tar.gz SNAPPY_INSTALL_DIR=/usr/local/snappy
yum install -y autoconf automake libtool gcc-c++ make
cd /tmp curl -L ${SNAPPY_DL_URL} -o snappy.tar.gz tar zxf snappy.tar.gz cd snappy-1.1.7.5 ./autogen.sh ./configure --prefix=${SNAPPY_INSTALL_DIR} make sudo make install
sudo sh /mnt/snappy_install.sh
--conf spark.executorEnv.LD_LIBRARY_PATH=/usr/local/snappy/lib/ --conf spark.driverEnv.LD_LIBRARY_PATH=/usr/local/snappy/lib/