python - spark submit的时候怎么导入自己写的依赖模块?
怪我咯
怪我咯 2017-04-17 17:23:34
0
1
866

python代码中的import

from spark_learning.utils.default_utils import setDefaultEncoding,initSparkContext,ensureOffset

submit命令:

bin/spark-submit --jars /home/jabo/software/spark-1.5.2-bin-hadoop2.6/lib/spark-streaming-kafka-assembly_2.10-1.5.2.jar\
/home/jabo/spark-by-python/spark_learning/third_day/streaming_kafka_avg.py\
--py-files /home/jabo/spark-by-python/spark_learning/utils/default_utils.py

官网解释:

For Python applications, simply pass a .py file in the place of <application-jar> instead of a JAR, and add Python .zip, .egg or .py files to the search path with --py-files.

但是会报错,找不到import模块:

Traceback (most recent call last):
  File "/home/jabo/spark-by-python/spark_learning/third_day/streaming_kafka_avg.py", line 10, in <module>
    import spark_learning.utils.default_utils
ImportError: No module named spark_learning.utils.default_utils

如何解决??

怪我咯
怪我咯

走同样的路,发现不同的人生

热门教程
更多>
最新下载
更多>
网站特效
网站源码
网站素材
前端模板
关于我们 免责声明 Sitemap
PHP中文网:公益在线PHP培训,帮助PHP学习者快速成长!