Spark java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dposv 异常解决办法

今天在跑一个Job的时候,遇到了如下异常:
15/01/05 16:54:06 INFO mapreduce.TableInputFormatBase: Input split length: 889 M bytes.
15/01/05 16:55:07 WARN scheduler.TaskSetManager: Lost TID 339 (task 74.0:5)
15/01/05 16:55:07 WARN scheduler.TaskSetManager: Loss was due to java.lang.UnsatisfiedLinkError
java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dposv(CII[DII[DII)I
	at org.jblas.NativeBlas.dposv(Native Method)
	at org.jblas.SimpleBlas.posv(SimpleBlas.java:369)
	at org.jblas.Solve.solvePositive(Solve.java:68)
	at org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateBlock$2.apply(ALS.scala:522)
	at org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateBlock$2.apply(ALS.scala:509)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:156)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.mutable.ArrayOps$ofInt.map(ArrayOps.scala:156)
	at org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommendation$ALS$$updateBlock(ALS.scala:509)
	at org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:445)
	at org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:444)
	at org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
	at org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:156)
	at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:154)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:154)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:31)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.Task.run(Task.scala:51)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
经查证后发现,Spark MLib是依赖于jblas这个库的,需要手动在每个节点上安装。

debian/ubuntu use: sudo apt-get install libgfortran3
for centos use: sudo yum install gcc-gfortran

参考以下资料:
1、Spark官网

Dependencies

MLlib uses the jblas linear algebra library, which itself depends on native Fortran routines. You may need to install the gfortran runtime library if it is not already present on your nodes. MLlib will throw a linking error if it cannot detect these libraries automatically.

To use MLlib in Python, you will need NumPy version 1.7 or newer and Python 2.7.

2、Github:http://stackoverflow.com/questions/24758314/apache-spark-mllib-collaborative-filtering
 

(责任编辑:)

分享到:

更多
发表评论
请自觉遵守互联网相关的政策法规,严禁发布色情、暴力、反动的言论。
评价:
表情:
  • 微笑/wx
  • 撇嘴/pz
  • 抓狂/zk
  • 流汗/lh
  • 大兵/db
  • 奋斗/fd
  • 疑问/yw
  • 晕/y
  • 偷笑/wx
  • 可爱/ka
  • 傲慢/am
  • 惊恐/jk
用户名: 验证码:点击我更换图片
资料下载专区
图文资讯

网络安全:互联网安全 如何防范木马及病毒

网络安全:互联网安全 如何防范木马及病毒的攻击

一、计算机中毒有哪些症状? 人生病了,会有各种各样的症状,同样,电脑中毒了也会有...[详细]

加强网站服务器安全维护的技巧

加强网站服务器安全维护的技巧

计算机系统服务器的维护工作十分重要,稍有不慎就会使整个网络陷入瘫痪。目前,网络经...[详细]

电脑离线就安全?这个软件一样能远程窃取数

电脑离线就安全?这个软件一样能远程窃取数据

以色列的一个研究团队已改良了窃取离线电脑数据的方法。人们一直认为这种离线电脑在面...[详细]

如何在Linux上最妥善地管理加密密钥?

如何在Linux上最妥善地管理加密密钥?

存储SSH加密密钥和牢记密码可能是一件让人很头痛的事儿。不过遗憾的是,在如今恶意黑...[详细]

如何在 Linux 上运行命令前临时清空 Bash

如何在 Linux 上运行命令前临时清空 Bash 环境变量

我是个 bash shell 用户。我想临时清空 bash shell 环境变量。但我不想删除或者 unset...[详细]

返回首页 返回顶部