{"id":414,"date":"2017-08-05T22:31:31","date_gmt":"2017-08-05T14:31:31","guid":{"rendered":"http:\/\/vinta.ws\/code\/?p=414"},"modified":"2026-03-17T01:19:15","modified_gmt":"2026-03-16T17:19:15","slug":"spark-troubleshooting","status":"publish","type":"post","link":"https:\/\/vinta.ws\/code\/spark-troubleshooting.html","title":{"rendered":"Spark troubleshooting"},"content":{"rendered":"

Apache Spark 2.x Troubleshooting Guide
\nhttps:\/\/www.slideshare.net\/jcmia1\/a-beginners-guide-on-troubleshooting-spark-applications<\/a>
\nhttps:\/\/www.slideshare.net\/jcmia1\/apache-spark-20-tuning-guide<\/a><\/p>\n

Check your cluster UI to ensure that workers are registered and have sufficient resources<\/h2>\n
PYSPARK_DRIVER_PYTHON=\"jupyter\" \nPYSPARK_DRIVER_PYTHON_OPTS=\"notebook --ip 0.0.0.0\" \npyspark \n--packages \"org.xerial:sqlite-jdbc:3.16.1,com.github.fommil.netlib:all:1.1.2\" \n--driver-memory 4g \n--executor-memory 20g \n--master spark:\/\/TechnoCore.local:7077<\/code><\/pre>\n
TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<\/code><\/pre>\n
\u53ef\u80fd\u662f\u4f60\u6307\u5b9a\u7684 --executor-memory<\/code> \u8d85\u904e\u4e86 worker \u7684 memory\u3002<\/p>\n
\u4f60\u53ef\u4ee5\u5728 Spark Master UI http:\/\/localhost:8080\/<\/a> \u770b\u5230\u5404\u500b worker \u7e3d\u5171\u6709\u591a\u5c11 memory \u53ef\u4ee5\u7528\u3002\u5982\u679c\u6bcf\u53f0 worker \u53ef\u4ee5\u7528\u7684 memory \u5bb9\u91cf\u4e0d\u540c\uff0cSpark \u5c31\u53ea\u6703\u9078\u64c7\u90a3\u4e9b memory \u5927\u65bc --executor-memory<\/code> \u7684 workers\u3002<\/p>\n
ref:
\nhttps:\/\/spoddutur.github.io\/spark-notes\/distribution_of_executors_cores_and_memory_for_spark_application<\/a><\/p>\n
SparkContext was shut down<\/h2>\n
ERROR Executor: Exception in task 1.0 in stage 6034.0 (TID 21592)\njava.lang.StackOverflowError\n...\nERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(55,1494185401195,JobFailed(org.apache.spark.SparkException: Job 55 cancelled because SparkContext was shut down))<\/code><\/pre>\n
\u53ef\u80fd\u662f executor \u7684\u8a18\u61b6\u9ad4\u4e0d\u5920\uff0c\u5c0e\u81f4 Out Of Memory (OOM) \u4e86\u3002<\/p>\n
ref:
\nhttp:\/\/stackoverflow.com\/questions\/32822948\/sparkcontext-was-shut-down-while-running-spark-on-a-large-dataset<\/a><\/p>\n
Container exited with a non-zero exit code 56 (or some other numbers)<\/h2>\n
WARN org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1504241464590_0001_01_000002 on host: albedo-w-1.c.albedo-157516.internal. Exit status: 56. Diagnostics: Exception from container-launch.\nContainer id: container_1504241464590_0001_01_000002\nExit code: 56\nStack trace: ExitCodeException exitCode=56:\n    at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)\n    at org.apache.hadoop.util.Shell.run(Shell.java:869)\n    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)\n    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)\n    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)\n    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)\n    at java.util.concurrent.FutureTask.run(FutureTask.java:266)\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n    at java.lang.Thread.run(Thread.java:748)\n\nContainer exited with a non-zero exit code 56<\/code><\/pre>\n
\u53ef\u80fd\u662f executor \u7684\u8a18\u61b6\u9ad4\u4e0d\u5920\uff0c\u5c0e\u81f4 Out Of Memory (OOM) \u4e86\u3002<\/p>\n
ref:
\nhttp:\/\/stackoverflow.com\/questions\/39038460\/understanding-spark-container-failure<\/a><\/p>\n
Exception in thread "main" java.lang.StackOverflowError<\/h2>\n
Exception in thread \"main\" java.lang.StackOverflowError\n    at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)\n    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)\n    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)\n    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)\n    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)\n    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)\n    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)\n    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)\n    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)\n    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)\n    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)\n    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)\n    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)\n    at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:468)\n    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\n    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n    at java.lang.reflect.Method.invoke(Method.java:498)\n    ...<\/code><\/pre>\n
\u89e3\u6c7a\u8fa6\u6cd5\uff1a<\/p>\n
import org.apache.spark.ml.recommendation.ALS\nimport org.apache.spark.sql.SparkSession\n\nval spark: SparkSession = SparkSession.builder().getOrCreate()\nval sc = spark.sparkContext\nsc.setCheckpointDir(\".\/spark-data\/checkpoint\")\n\n\/\/ \u56e0\u70ba sc.setCheckpointDir() \u5c31\u6703\u555f\u7528 checkpoint \u4e86\n\/\/ \u6240\u4ee5\u53ef\u4ee5\u4e0d\u7528\u7279\u5225\u6307\u5b9a checkpointInterval\nval als = new ALS()\n  .setCheckpointInterval(2)<\/code><\/pre>\n
ref:
\nhttps:\/\/stackoverflow.com\/questions\/31484460\/spark-gives-a-stackoverflowerror-when-training-using-als<\/a>
\nhttps:\/\/stackoverflow.com\/questions\/35127720\/what-is-the-difference-between-spark-checkpoint-and-persist-to-a-disk<\/a><\/p>\n
Randomness of hash of string should be disabled via PYTHONHASHSEED<\/h2>\n
\u89e3\u6c7a\u8fa6\u6cd5\uff1a<\/p>\n
$ cd $SPARK_HOME\n$ cp conf\/spark-env.sh.template conf\/spark-env.sh\n$ echo \"export PYTHONHASHSEED=42\" >> conf\/spark-env.sh<\/code><\/pre>\n
ref:
\nhttps:\/\/issues.apache.org\/jira\/browse\/SPARK-13330<\/a><\/p>\n
It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion<\/h2>\n
Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.<\/code><\/pre>\n
\u56e0\u70ba spark.sparkContext<\/code> \u53ea\u80fd\u5728 driver program \u88e1\u5b58\u53d6\uff0c\u4e0d\u80fd\u88ab worker \u5b58\u53d6\uff08\u4f8b\u5982\u90a3\u4e9b\u4e1f\u7d66 RDD \u57f7\u884c\u7684 lambda function \u6216\u662f UDF \u5c31\u662f\u5728 worker \u4e0a\u57f7\u884c\u7684\uff09\u3002<\/p>\n
ref:
\nhttps:\/\/spark.apache.org\/docs\/latest\/rdd-programming-guide.html#passing-functions-to-spark<\/a>
\nhttps:\/\/engineering.sharethrough.com\/blog\/2013\/09\/13\/top-3-troubleshooting-tips-to-keep-you-sparking\/<\/a><\/p>\n
Spark automatically creates closures:<\/p>\n