問題描述
java.io.IOException：作業失敗！使用 hadoop‑0.19.1 在我的 osx 上運行示例應用程序時 (java.io.IOException: Job failed! when running a sample app on my osx with hadoop‑0.19.1)
bash‑3.2$ echo $JAVA_HOME /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home bash‑3.2$ bin/hadoop dfs ‑copyFromLocal conf /user/yokkom/input2 bash‑3.2$ bin/hadoop jar hadoop‑*‑examples.jar grep input2 output 'dfs[a‑z.]+' 09/04/17 10:09:32 INFO mapred.FileInputFormat: Total input paths to process : 10 09/04/17 10:09:33 INFO mapred.JobClient: Running job: job_200904171309_0001 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at org.apache.hadoop.examples.Grep.run(Grep.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.Grep.main(Grep.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:165) at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Do anyone have any idea why this happens? The same job runs perfectly well on linux machines. And, after "Job failed" happens, the whole Hadoop cluster stops responding.
My MacOS version is 10.5.6.
EDIT The same result for hadoop‑0.20.1
‑‑‑‑‑
參考解法
方法 1:
I googled for it and the same error (at the exact same line) has happened before. Read about it here and here.
The second link suggests the following:
- Delete conf/log4j.properties to see a detailed report
- Try to increase heap memory (using ‑Xms and ‑Xmx switches)
方法 2:
Yeah grep can suck up memory, especially on large data sets. How big is your input? BTW, there's also Perm memory.
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
There's a few interesting switches to perform custom behaviour on OutOfMemory however I suspect the jobRunner is just catching everything before it gets to the VM. You may need another way to debug it.
(by yogman、kgiannakakis、SpliFF)