@hadoopusers

Страница 101 из 101
 
Grigory
17.05.2018
15:32:38
мож и правда чтото не докинуто

Polina
17.05.2018
15:36:01
в логах нашлось такое 18/05/17 18:12:24 DEBUG Shell: Failed to detect a valid hadoop home directory java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:302) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:327) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:601) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:64) at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:50) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:49) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:387) at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:385) at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:385) at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:410) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764) at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:787) at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) 18/05/17 18:12:24 DEBUG Shell: setsid exited with exit code 0опять хомяк хадупа не виден у меня он задан в /usr/local/spark/conf/spark-env.shнаверное это что-то не то, да? ? и дальше много такого 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:36 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ...172.18.0.5 - ip контейнера

сейчас поменяю бридж на хост, но у меня не всегда компоуз рестартится)

Admin


Google
Евгений
17.05.2018
16:06:46
в логах нашлось такое 18/05/17 18:12:24 DEBUG Shell: Failed to detect a valid hadoop home directory java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:302) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:327) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:601) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:64) at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:50) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:49) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:387) at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:385) at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:385) at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:410) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764) at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:787) at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) 18/05/17 18:12:24 DEBUG Shell: setsid exited with exit code 0опять хомяк хадупа не виден у меня он задан в /usr/local/spark/conf/spark-env.shнаверное это что-то не то, да? ? и дальше много такого 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:35 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ... 18/05/17 18:12:36 ERROR ApplicationMaster: Failed to connect to driver at 172.18.0.5:45867, retrying ...172.18.0.5 - ip контейнера
Я в такие моменты начинаю втыкать параметры во все конфиги, в которые их можно воткнуть, а потом, если заработало, то убираю по одному, пока не перестанет работать)

Страница 101 из 101