python - Spark worker keep removing and adding executors -

-

i tried build spark cluster using local ubuntu virtual machine master, , remote ubuntu virtual machine worker. local virtual machine running in virtualbox, make accessible remote guest, forwarded virtual machine's 7077 port host's 7077 port. start master by:

./sbin/start-master.sh -h 0.0.0.0 -p 7077

i made listening on 0.0.0.0, because if use default 127.0.1.1, remote guest won't able connect it. start worker executing following command on remote machine:

./bin/spark-class org.apache.spark.deploy.worker.worker    spark://129.22.151.82:7077

the worker able connect master, can seen on ui: screen shot

then tried run "pi" example python code:

from pyspark import sparkcontext, sparkconf conf=sparkconf().setappname("pi").setmaster("spark://0.0.0.0:7077) sc=sparkcontext(conf=conf)

.... once run it, program never stops, noticed program removing , adding executors, because executors exits error code 1. , executor's stderr:

    using spark's default log4j profile: org/apache/spark/log4j-  defaults.properties     16/02/25 13:22:22 info coarsegrainedexecutorbackend:  registered signal handlers [term, hup, int]     16/02/25 13:22:22 warn nativecodeloader: unable load  native-hadoop library platform... using builtin-java  classes applicable     16/02/25 13:22:23 info securitymanager: changing view acls to: kxz138,adminuser     16/02/25 13:22:23 info securitymanager: changing modify acls to: kxz138,adminuser     16/02/25 13:22:23 info securitymanager: securitymanager:  authentication disabled; ui acls disabled; users view  permissions: set(kxz138, adminuser); users modify permissions:  set(kxz138, adminuser)     **16/02/25 13:22:23 error usergroupinformation:  priviledgedactionexception as:adminuser (auth:simple)  cause:java.io.ioexception: failed connect /10.0.2.15:34935     exception in thread "main" java.io.ioexception: failed connect /10.0.2.15:34935**     @ org.apache.spark.network.client.transportclientfactory.createclient(transportclientfactory.java:216)     @ org.apache.spark.network.client.transportclientfactory.createclient(transportclientfactory.java:167)     @ org.apache.spark.rpc.netty.nettyrpcenv.createclient(nettyrpcenv.scala:200)     @ org.apache.spark.rpc.netty.outbox$$anon$1.call(outbox.scala:187)

i noticed error here network problem. worker trying access 10.0.2.15 local nat ip address of virtual machine, failed. error never occurs when deploy worker local computer. has idea why error occurs? why worker trying access ip address 10.0.2.15 instead of public ip?

btw, i've set key-less ssh access master slave.


Comments

Post a Comment

Popular posts from this blog

java - Suppress Jboss version details from HTTP error response -

-
my jboss verson jboss eap 6.3 when kind of error response sent jboss, adds server information response. following sample row response: http/1.1 400 bad request server: apache-coyote/1.1 content-type: text/html;charset=utf-8 content-length: 2391 date: tue, 19 jul 2016 09:30:46 gmt connection: close jboss web/7.4.8.final-redhat-4 - jbweb000064: error report jbweb000065: http status 400 - org.codehaus.jackson.map.jsonmappingexception: can not construct instance of mybean.type string value 'xyz': value not 1 of declared enum instance names @ [source: org.jboss.resteasy.core.interception.messagebodyreadercontextimpl$inputstreamwrapper@5e421c2b; line: 14, column: 26] (through reference chain: requestargs["request"]) jbweb000309: type jbweb000067: status report jbweb000068: message org.codehaus.jackson.map.jsonmappingexception: can not construct instance of mybean.type string value 'xyz': value not 1 of declared enum instance names @ [source: org.jboss...
Read more

gridview - Yii2 DataPorivider $totalSum for a column -

-
i have order , ordersearch models. in order list (actionindex) gridview there filtering , sorting. on of columns in order order_total (sum of products in order). i need implement sum of order_total in gridview. if manually counting activedataprovider->getmodels() array_map spend 3 seconds 3000 orders (localhost). don't want miss time. i see 2 ways make faster: create cache every filter , update lifetime (horrible) the interesting can right in ordersearch->search() method $query method. don't understand how can pass in controller. example of code 2nd way: class ordersearch extends order { public $totalsum; public function search($params) { $query = order::find(); $dataprovider = new activedataprovider([ 'query' => $query, ]); $this->load($params); $this->totalsum = $query->sum('order_total'); // works fast return $dataprovider; } } after i...
Read more

Sass watch command compiles .scss files before full sftp upload -

-
i use winscp download , edit .scss files , sass on linux (on server) compile them .css . after saving file, use: sass scss/style.scss css/style.css , replaces css file compiled sccs. the problem i want skip part return command line after editing scss file, sake of automation , saving time. but, if use watch command: sass --watch scss/style.scss:css/style.css synchronize 2 files, alerts nonexistent css errors pop up: change detected to: scss/style.scss error scss/style.scss (line 232: invalid css after "...ht: bold; line-": expected "{", "") note that when file uploads quicker usual (sometimes happens) watch command job, no errors. this because, when file upload takes time, sass compiles scss file (on server) before uploaded remote folder. thus, compiles part of file, resulting in css errors. is there way set timer watch command waits few seconds after detects changes , before compiling? any other way overcome is, of course, ...
Read more