apache spark - Exception: could not open socket on pyspark -


whenever trying execute simple processing in pyspark, fails open socket.

>>> myrdd = sc.parallelize(range(6), 3) >>> sc.runjob(myrdd, lambda part: [x * x x in part])

above throws exception -

port 53554 , proto 6 , sa ('127.0.0.1', 53554) traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/volumes/work/bigdata/spark-custom/python/pyspark/context.py", line 917, in runjob     return list(_load_from_socket(port, mappedrdd._jrdd_deserializer))   file "/volumes/work/bigdata/spark-custom/python/pyspark/rdd.py", line 143, in _load_from_socket     raise exception("could not open socket") exception: not open socket  >>> 15/08/30 19:03:05 error pythonrdd: error while sending iterator java.net.sockettimeoutexception: accept timed out     @ java.net.plainsocketimpl.socketaccept(native method)     @ java.net.abstractplainsocketimpl.accept(abstractplainsocketimpl.java:404)     @ java.net.serversocket.implaccept(serversocket.java:545)     @ java.net.serversocket.accept(serversocket.java:513)     @ org.apache.spark.api.python.pythonrdd$$anon$2.run(pythonrdd.scala:613)

i checked through rdd.py _load_from_socket , realised gets port , server not started or sp runjob might issue-

port = self._jvm.pythonrdd.runjob(self._jsc.sc(), mappedrdd._jrdd, partitions)

its not ideal solution aware of cause. pyspark unable create jvm socket jdk 1.8 (64-bit) version, set java path jdk 1.7 , worked.


-

Comments

Post a Comment

Popular posts from this blog

c# - Binding a comma separated list to a List<int> in asp.net web api -

the default binder in web api expecst http://url.com/webapi/report/?pageids=3243&pageids=2365 to bind public ihttpactionresult report(list<int> pageids){ // exciting webapi code} i wish bind http://url.com/webapi/report/?pageids=3243,2365 as running out of space in url get . i have created class public class commaseparatedmodelbinder : system.web.http.modelbinding.imodelbinder { public bool bindmodel(httpactioncontext actioncontext, modelbindingcontext bindingcontext) { //binding in here } } and registered in webapiconfig.cs var provider = new simplemodelbinderprovider( typeof(list<int>), new commaseparatedmodelbinder()); config.services.insert(typeof(modelbinderprovider), 0, provider); i have altered method signature use model binder so public ihttpactionresult report( [modelbinder] list<int> pageids){ // exciting webapi code} however break point in binder not being
Read more

Delphi 7 and decode UTF-8 base64 -

in delphi 7, have widestring encoded base64(that received web service widestring result) : pd94bwwgdmvyc2lvbj0ims4wij8+dqo8c3ryaw5nptiq2lpyqjwvc3ryaw5npg== when decoded it, result not utf-8: <?xml version="1.0"?> <string>طھط³طھ</string> but when decoded base64decode.org, result true : <?xml version="1.0"?> <string>تست</string> i have use encddecd unit decodestring function. the problem have using decodestring . function, in delphi 7, treats decoded binary data being ansi encoded. , problem text utf-8 encoded. to continue encddecd unit have couple of options. can switch decodestream . instance, code produce utf-8 encoded text file data: {$apptype console} uses classes, encddecd; const data = 'pd94bwwgdmvyc2lvbj0ims4wij8+dqo8c3ryaw5nptiq2lpyqjwvc3ryaw5npg=='; var input: tstringstream; output: tfilestream; begin input := tstringstream.create(data); try output := tfilestream.
Read more

html - Is there any way to exclude a single element from the style? (Bootstrap) -

so, have img logo, overlaid on div styled background-image: url("...") , , both elements inside 2 div classes .to-color , .to-overlay respectively, black transparent overlay. is there way exclude logo style given these 2 classes? here's jsfiddle example: link html , css example: - html - <div class="to-color"> <div class="to-overlay"> <div class="respo"> <div class="row"> <div class="col-xs-4 col-xs-offset-4 text-center"> <img src="http://ansonalex.com/wp-content/uploads/2011/06/google-logo-768x260.png" alt="..." class="img-responsive"> </div> </div> </div> </div> </div> - css - .to-color { background-color: #000; } .to-overlay { opacity : 0.5; } .respo { background-image: url("https://source.unsplash.com/category/objects/1600x900"); background-position: center center;
Read more