apache spark - Exception: could not open socket on pyspark -


whenever trying execute simple processing in pyspark, fails open socket.

>>> myrdd = sc.parallelize(range(6), 3) >>> sc.runjob(myrdd, lambda part: [x * x x in part])

above throws exception -

port 53554 , proto 6 , sa ('127.0.0.1', 53554) traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/volumes/work/bigdata/spark-custom/python/pyspark/context.py", line 917, in runjob     return list(_load_from_socket(port, mappedrdd._jrdd_deserializer))   file "/volumes/work/bigdata/spark-custom/python/pyspark/rdd.py", line 143, in _load_from_socket     raise exception("could not open socket") exception: not open socket  >>> 15/08/30 19:03:05 error pythonrdd: error while sending iterator java.net.sockettimeoutexception: accept timed out     @ java.net.plainsocketimpl.socketaccept(native method)     @ java.net.abstractplainsocketimpl.accept(abstractplainsocketimpl.java:404)     @ java.net.serversocket.implaccept(serversocket.java:545)     @ java.net.serversocket.accept(serversocket.java:513)     @ org.apache.spark.api.python.pythonrdd$$anon$2.run(pythonrdd.scala:613)

i checked through rdd.py _load_from_socket , realised gets port , server not started or sp runjob might issue-

port = self._jvm.pythonrdd.runjob(self._jsc.sc(), mappedrdd._jrdd, partitions)

its not ideal solution aware of cause. pyspark unable create jvm socket jdk 1.8 (64-bit) version, set java path jdk 1.7 , worked.


-

Comments

Post a Comment

Popular posts from this blog

c# - Binding a comma separated list to a List<int> in asp.net web api -

the default binder in web api expecst http://url.com/webapi/report/?pageids=3243&pageids=2365 to bind public ihttpactionresult report(list<int> pageids){ // exciting webapi code} i wish bind http://url.com/webapi/report/?pageids=3243,2365 as running out of space in url get . i have created class public class commaseparatedmodelbinder : system.web.http.modelbinding.imodelbinder { public bool bindmodel(httpactioncontext actioncontext, modelbindingcontext bindingcontext) { //binding in here } } and registered in webapiconfig.cs var provider = new simplemodelbinderprovider( typeof(list<int>), new commaseparatedmodelbinder()); config.services.insert(typeof(modelbinderprovider), 0, provider); i have altered method signature use model binder so public ihttpactionresult report( [modelbinder] list<int> pageids){ // exciting webapi code} however break point in binder not being ...
Read more

spring - "Dispatcher has no subscribers" during startup -

i facing problem happens , hard reproduce. therefore have problems creating compelling test case. our setup looks this: spring integration spring boot rabbitmq listener custom bus manages transaction aware messages the setup not newest anymore , lot of code replaced more idiomatic spring wiring. though exception getting when service started: dispatcher has no subscribers, failedmessage=genericmessage [payload=com.foobar.service.greencheckout.message.greenorderpropertyaddedmessage@2ba0efb2, headers={id=f750a792-6b01-16d3-8206-6e553a03f8fa, type=com.foobar.service.greencheckout.message.greenorderpropertyaddedmessage, amqp_deliverymode=persistent, timestamp=1505797680967}], failedmessage=genericmessage [payload=com.foobar.service.greencheckout.message.greenorderpropertyaddedmessage@2ba0efb2, headers={id=f750a792-6b01-16d3-8206-6e553a03f8fa, type=com.foobar.service.greencheckout.message.greenorderpropertyaddedmessage, amqp_deliverymode=persistent, timestamp=1505797680967...
Read more

how to prompt save As Box in Excel Interlop c# MVC 4 -

i want export excel sheet using excel interloop... i have datatable convert excel sheet... now there way save file using save dialogue box in mvc 4 here code public string datatabletoexcel(datatable dt, string htmlheading, string key) { try { microsoft.office.interop.excel.application excel; microsoft.office.interop.excel.workbook excelworkbook; microsoft.office.interop.excel.worksheet excelsheet; microsoft.office.interop.excel.range excelcellrange; excel = new microsoft.office.interop.excel.application(); excel.visible = false; excel.displayalerts = false; excelworkbook = excel.workbooks.add(type.missing); excelsheet = (microsoft.office.interop.excel.worksheet)excelworkbook.activesheet; excelsheet.name = "sheet1"; dataset ...
Read more