How to process tab-separated files in Spark? -


i have file tab separated. third column should key , entire record should value (as per map reduce concept). 

val ceffile = sc.textfile("c:\\text1.txt") val cefdim1 =  ceffile.filter { line => line.startswith("1") } val joinedrdd = ceffile.map(x => x.split("\\t"))  joinedrdd.first().foreach { println }

i able value of first column not third. can suggest me how accomplish this?

after you've done split x.split("\\t") rdd (which in example called joinedrdd i'm going call parsedrdd since haven't joined yet) going rdd of arrays. turn array of key/value tuples doing parsedrdd.map(r => (r(2), r)). being said - aren't limited map & reduce operations in spark possible data structure might better suited. tab separated files, use spark-csv along spark dataframes if fit eventual problem looking solve.


-

Comments

Post a Comment

Popular posts from this blog

renaming files in a directory using python or R -

i went across similar questions of renaming files in directory using python. i have these files in directory want rename: -statistical_analysis_with_r -statistical_pattern_recognition_3rd_edition -statistical analysis r -statistical pattern recognition 3rd edition for wrote script in windows: def naming(so): import re w=re.split('[ _]+',so) r=[] in w: r.append(i.capitalize()) print(' '.join(r)) import os c in os.listdir(os.getcwd()): if c.endswith(".pdf"): os.rename(c,naming(c)) print(os.listdir(os.getcwd())) but getting error: statistical analysis r.pdf traceback (most recent call last): file "<ipython-input-79-d7f645d6d3e5>", line 4, in <module> os.rename(c,naming(c)) typeerror: rename: can't specify none path argument can what's going on? and doing same thing using r (www.r-project.org)? loads of in advance. your naming functio...
Read more

html - outline-style different in chrome compared to firefox and internet explorer -

seems firefox , ie have default of black dotted outline while chrome has default of light blue solid outline. i tried change doing this: a{ outline-style: dotted; outline-color: black; } this seems catastrophe links got dotted black outline not hovered or focused. terrible because "outline" thing can differentiate mouse focus keyboard key focus. any other idea on how make chrome's outline others ? <!doctype html> <html> <head> <style type="text/css"> body{ font-size: 1.5em; font-family: verdana; } .zone2 a{ outline-width: thick; outline-color: red; } .zone3 a{ outline-style: solid; } </style> </head> <body> outline tests:<br/> <div class="zone1"> <a href="#">test link 1</a><br/> <a href=...
Read more

vbscript - How to batch combine PDFs with sub-folder based non-JavaScript bookmarks -

having been avid reader here many years, have question of own couldn't find answer for. i looking solution, preferably vb scriptable, combines several pdf files different subfolders 1 new pdf , adds bookmarks based on folder structure, i.e. first page originating first file folder gets first level bookmark based on folder name, , second level bookmark based on file name. subsequent files folder second level bookmarks linked under first level bookmark. as added complication, bookmarks must point location in document; must not javascript based , call actions. this should happening on win7 pc. unfortunately don't know how internal workings of pdfs (yet), can't quite believe that problem uncommon isn't solved already!? can pointing me in right direction (or providing script, of course, if it's easy), or confirming above isn't possible, or else, really? thank much! best
Read more