python - Process json data using Pyspark -


i building python script executed through apache spark in generating rdd json file stored on s3 bucket. need filter json rdd according data in json document , thereby generating new json file consist of filtered json documents.that json file needs uploaded s3 bucket. please suggest me appropriate solution implementation through pyspark.

input json

{ "_id" : objectid("55a787ee9efccaeb288b457f"), "data" : {     "n◦ categoria" : 102.0,      "nombre categoria" : "gaseosas",      "variable" : "top of heart",      "var." : "toh",      "marca" : "coca cola zero",      "mes" : "enero",      "mes_n" : 1.0,      "aÑo" : 2014.0,      "universo_total" : 1.0433982e7,      "universo_femenino" : 5529024.0,      "universo_masculino" : 4904958.0,      "porcentaje_total" : 0.0066,      "porcentaje_femenino" : 0.0125,      "porcentaje_masculino" : null },  "app_id" : objectid("5376349e11bc073138c33163"),  "category" : "excel_rac",  "subcategory" : "rac",  "created_time" : numberlong(1437042670),  "instance_id" : null,  "metric_date" : numberlong(1437042670),  "campaign_id" : objectid("5386602ba102b6cd4528ed93"),  "datasource_id" : objectid("559f5c8f9efccacf0a3c9875"),  "duplicate_id" : "695a3f5f562d0a02f1820fe5d91642a5" }

the input json data needs filtered according variable : "top of heart" , there generate output json following

output json 

{   "_id" : objectid("55b5d19e9efcca86118b45a2"),  "widget_type" : "rac_toh_excel",  "campaign_id" : objectid("558554b29efccab00a3c987c"),  "datasource_id" : objectid("55b5d18f9efcca770b3c986a"),  "date_time" : numberlong(1388530800),  "data" : {     "key" : "coca cola zero",      "values" : {         "x" : numberlong(1388530800),          "y" : 1.0433982e7,          "data" : {             "id" : objectid("553a151e5c93ffe0408b46f9"),              "month" : 1.0,              "year" : 2014.0,              "total" : 1.0433982e7,              "variable" : "toh",              "total_percentage" : 0.0066         }     } },  "filter" : [  ] }


-

Comments

Post a Comment

Popular posts from this blog

c# - Binding a comma separated list to a List<int> in asp.net web api -

the default binder in web api expecst http://url.com/webapi/report/?pageids=3243&pageids=2365 to bind public ihttpactionresult report(list<int> pageids){ // exciting webapi code} i wish bind http://url.com/webapi/report/?pageids=3243,2365 as running out of space in url get . i have created class public class commaseparatedmodelbinder : system.web.http.modelbinding.imodelbinder { public bool bindmodel(httpactioncontext actioncontext, modelbindingcontext bindingcontext) { //binding in here } } and registered in webapiconfig.cs var provider = new simplemodelbinderprovider( typeof(list<int>), new commaseparatedmodelbinder()); config.services.insert(typeof(modelbinderprovider), 0, provider); i have altered method signature use model binder so public ihttpactionresult report( [modelbinder] list<int> pageids){ // exciting webapi code} however break point in binder not being ...
Read more

how to prompt save As Box in Excel Interlop c# MVC 4 -

i want export excel sheet using excel interloop... i have datatable convert excel sheet... now there way save file using save dialogue box in mvc 4 here code public string datatabletoexcel(datatable dt, string htmlheading, string key) { try { microsoft.office.interop.excel.application excel; microsoft.office.interop.excel.workbook excelworkbook; microsoft.office.interop.excel.worksheet excelsheet; microsoft.office.interop.excel.range excelcellrange; excel = new microsoft.office.interop.excel.application(); excel.visible = false; excel.displayalerts = false; excelworkbook = excel.workbooks.add(type.missing); excelsheet = (microsoft.office.interop.excel.worksheet)excelworkbook.activesheet; excelsheet.name = "sheet1"; dataset ...
Read more

xslt 1.0 - How to access or retrieve mets content of an item from another item? -

my use case have item has link in item. example, item 123456789/152 has metadata field dc.relation.hasversion=123456789/717 . within item view of 123456789/152 , how can retrieve values of metadata of 123456789/717 ? example, want retrieve dc.language.iso value of 123456789/717 123456789/152 . i looked @ related items feature in dspace don't know how metadata displayed in related items list pulled items in list. i using dspace version 5.3 mirage 2 theme . edit based on schweerelos answer, below actual code. using custom metadata field dc.relation.languageversion . basically, want link other version instead of displaying value, display dc.language.iso of other version in item-view.xsl . i've incorporated answer of schweerelos question in code just displaying value of dc.relation.languageversion . sample values of dc.relation.languageversion 10665.1/9843 ; 10665.1/9844 ie not complete uri handle. thanks in advance! actual code <xsl:template name=...
Read more