Python: Problems sending 'list' of urls to scrapy spider to scrape -


trying send 'list' of urls scrapy crawl via spider via using long string, splitting string inside crawler. i've tried copying format given in this answer.

the list i'm trying send crawler future_urls

    >>> print future_urls     set(['https://ca.finance.yahoo.com/q/hp?s=alxn&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m', 'http://finance.yahoo.com/q/hp?s=tfw.l&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m', 'https://ca.finance.yahoo.com/q/hp?s=dltr&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m', 'https://ca.finance.yahoo.com/q/hp?s=agnc&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m', 'https://ca.finance.yahoo.com/q/hp?s=hmsy&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m', 'http://finance.yahoo.com/q/hp?s=bats.l&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m'])

then sending crawler through:

command4 = ("scrapy crawl future -o future_portfolios_{0} -t csv -a future_urls={1}").format(input_file, str(','.join(list(future_urls))))  >>> print command4 scrapy crawl future -o future_portfolios_input_10062008_10062012_ver_1.csv -t csv -a future_urls=https://ca.finance.yahoo.com/q/hp?s=alxn&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m,http://finance.yahoo.com/q/hp?s=tfw.l&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m,https://ca.finance.yahoo.com/q/hp?s=dltr&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m,https://ca.finance.yahoo.com/q/hp?s=agnc&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m,https://ca.finance.yahoo.com/q/hp?s=hmsy&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m,http://finance.yahoo.com/q/hp?s=bats.l&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m >>> type(command4) <type 'str'>

my crawler (partial):

class futurespider(scrapy.spider): name = "future" allowed_domains = ["finance.yahoo.com", "ca.finance.yahoo.com"] start_urls = ['https://ca.finance.yahoo.com/q/hp?s=%5eixic']  def __init__(self, *args, **kwargs):     super(futurespider, self).__init__(*args,**kwargs)     self.future_urls = kwargs.get('future_urls').split(',')     self.rate_returns_len_min = 12     self.required_amount_of_returns = 12     x in self.future_urls:             print "going scrape:"             print x  def parse(self, response):      if self.future_urls:         x in self.future_urls:             yield scrapy.request(x, self.stocks1)

however, printed out print 'going scrape:', x is:

going scrape: https://ca.finance.yahoo.com/q/hp?s=alxn

only 1 url, , it's portion of first url in future_urls problematic.

can't seem figure out why crawler won't scrape of urls in future_urls...

i think it's stopping when hits ampersand (&), can escape using urllib.quote.

for example:

import urllib  escapedurl = urllib.quote('https://ca.finance.yahoo.com/q/hp?s=alxn&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m')

then normal can do:

>>>>urllib.unquote(escapedurl) https://ca.finance.yahoo.com/q/hp?s=alxn&a=06&b=10&c=2012&d=06&e=10&f=2015&g=m

-

Comments

Post a Comment

Popular posts from this blog

c# - Binding a comma separated list to a List<int> in asp.net web api -

the default binder in web api expecst http://url.com/webapi/report/?pageids=3243&pageids=2365 to bind public ihttpactionresult report(list<int> pageids){ // exciting webapi code} i wish bind http://url.com/webapi/report/?pageids=3243,2365 as running out of space in url get . i have created class public class commaseparatedmodelbinder : system.web.http.modelbinding.imodelbinder { public bool bindmodel(httpactioncontext actioncontext, modelbindingcontext bindingcontext) { //binding in here } } and registered in webapiconfig.cs var provider = new simplemodelbinderprovider( typeof(list<int>), new commaseparatedmodelbinder()); config.services.insert(typeof(modelbinderprovider), 0, provider); i have altered method signature use model binder so public ihttpactionresult report( [modelbinder] list<int> pageids){ // exciting webapi code} however break point in binder not being ...
Read more

how to prompt save As Box in Excel Interlop c# MVC 4 -

i want export excel sheet using excel interloop... i have datatable convert excel sheet... now there way save file using save dialogue box in mvc 4 here code public string datatabletoexcel(datatable dt, string htmlheading, string key) { try { microsoft.office.interop.excel.application excel; microsoft.office.interop.excel.workbook excelworkbook; microsoft.office.interop.excel.worksheet excelsheet; microsoft.office.interop.excel.range excelcellrange; excel = new microsoft.office.interop.excel.application(); excel.visible = false; excel.displayalerts = false; excelworkbook = excel.workbooks.add(type.missing); excelsheet = (microsoft.office.interop.excel.worksheet)excelworkbook.activesheet; excelsheet.name = "sheet1"; dataset ...
Read more

xslt 1.0 - How to access or retrieve mets content of an item from another item? -

my use case have item has link in item. example, item 123456789/152 has metadata field dc.relation.hasversion=123456789/717 . within item view of 123456789/152 , how can retrieve values of metadata of 123456789/717 ? example, want retrieve dc.language.iso value of 123456789/717 123456789/152 . i looked @ related items feature in dspace don't know how metadata displayed in related items list pulled items in list. i using dspace version 5.3 mirage 2 theme . edit based on schweerelos answer, below actual code. using custom metadata field dc.relation.languageversion . basically, want link other version instead of displaying value, display dc.language.iso of other version in item-view.xsl . i've incorporated answer of schweerelos question in code just displaying value of dc.relation.languageversion . sample values of dc.relation.languageversion 10665.1/9843 ; 10665.1/9844 ie not complete uri handle. thanks in advance! actual code <xsl:template name=...
Read more