php - Crawling a second page answer; curl -
i'm crawling few websites, it's working fine,
but .... have 1 specific website i'm trying crawl, , it's making few "redirects" before landing web want.
so it's ...
http://www.example.com/?day=01/01/2016&action=search_prices
this go http://www.example.com/search/default.aspx take few seconds search answer page , show on there.
is there way this? hint, clue, etc awesome
simple code right (almost sites crawling jsons):
function get_web_page( $url ){ $options = array( curlopt_returntransfer => true, // return web page curlopt_header => false, // don't return headers curlopt_followlocation => true, // follow redirects curlopt_encoding => "", // handle encodings curlopt_useragent => "spider", // curlopt_autoreferer => true, // set referer on redirect curlopt_connecttimeout => 120, // timeout on connect curlopt_httpheader => array('headername: headervalue'), curlopt_timeout => 120, // timeout on response curlopt_maxredirs => 10, // stop after 10 redirects curlopt_ssl_verifypeer => false // disabled ssl cert checks ); $ch = curl_init( $url ); curl_setopt_array( $ch, $options ); $content = curl_exec( $ch ); $err = curl_errno( $ch ); $errmsg = curl_error( $ch ); $header = curl_getinfo( $ch ); curl_close( $ch ); $header['errno'] = $err; $header['errmsg'] = $errmsg; $header['content'] = $content; return $header; }
Comments
Post a Comment