Assuring Scraping Success with Proxy Data Scraping

Assuring Scraping Success with Proxy Data Scraping


Have you ever heard of "Data Scraping?" Data Scraping is the process of collecting useful data that has been placed in the public domain of the internet (private areas too if conditions are met) and storing it in databases or spreadsheets for later use in various applications. Data Scraping technology is not new and many a successful businessman has made his fortune by taking advantage of data scraping technology.

Sometimes website owners may not derive much pleasure from automated harvesting of their data. Webmasters have learned to disallow web scrapers access to their websites by using tools or methods that block certain ip addresses from retrieving website content. Data scrapers are left with the choice to either target a different website, or to move the harvesting script from computer to computer using a different IP address each time and extract as much data as possible until all of the scraper's computers are eventually blocked.

Thankfully there is a modern solution to this problem. Proxy Data Scraping technology solves the problem by using proxy IP addresses. Every time your data scraping program executes an extraction from a website, the website thinks it is coming from a different IP address. To the website owner, proxy data scraping simply looks like a short period of increased traffic from all around the world. They have very limited and tedious ways of blocking such a script but more importantly -- most of the time, they simply won't know they are being scraped.

You may now be asking yourself, "Where can I get Proxy Data Scraping Technology for my project?" The "do-it-yourself" solution is, rather unfortunately, not simple at all. Setting up a proxy data scraping network takes a lot of time and requires that you either own a bunch of IP addresses and suitable servers to be used as proxies, not to mention the IT guru you need to get everything configured properly. You could consider renting proxy servers from select hosting providers, but that option tends to be quite pricey but arguably better than the alternative: dangerous and unreliable (but free) public proxy servers.

There are literally thousands of free proxy servers located around the globe that are simple enough to use. The trick however is finding them. Many sites list hundreds of servers, but locating one that is working, open, and supports the type of protocols you need can be a lesson in persistence, trial, and error. However if you do succeed in discovering a pool of working public proxies, there are still inherent dangers of using them. First off, you don't know who the server belongs to or what activities are going on elsewhere on the server. Sending sensitive requests or data through a public proxy is a bad idea. It is fairly easy for a proxy server to capture any information you send through it or that it sends back to you. If you choose the public proxy method, make sure you never send any transaction through that might compromise you or anyone else in case disreputable people are made aware of the data.

A less risky scenario for proxy data scraping is to rent a rotating proxy connection that cycles through a large number of private IP addresses. There are several of these companies available that claim to delete all web traffic logs which allows you to anonymously harvest the web with minimal threat of reprisal. Companies such as offer large scale anonymous proxy solutions, but often carry a fairly hefty setup fee to get you going.

Source:http://ezinearticles.com/?Assuring-Scraping-Success-with-Proxy-Data-Scraping&id=248993

How to scrape search results from search engines like Google, Bing and Yahoo

How to scrape search results from search engines like Google, Bing and Yahoo

Search giants like Google, Yahoo and Bing made their empire on scraping others content. However, they don’t want you to scrape them. How ironic, isn’t it?

Search engine performance is a very important metric all digital marketers want to measure and improve. I’m sure you will be using some great SEO tools to check how your keywords perform. All great SEO tool comes with a search keyword ranking feature. The tools will tell you how your keywords are performing in google, yahoo bing etc.

 How will you get data from search engines If you want to build a keyword ranking app?

 These search engines have API’s but the daily query limit is very low and not useful for the commercial purpose. The only solution is to scrape search results. Search engine giants obviously know this :). Once they know that you are scraping, they will  block your IP, Period!

 How do Search engines detect bots?

 Here are the common methods of detection of bots.

* IP address: Search engines can detect if there are too many requests coming from a single IP. If a high amount of traffic is detected, they will throw a captcha.

 * Search patterns: Search engines match traffic patterns to an existing set of patterns and if there is huge variation, they will classify this as a bot.

 If you don’t have access to sophisticated technology, it is impossible to scrape search engines like google, Bing or Yahoo.

 How to avoid detection

There are some things you can do to  avoid detection.

    Scrape slowly and don’t try to squeeze everything at once.
    Switch user agents between queries
    Scrape randomly and don’t follow the same pattern
    Use intelligent IP rotations
    Clear Cookies after each IP change or disable them completely

Thanks for reading this blog post.

Source: http://blog.datahut.co/how-to-scrape-search-results-from-search-engines-like-google-bing-and-yahoo/

Data Mining Process - Why Outsource Data Mining Service?

Data Mining Process - Why Outsource Data Mining Service?

Overview of Data Mining and Process:

Data mining is one of the unique techniques for investigating information to extract certain data patterns and decide to outcome of existing requirements. Data mining is widely use in client research, services analysis, market research and so on. It is totally based on mathematical algorithm and analytical skills to drive the desired results from the huge database collection.

Information mining is mostly used by financial analyzer, business and professional organization and also there are many growing area of business that are get maximum advantages of data extract with use of data warehouses in their small to large level of businesses.

Most of functionalities which are used in information collecting process define as under:

* Retrieving Data

* Analyzing Data

* Extracting Data

* Transforming Data

* Loading Data

* Managing Databases

Most of small, medium and large levels of businesses are collect huge amount of data or information for analysis and research to develop business. Such kind of large amount will help and makes it much important whenever information or data required.

Why Outsource Data Online Mining Service?

Outsourcing advantages of data mining services:
o Almost save 60% operating cost
o High quality analysis processes ensuring accuracy levels of almost 99.98%
o Guaranteed risk free outsourcing experience ensured by inflexible information security policies and practices
o Get your project done within a quick turnaround time
o You can measure highly skilled and expertise by taking benefits of Free Trial Program.
o Get the gathered information presented in a simple and easy to access format

Thus, data or information mining is very important part of the web research services and it is most useful process. By outsource data extraction and mining service; you can concentrate on your co relative business and growing fast as you desire.

Outsourcing web research is trusted and well known Internet Market research organization having years of experience in BPO (business process outsourcing) field.

If you want to more information about data mining services and related web research services, then contact us.

Source: http://ezinearticles.com/?Data-Mining-Process---Why-Outsource-Data-Mining-Service?&id=3789102

Scraping Yelp Data and How to use?

Scraping Yelp Data and How to use?

We get a lot of requests to scrape data from Yelp. These requests come in on a daily basis, sometimes several times a day. At the same time we have not seen a good business case for a commercial project with scraping Yelp.

We have decided to release a simple example Yelp robot which anyone can run on Chrome inside your computer, tune to your own requirements and collect some data. With this robot you can save business contact information like address, postal code, telephone numbers, website addresses etc.  Robot is placed in our Demo space on Web Robots portal for anyone to use, just sign up, find the robot and use it.

How to use it:

    Sign in to our portal here.
    Download our scraping extension from here.
    Find robot named Yelp_us_demo in the dropdown.
    Modify start URL to the first page of your search results. For example: http://www.yelp.com/search?find_desc=Restaurants&find_loc=Arlington,+VA,+USA
    Click Run.
    Let robot finish it’s job and download data from portal.

Some things to consider:

This robot is placed in our Demo space – therefore it is accessible to anyone. Anyone will be able to modify and run it, anyone will be able to download collected data. Robot’s code may be edited by someone else, but you can always restore it from sample code below. Yelp limits number of search results, so do not expect to scrape more results than you would normally see by search.

In case you want to create your own version of such robot, here it’s full code:

// starting URL above must be the first page of search results.
// Example: http://www.yelp.com/search?find_desc=Restaurants&find_loc=Arlington,+VA,+USA

steps.start = function () {

   var rows = [];

   $(".biz-listing-large").each (function (i,v) {
     if ($("h3 a", v).length > 0)
       {
        var row = {};
        row.company = $(".biz-name", v).text().trim();
        row.reviews =$(".review-count", v).text().trim();
        row.companyLink = $(".biz-name", v)[0].href;
        row.location = $(".secondary-attributes address", v).text().trim();
        row.phone = $(".biz-phone", v).text().trim();
        rows.push (row);
      }
   });

   emit ("yelp", rows);
   if ($(".next").length === 1) {
     next ($(".next")[0].href, "start");
   }
 done();
};

Source: https://webrobots.io/scraping-yelp-data/

Easy Web Scraping using PHP Simple HTML DOM Parser Library

Easy Web Scraping using PHP Simple HTML DOM Parser Library

Web scraping is only way to get data from website when  website don’t provide API to access it’s data. Web scraping involves following steps to get data:

    Make request to web page
    Parse/Extract data that you want to scrape from website.
    Store data for final output (excel, csv,mysql database etc).

Web scraping can be implemented in any language like PHP, Java, .Net, Python and any language that allows to make web request to get web page content (HTML text) in to variable. In this article I will show you how to use Simple HTML DOM PHP library to do web scraping using PHP.
PHP Simple HTML DOM Parser

Simple HTML DOM is a PHP library to parse data from webpages, in short you can use this library to do web scraping using PHP and even store data to MySQL database.  Simple HTML DOM has following features:

    The parser library is written in PHP 5+
    It requires PHP 5+ to run
    Parser supports invalid HTML parsing.
    It allows to select html tags like Jquery way.
    Supports Xpath and CSS path based web extraction
    Provides both the way – Object oriented way and procedure way to write code

Scrape All Links

<?php
include "simple_html_dom.php";

//create object
$html=new simple_html_dom();

//load specific URL
$html->load_file("http://www.google.com");

// This will Find all links
foreach($html->find('a') as $element)
   echo $element->href . '<br>';

?>

Scrape images

<?php
include "simple_html_dom.php";

//create object
$html=new simple_html_dom();

//load specific url
$html->load_file("http://www.google.com");

// This will Find all links
foreach($html->find('img') as $element)
   echo $element->src . '<br>';

?>

This is just little idea how you can do web scraping using PHP.Keep in mind that Xpath can make your job simple and fast. You can find all methods available in SimpleHTMLDom documentation page.

Source: http://webdata-scraping.com/web-scraping-using-php-simple-html-dom-parser-library/

Things to take care while doing Web Scraping!!!

Things to take care while doing Web Scraping!!!

In the present day and age, web scraping word becomes most popular in data science. Basically web scraping is extracting the information from the websites using pre-written programs and web scraping scripts. Many organizations have successfully used web site scraping to build relevant and useful database that they use on a daily basis to enhance their business interests. This is the age of the Big Data and web scraping is one of the trending techniques in the data science.

Throughout my journey of learning web scraping and implementing many successful scraping projects, I have come across some great experiences we can learn from.  In this post, I’m going to discuss some of the approaches to take and approaches to avoid while executing web scraping.

User Proxies: Anonymously scraping data from websites

One should not scrape website with a single IP Address. Because when you repeatedly request the web page for web scraping, there is a chance that the remote web server might block your IP address preventing further request to the web page. To overcome this situation, one should scrape websites with the help of proxy servers (anonymous scraping). This will minimize the risk of getting trapped and blacklisted by a website. Use of Proxies to hide your identity (network details) to remote web servers while scraping data. You may also use a VPN instead of proxies to anonymously scrape websites.

Take maximum data and store it.

Do not follow “process the web page as it comes from the remote server”. Instead take all the information and store it to disk. This approach will be useful when your scraping algorithm breaks in the middle. In this case you don’t have to start scraping again. Never download the same content more than once as you are just wasting bandwidth. Try and download all content to disk in one go and then do the processing.

Follow strict rules in parsing:

Check various rules while parsing the information from the web site. For example if you expect a value to be a date then check that it’s really a date. This may greatly improve the quality of information. When you get unexpected data, then the algorithm need to be changed accordingly.

Respect Robots.txt

Robots.txt specifies the set of rules that should be followed by web crawlers and robots. I strongly advise you to consider and adjust your crawler to fully respect robots.txt. Robots.txt contains instructions on the exact pages that you are allowed to crawl, user-agent, and the requisite intervals between page requests. Following to these instructions minimizes the chance of getting blacklisted and banned from website owner.

Use XPath Smartly

XPath is a nice option to select elements of the HTML document more flexibly than CSS Selectors.  Be careful about HTML structure change through page to page so one xpath you made may be failed to extract data on another page due to changes in HTML structure.

Obey Website TOC:

Some websites make it absolutely apparent in their terms and conditions that they are particularly against to web scraping activities on their content. This can make you vulnerable against possible ethical and legal implications.

Test sample scrape and verify the data with actual scrape

Once you are done with web scraping project set up, you need to test it for sometimes. Check the extracted data. If something is not good, find out the cause and make changes accordingly and finally come to a perfect web scraping project.

Source: http://webdata-scraping.com/things-take-care-web-scraping/

Benefits of Ruby over Python & R for Web Scraping

Benefits of Ruby over Python & R for Web Scraping

In this data driven world, you need to be constantly vigilant, as information and key data for an organization keeps changing all the while. If you get the right data at the right time in an efficient manner, you can stay ahead of competition. Hence, web scraping is an essential way of getting the right data. This data is crucial for many organizations, and scraping technique will help them keep an eye on the data and get the information that will benefit them further.

Web scraping involves both crawling the web for data and extracting the data from the page. There are several languages which programmers prefer for web scraping, the top ones are Ruby, Python & R. Each language has its own pros and cons over the other, but if you want the best results and a smooth flow, Ruby is what you should be looking for.

Ruby is very good at production deployments and using Ruby, Redis & Chef have proven to be a great combination. String manipulation in Ruby is very easy because it is based on Perl syntax. Also, Ruby is great for analyzing web pages using  one of the very powerful gems called Nokogiri. Nokogiri is much easier to use as compared to other packages and libraries used by R and Python respectively. Nokogiri can deal with broken HTML / HTML fragments easily. Ruby also has many extensions, such as Sanitize and Loofah, that can help clean up broken HTML.

Python programmers widely use a library called Beautiful Soup for pulling data out of HTML & XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work. R programmers have a new package called rvest that makes it easy to scrape data from html web pages, by libraries like beautiful soup. It is designed to work with magrittr so that you can express complex operations as elegant pipelines composed of simple, easily understood pieces.

To help you understand it more effectively, below is a comprehensive infographic for the same.

Ruby is far ahead of Python & R for cloud development and deployments.  The Ruby Bundler system is just great for managing and deploying packages from Github. Using Chef, you can start up and tear down nodes on EC2, at will, and monitor for failures,  scale up or down, reset your IP addresses, etc. Ruby also has great testing frameworks like Fakeweb and Capybara, making it almost trivial to build a great suite of unit tests and to include advanced features, like crawling  and scraping using webkit / selenium. 

The only disadvantage to Ruby is lack of machine learning and NLP toolkits, making it much harder to emulate the capacity of a tool like Pattern.  It can still be done, however, since most of the heavy lifting can be done asynchronously using Unix tools like liblinear or vowpal wabbit.

Conclusion

Each language has its plus point and you can pick the one which you are most comfortable with. But if you are looking for smooth web scraping experience, then Ruby is the best option. That has been our choice too for years at PromptCloud for the best web scraping results. If you have any further questions about this, then feel free to get in touch with us.

Source: https://www.promptcloud.com/blog/benefits-of-ruby-for-web-scraping