In most websites, large volumes of data are collected and stored in an unorganized manner. In order to generate useful information and gain insights from the collected data, it is important to use web scraping techniques. A comprehensive Web Scraping Tutorial can help us extract this unstructured data, convert it into structured data-sets, and stockpile them in a centralized databank.
“Google extracts PDF documents to index more than 70% of its search-result content.”
The value of data is growing at a fast pace and a majority of businesses and entrepreneurs are leveraging the power of Big Data applications. Procuring structured data from a huge bulk of unorganized data, however, is a tedious task.
It can be only be done by employing the techniques and principles of web scraping. Most of us are unfamiliar with the concept of web scraping; this web scraping tutorial for beginners can help you learn the basics and get started.
Download Detailed Brochure and Get Complimentary access to Live Online Demo Class with Industry Expert.
Web Scraping Tutorial: What You’ll Learn
The following web scraping tutorial comprises multiple sections that offer a working knowledge of using different methods of web scraping on data resources. The tutorial includes:
- What is Web Scraping?
- The Process of Web Scraping: How it Works
- Web Scraping Tutorial for Beginners
- Python Web Scraping Tutorial
- Web Scraping with R Tutorial
- Applications of Web Scraping
- Common Challenges in Web Scraping
With this tutorial, you’ll clearly understand the basics of data mining, web content extraction, and other web scraping processes. It will also equip you with the practical skills of using these techniques with programming languages such as Python and R. At the end of the tutorial, you’ll be a master web scraper and will apply the techniques to organize and analyze the data collected from your own websites.
This web scraping tutorial also provides answers to common web scraping questions such as:
- Is web scraping legal?
- How to make the right HTTP request for web scraping?
- Is it possible to extract data from the entire web?
- Can Facebook or Twitter be crawled with web scraping techniques?
- Is web scraping used for lead generation?
- Which is the best programming language for web scraping?
- Which is the best tool for web scraping?
After knowing the answers to these questions, you’ll find yourself more proficient in hands-on use of web scraping. So let’s begin.
What is Web Scraping?
Web scraping is the process of extracting huge volumes of data from websites, converting them into a structured and easy-to-use format, and storing them in a centralized data repository. A large amount of raw data collected from websites is available in an unstructured format. This data is hard to extract and handle. It needs to be converted into various formats such as PDF, XML, HTML, or CSV for further use.
The process of web scraping basically helps you extract this data, transform it, and make it available in an organized manner for reuse. With the use of proper tools, valuable information is pulled out from innumerable web pages. This information is saved in a local file and the data contained in it repurposed for APIs, web interfaces, record-keeping, and open-source projects.
Is web scraping legal? Web scraping is legal only if you use it ethically. If you use the web scraping tools to extract public data, it will be considered as a good web development practice. But if you use it to procure non-public data (information not accessible for everyone), it will be considered as malpractice.
The Process of Web Scraping: How it Works
Web scraping is a technical process, and web-scraping code can be written in different programming languages. Before developing the process, it is important to know the stages that are involved in it. This web scraping tutorial breaks down the process into:
- Extraction
- Conversion
- Re-usage
In the extraction phase, specific data from web pages can be downloaded under certain parameters or criteria with the use of data extraction tools. These tools are nothing but bots called “crawlers” and they find web pages that match the criteria. By using an HTTP protocol, the bot sends a GET request to the targeted site.
If the GET request is legitimate, the webserver processes it and enables the scraper bot or tool to read and extract the web page data in HTML code. The matching web pages are downloaded and their data is fetched for further processing. In this conversion phase, the collected unstructured data is analyzed, copied, and reformatted according to the specific purpose of scraping.
A web scraper tool is used to locate the targeted elements from the data associated with the text, videos, images, product information, contact information, and user responses. This data is extracted and organized in the set variables for further use. At the end of this re-usage phase, contextualized and structured data is made available for the impending application and utilization.
Is it possible to extract data from the entire web? No company, not even Google, has been able to crawl, extract, and scrape all the web pages available in the world. Scraping the entire web is not possible as the websites on the surface of the Internet are only a small proportion of the total number of webpages. On any website, the majority of webpages are running on the back-end.
Web Scraping Tutorial for Beginners
For beginners, this web scraping tutorial discusses two main fundamentals:
(i) Generating accurate HTTP GET requests
(ii) Parsing the content from HTML responses
An HTTP request is made when we bookmark a page, click a link, or even press the Enter key after typing a URL in the address bar. Each HTTP request contains information on parameters such as query, path, domain, sub-domain, and protocol.
How to make the right HTTP request for web scraping? In web scraping, the two key components of an HTTP request are the headers and the cookies. To get the exact data that we want, the header and the cookie information that goes with the HTTP GET request should be precise. A web server can decline the request if it contains unnecessary and incorrect information on the header and/or the cookies.
Once the request is made, you get a response in a simple HTML document. Since HTML is a simple markup language, the browser can extract and store these responses easily. The HTML responses can be further used to search for specific bits of text and locate raw HTML tags.
Python Web Scraping Tutorial
This web scraping tutorial for beginners also discusses how to implement the process in the Python programming language. Python is a popular programming language used in web development. With its simple code and easy-to-understand syntax, Python is easy to learn.
Python web scraping allows us to use a large collection of libraries such as Pandas and Numpy, where there are ample services and methods suitable for manipulating the extraction of web data.
In this Python web scraping tutorial, let’s learn how to extract product information from an e-commerce website, say Amazon.com.
Step 1: Find the URL of the targeted website
For this instance, we’ll target the URL for the “Computers & Tablets” product page on Amazon.com.
Log in on Amazon and type “Computers & Tablets”.
Step 2: Inspect the Page
Once the webpage gets loaded on the browser, you need to find and extract the data specific to product information. This data is situated in different tags of the web page. To find the specific data, we need to inspect the page as follows:
(i) Hover the cursor over a specific product
(ii) Right-click on it
(iii) Click on “Inspect” from the drop-down menu
The Browser Inspector Box opens up next to the web page in the same browser window.
Step 3: Find the Data for Extraction
In the Browser Inspector Box, go to the Elements section and locate the highlighted tag. In this case, the name, price, and rating of the product will be present inside the respective “div” tags.
Step 4: Create a Python File & Write the Code
Download and install the Python Terminal on your operating system (Windows, Ubuntu, or iOS). Open a terminal and execute the command lines as follows:
To create a test Python file called “web-test”, add the .py extension.
gedit web-test.py
Import the necessary libraries for web scraping. Here, Beautiful Soup, Selenium, and Panda will be imported as follows:
from selenium import webdriver from BeautifulSoup import BeautifulSoup import pandas as pd
Set the path of the web driver to the driver of the Chrome browser:
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")
To open the targeted URL, run the following code:
products=[] #List to store name of the product
prices=[] #List to store price of the product
ratings=[] #List to store rating of the product
driver.get("<a href="https://www.amazon.com/s?i=specialty-aps&bbn=16225007011&rh=n%3A16225007011%2Cn%3A13896617011&ref=nav_em_T1_0_4_NaN_3__nav_desktop_sa_intl_computers_tablets")
To extract data from the respective div tags, run the following code:
content = driver.page_source
soup = BeautifulSoup(content)
for a in soup.findAll('a',href=True, attrs={'class':'_31qSD5'}):
name=a.find('div', attrs={'class':'_3wU53n'})
price=a.find('div', attrs={'class':'_1vC4OE _2rQ-NK'})
rating=a.find('div', attrs={'class':'hGSR34 _2beYZw'})
products.append(name.text)
prices.append(price.text)
ratings.append(rating.text)
Step 5: Extract & Store the Data
After finding the specific data, extract it with the following code:
Python web-test.py
The data can be stored in a specific format, let’s say in a CSV file, as follows:
df = pd.DataFrame({'Product Name':products,'Price':prices,'Rating':ratings})
df.to_csv('products.csv', index=False, encoding='utf-8')
This creates the “products.csv” file which contains all the extracted data. You can browse the product names, prices, ratings, and the respective product information structurally from the CSV file by opening it in a spreadsheet application such as Microsoft Excel and VisiCalc.
If you want to delve more deeply into the topic of web scraping with Python, you can watch this video.
Web Scraping with R Tutorial
Web scraping can also be implemented with R (Ruby) programming language. It is relatively easy as you need to run only five command lines and use only a couple of libraries for web scraping.
Let’s say you want to extract data from “List of members of the United Nations Security Council” page of Wikipedia.org. Open the R command-line interface, install the “rvest” and “tidyverse” packages and run the following five commands:
library(rvest)
library(tidyverse)
h <- read_html("https://en.wikipedia.org/wiki/List_of_members_of_the_United_Nations_Security_Council")
reps <- h %>%
html_node("#mw-content-text > div > table:nth-child(18)") %>%
html_table()
reps <- reps[,c(1:2,4:9)] %>% as_tibble()
Open the “reps” object file on the IDE (RStudio viewer) and you’ll be able to see all the data organized in a structured format of rows and columns.
Which is the best programming language for web scraping? Python is the best programming language used for web scraping. It is crawling-friendly and has extensive libraries for data mining and data extraction.
However, if you find Python difficult, you can use our web scraping with R tutorial to meet your business requirements.
Key Applications of Web Scraping
There are various business applications and practical uses of web scraping techniques. In this web scraping tutorial, let’s look at some examples where web scraping can be applied effectively:
Legal Technology
Lawyers use web scraping tools to find case references from past judgments and popular case studies.
Business Profiling
Several companies create directories by scraping information of business profiles such as email IDs, contact numbers, addresses, products & services, and geographical locations.
Media
For research purposes, web scraping is prominently used in collecting information from user profiles on social media websites and other platforms.
Digital marketing is the most effective application of web scraping. Retails and e-commerce players use web scraping tools to procure information about competitors and improve their own product lines. Is web scraping used for lead generation? Yes, web scraping is used for sentimental analysis, which in turn gives insights to drive product leads and indirectly boost revenues.
Can Facebook or Twitter be crawled with web scraping techniques? These websites have blocked automated web scraping tools and web crawler bots. It is difficult to scrape these websites, and it is even considered illegal.
Common Challenges in Web Scraping
With the growing use of web scraping, people are also facing several problems in the process. This web scraping tutorial highlights the top four challenges that people commonly face while scraping websites.
Honeypots
In several cases, website developers install traps called honeypots, where scraping tools are not able to crawl and detect information. These honeypots are either color-disguised on the webpage or nested under “display:none” CSS (Cascading Style Sheet) tags.
Data Warehousing
Improper data storage infrastructure can downgrade the outcome of web scraping. For large-scale web scraping projects, it is important to facilitate scalable data warehousing.
Anti-Scraping & Crawler-Hostile Technologies
Dynamic coding algorithms disable scraping activities. Websites such as LinkedIn and Facebook use these algorithms to block scraping tools. These websites are also built on JavaScript technologies that create a hostile environment for web crawlers and scraping bots.
Changes in Website Structures
A simple change in website structure can create multiple changes in the outcome of web scraping processes. Hence, the scrapers are required to find the correct field and develop a relevant logic depending on the current website structure.
Which is the best tool for web scraping? There are several tools used for web scraping. Some of the best web scraping tools available to us are Dexi.io, Web Harvey, Apify SDK, Mozenda, Octoparse, PySpider, Content Grabber, and Cheerio.
Final Thoughts
In this web scraping tutorial, we understood the meaning of the web scraping process. We learned how the process works and why is it important to extract information from web pages. We also looked at the web scraping methods implemented with Python and Ruby programming languages. After going through this web scraping tutorial, consider collecting specific data from any targeted website of your choice and try to structurally organize it in the output file.
We hope our web scraping tutorial for beginners serves as a comprehensive guide for you. With our Python web scraping tutorial and web scraping with R tutorial, you can easily track your company’s performance and gain crucial insights about it.
Join the Python Programming Course and start your career as a Data Scientist.
If you’d like to get more information on some concepts of web scraping, leave your queries in the comments below.
