Part I, Part II, Part III, Part IV, Part V

In this last part of this tutorial series, I am going to briefly cover a very important topic: dynamic scraping. The example code can be found in the following repo:

In real web scraping projects, you often cannot directly crawl the websites using the approach presented in the first four parts of this tutorial for various reasons, such as the web page is dynamically generated (such as the example in this tutorial, where the web page is loaded when a user scrolls to the bottom of the current page), you…

I often use and study open-source code from the Internet and run into the following situation:

I found a nice code repo on Github, clone it to my local computer, and then have to spend a lot of time just trying to get the code up and running because I have to figure out things like Python version, the required packages with specific versions, Jupyter configurations, etc.

In this short tutorial, I show how I organize a self-contained Python project, which can be up and running with minimal effort. …

Github Pages allow you to host static markdown/web pages (HTML and JS) for free. You only need a few steps to have a website up and running.

I will show you how to build such a website using a course website as an example: https://harrywang.github.io/misy331/.

Step 1. Create an empty repo with basic information (you can leave .gitignore and license empty as the default setting):

Part I, Part II, Part III, Part IV, Part V

In the previous three parts, you have developed a spider that extracts quote information from http://quotes.toscrape.com and stores the data into a local SQLite database. In this part, I will show you how to deploy the spider to the cloud.

First, let’s see how you can deploy to https://scrapinghub.com — the commercial service ran by the team behind the open-source Scrapy framework.

Create a free account and a new project:

Part I, Part II, Part III, Part IV, Part V

In Part II, you have extracted all the required data from the website and stored them in Items. In Part III, I will introduce Item Pipelines to save the extracted data into a database using ORM (SQLAlchemy) and handle the duplicate data issue.

Each item returned by the spider is sent to Item Pipelines (if any) sequentially for additional processing, such as saving items to the database, data validation, removing duplicates, etc. Item pipelines are defined as classes in the pipelines.py …

Part I, Part II, Part III, Part IV, Part V

In Part I, you learned how to setup Scrapy project and write a basic spider to extract web pages by following page navigation links. However, the extracted data are merely displayed to the console. In Part II, I will introduce the concepts of Item and ItemLoader and explain why you should use them to store the extracted data.

Let’s first look at Scrapy Architecture:

Part I, Part II, Part III, Part IV, Part V

Web scraping is an important skill for data scientists. I have developed a number of ad hoc web scraping projects using Python, BeautifulSoup, and Scrapy in the past few years and read a few books and tons of online tutorials along the way. …

Start a Deep Learning AMI — make sure to choose a GPU one!