labs/web-scraping/web-scraping-in-python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GISC 420 T1 2022\n",
    "# Web scraping in `python`\n",
    "There's a lot of data out there on the web these days, much of it spatial in nature. How can we get those data into a form we can use in GIS or other geospatial tools? That's what this week's lab materials focus on.\n",
    "\n",
    "To get there, we need to understand a little bit about how websites and web pages work.\n",
    "\n",
    "## What's in a web page\n",
    "OK... let's take a look at a [simple web page](https://southosullivan.com/gisc420/other/simple-web-page.html). This is an example of very simple web map.\n",
    "\n",
    "In the browser you are viewing it with, do something like right-click **View page source**.\n",
    "\n",
    "You should see the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "<!DOCTYPE html>\n",
    "\n",
    "<html>\n",
    "    <head>\n",
    "        <title>A very simple web page</title>\n",
    "    </head>\n",
    "\n",
    "    <body>\n",
    "        <h1>My first web page\n",
    "        </h1>\n",
    "        <p>This page demonstrates some <i>very</i> basic HTML for use in my classes.\n",
    "        </p>\n",
    "        <p>This page was written by <a href=\n",
    "        \"https://southosullivan.com/geodos\">Prof David O'Sullivan</a>.\n",
    "        </p>\n",
    "        <p>David is from Ireland.  Sometimes, he misses Ireland's glorious summers, like the one in\n",
    "        this picture.\n",
    "        </p>\n",
    "        <img src=\"muckish.jpg\" width=640>\n",
    "        <figcaption>Yes, that picture really was taken in August</figcaption>\n",
    "        <p>That's Muckish Mountain in Donegal.  Here is a map.</p>\n",
    "        <iframe src=\"https://www.google.com/maps/embed?pb=!1m18!1m12!1m3!1d51581.682424507155!2d-8.022036861268594!3d55.097443300030626!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!3m3!1m2!1s0x0000000000000000%3A0xd25d3a3fef4d920c!2sMt.+Muckish!5e0!3m2!1sen!2sus!4v1441395150485\" width=\"400\" height=\"300\" frameborder=\"0\" style=\"border:0\" allowfullscreen></iframe>\n",
    "        <p>The picture was taken from <a href=\n",
    "\"https://www.google.com/maps/@55.214797,-7.978438,3a,75y,205.64h,91.69t/data=!3m4!1e1!3m2!1ssdYiz3D8tz6atCSdR9200w!2e0?hl=en\">roughly here</a>.  The google car got better weather than I did.\n",
    "        </p>\n",
    "    </body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although, what you see will be syntax-coloured to help understand it a little better.\n",
    "\n",
    "The key thing from our present perspective is to note that there is *a tree like structure* to this page.\n",
    "\n",
    "At the 'trunk' of the tree is the whole page denoted by `<html></html>` **tags**.\n",
    "\n",
    "Nested below this are the `<head></head>` and `<body></body>` **elements**.\n",
    "\n",
    "Inside each of these are other elements, headings, paragraphs, an image, a figure caption, and something called an iframe, inside of which is the map.\n",
    "\n",
    "This is an exceedingly simple web page. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Something a bit more complicated\n",
    "So now look at [this page](https://southosullivan.com/gisc422/interpolation/#/).\n",
    "\n",
    "Let's again, take a closer look, and also examine it with your browser's **web console** tools.\n",
    "\n",
    "**I'll talk you through this**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summing things up\n",
    "Web pages are assembled from several different pieces\n",
    "\n",
    "1. **Hyper Text Markup Language** (HTML) tags describe the structure of the page using hierarchically  nested tags that defined the *elements* which make up the page.\n",
    "2. **Cascading Style Sheets** (CSS) determine how various elements in a page should be displayed.\n",
    "3. Client-side code in **JavaScript** which is often used to build interactive elements in the page, which are responsive to the page's readers.\n",
    "\n",
    "There is a fourth 'hidden' element to most pages, which is the server-side database(s) used to populate typical pages with requested information. On human-facing pages, this part of the process is generally handled by a combination of client-side interface elements that obtain information from users (search terms and so on), and form these into query strings that are decoded into database queries on the server-side. Responses from the database populate the page that is eventually seen by a user.\n",
    "\n",
    "[DuckDuckGo](https://duckduckgo.com) and [Google maps](https://maps.google.com) and [Let Me Google That For You](https://lmgtfy.com) provide simple examples of query strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So much for simple\n",
    "The simple structure of the web pages we have looked at is about the limit of what can be built by hand by a semi-skilled human (like me...). Almost all web pages today are assembled in automated fashion. Take a look for example at [this page](https://www.pbtech.co.nz/product/BATPAN5254/Panasonic-K-KJ55MCC4TA-3hr-Quick-Charger--4-x-AA-e). Click on another page at the same site, and compare the code you can see in the web console inspector. Or here is [another example](https://en.wikipedia.org/wiki/HTML).\n",
    "\n",
    "The point is that every page has a similar structure, and it is this structure that we can use to automate the process of obtaining useable data from webpages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But... before we go there\n",
    "OK... it's important to realise that there are situations where automation may not be the point. There are two extremes to this.\n",
    "\n",
    "**First**, you might just be interested in a particular dataset in a once-off fashion. Here's an example [the New Zealand AED map](https://aedlocations.co.nz/) of how that can work.\n",
    "\n",
    "**Alternatively**, the website in question may have an API, which allows you to make queries in an organised way. Many companies publish APIs because they _want_ other websites to be able to link to theirs in an interactive way. Here's [an example](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets.html). There are downsides to using an API. For a start, they might not be free. Or at any rate, the free version may be limited in some way. In many cases you are rate limited to the number of searches allowed in a given time period, and also required to authenticate (i.e. you need an account with the service). Furthermore, even premium versions may not offer all the functionality you would like. On the other hand, they are generally well documented, and greatly simplify the business of regularly pulling data from websites of interest. \n",
    "\n",
    "A middle ground is that you might find that others have written tools to interface with popular APIs. For example,  see this list of [python flickr related projects on github](https://github.com/search?q=python+flickr)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting serious about web scraping\n",
    "Assuming none of the above options are available or useful for whatever reason, you may want to develop code to *scrape* data from websites. There may be additional advantages to going down this path\n",
    "\n",
    "* no registration necessary\n",
    "* no rate limiting (although this comes with **serious caveats**)\n",
    "\n",
    "Ultimately, anything viewable on a website can be scraped. Having said that:\n",
    "\n",
    "* Check terms and conditions before scraping\n",
    "* Be nice: it's unfortunately easy to inadvertently launch a denial of service attack on a website if you hit it too often with too many requests\n",
    "* Website change all the time, so scraping code that works one day might break at any moment; be prepared to have to continually update your code\n",
    "\n",
    "Getting into web scraping requires us to delve a bit more deeply into that tree structure of web pages that I have already mentioned. And that means..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ... getting to know the DOM\n",
    "Like everything else in computing, that tree-like structure has an acronym. It is the key idea behind the **Document Object Model** by which web pages are organised.\n",
    "\n",
    "The DOM is a language independent API that allows manipulation of the contents of a document such that its contents and appearance can be changed programmatically. It allows automatic navigation and search of the structure of a document.\n",
    "\n",
    "So, how does it work?\n",
    "\n",
    "### A tree\n",
    "Every document is organised as a nested set of *elements* in tree-like fashion.\n",
    "\n",
    "<img src='dom.png'>\n",
    "\n",
    "**Source**: [DOM and JQuery](https://cs.wellesley.edu/~cs110/reading/DOM-JQ.html)\n",
    "\n",
    "This nested structure, and the fact that all the pages on a given site of a given kind will have the same structure, makes it possible to write code to *traverse* the tree and extract the elements of interest, including any data they may contain.\n",
    "\n",
    "Modern browsers *parse* HTML into the DOM format, and then render the webpage. In web scraping we use the same information the browsers uses to render the page to figure out which pieces of the page we want to scrape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A tree with labels\n",
    "The HTML elements in the DOM tree usually have three components. \n",
    "\n",
    "First there are **tags** which designate the beginning and end of the element. These are the opening `<tagname>` and closing `</tagname>` chunks we have seen all over the web page source material we have been looking at.\n",
    "\n",
    "Second tags have associated with them **attributes** and **content**. The content is all the material (which may include additional nested elements) between the tag start and end markers. The attributes are additional information describing the tag (in effect data about the tag itself.\n",
    "\n",
    "Some common HTML tags.\n",
    "\n",
    " Tag   | Usage\n",
    " --- | -----\n",
    "`<html>` | Designates a HTML document\n",
    "`<head><body>` | The header and main body of a document\n",
    "`<div>` | A general container for content used to structure a document into sections\n",
    "`<span>` | A container for inline material, often used to mark particular items\n",
    "`<h1><h2><h3><h4><h5><h6>` | Headings (6 levels)\n",
    "`<ol><ul><li>` | Lists of various kinds\n",
    "`<a>` | Anchors - most often used for hyperlinks\n",
    "`<p>` | Paragraphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More labels\n",
    "In addition to the tags themselves, and any inbuilt attributes they might have (such as, e.g. `<a href=\"..\">` where href is an expected attribute of an anchor tag, most web pages have additional `class` or `id` tags that web designers use in tandem with CSS to control the format of the web page.\n",
    "\n",
    "We aren't too concerned about the details of this (if you want to know more, then you'll need to learn web design...). For our purposes, the important thing about the CSS class and id features is that they often make it easier to search for particular items of interest on a page.\n",
    "\n",
    "So the basic idea is to investigate the web page you are interested in scraping, using the web tools, particularly the inspector, to identify the particular combination of tag, class and id selectors associated with the information you are interested in. Hopefully there is a pattern, and hopefully that will enable automated collection of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enough already, show us some code!\n",
    "OK... to do this we need to install a new module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install BeautifulSoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`BeautifulSoup`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a python module that makes it fairly easy to parse the DOM of a web page, and thus assemble the information of interest in an automated way. Once installed, we need to import it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to point it at some content. To do this we use another module, called [`requests`](http://docs.python-requests.org/en/latest/user/quickstart/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requests allows you to get all the information from a URL. It's pretty easy to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simplepage = requests.get('http://southosullivan.com/gisc425/other/simple-web-page.html')\n",
    "simplepage.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the content of the page at the URL requested is now contained in a text string, and we can analyse it at our leisure.\n",
    "\n",
    "To do anything serious we probably need to request not just fixed web pages, but ones with associated search parameters. This is also easy with `requests`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raywhite = requests.get('https://raywhite.co.nz/search/', params={'regionSelect': 7, 'districtSelect': 20})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't try to look at all the text (it's a big page):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raywhite.text[:5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How did I figure out what parameters to send the site? These are encoded in the extended URL that you see when you've done a search on a website. To figure this out you have to do some sleuthing on the web page. \n",
    "\n",
    "You can check that the correct URL was sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raywhite.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now pick the data apart\n",
    "I had a look at the Ray White web page before, and figured out where the addresses are in the structure of the pages. I can use this information to extract only that information from the mess of nonsense in the raw HTML above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = BeautifulSoup(raywhite.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has returned a `BeautifulSoup` object that can parse HTML just like a web browser does. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how smart it is, we can ask it to make the mess we saw above prettier, and easier to read. Note that I am only asking for the first 1000 characters to save space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(result.prettify()[:5000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main thing we need to be able to do, is to use `BeautifulSoup` to select only the elements we are interested in. It can make selections using HTML tags, HTML attributes or CSS selectors (named classes or ids). So... for example, to get all paragraphs on a page..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.find_all(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can refine searches by specifying classes and other aspects of the tags of interest. Again, this is based on sleuthing on the web page with the developer tools in the browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.find_all(\"h5\", class_ = \"card-title font-lato font-bold text-grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are getting warm! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for element in result.find_all(\"h5\", class_ = \"card-title font-lato font-bold text-grey\"):\n",
    "    print(element.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking a bit more closely at the content, we might do even better with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for element in result.find_all(\"h5\", \"card-title font-lato font-bold text-grey\"):\n",
    "    print(element.text.split(\"\\n\")[1].strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, that is the first page of results. How do we get more? Going back to the website, after a bit more sleuthing we can see how the URL for a series of pages of results is structured with a `result_p` element added to the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time # Now there's a weird command to issue..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here we are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addresses = []\n",
    "# Notice we have a loop starting from 1 and going to 5, not 0 to 4\n",
    "for i in range(1, 6):\n",
    "    page = requests.get(\"https://raywhite.co.nz/search/\", \n",
    "                        params = {\"regionSelect\": 7, \"districtSelect\": 20, \"result_p\": i})\n",
    "    # page = requests.get(\"https://raywhite.co.nz/search/?regionSelect=7&districtSelect=20&result_p=\" + str(i))\n",
    "    print(page.url) # just for some reassurance\n",
    "    pagecontent = BeautifulSoup(page.text)\n",
    "    # add the results to our empty list\n",
    "    for e in pagecontent.find_all(\"h5\", \"card-title font-lato font-bold text-grey\"):\n",
    "        addresses.append(e.text.split(\"\\n\")[1].strip())\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addresses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That, believe it or not, is more or less all you need to know to scrape websites.\n",
    "\n",
    "For fun, you might see if you can do a similar search on one of the other property websites. Or maybe change the example above to look for 2 bedroom rentals.\n",
    "\n",
    "### More\n",
    "There are other web scraping tools out there. One that is popular is [`scrapy`](https://scrapy.org/) a  framework for building web crawlers and scrapers, so it handles more of the complexities of multiple searches and so on.\n",
    "\n",
    "But really that's it! Potentially a useful tool for research projects."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b9476ba453617cf67dc901a6664618619892412b6f80972fafe3f677f6bd1d4c"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 ('g420')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}