Loading packages:

```{r}
library(rvest)
library(tidyverse)
library(xml2)
```

In this example, we will scrape an RSS feed and some articles from the [The Guardian](www.theguardian.com). Combining the techniques we have covered in the course so far, the goal is to produce a tabular dataset in which one line represents an article.

You can read through the [Guardian's RSS documentation](https://www.theguardian.com/help/feeds). As you can see, RSS is provided for each of the news category and the url is always https://www.theguardian.com/####/rss. You can even find some sub category of a specific news category. For instance, the following works: [https://www.theguardian.com/world/japan/rss](https://www.theguardian.com/world/japan/rss). As an example, let us look at the RSS feed about space-related articles:

```{r}
url <- "https://www.theguardian.com/science/space/rss"
```

Before scraping, we can check the structure of the xml document in the browser. We need to know what the element for each article is called and find the xml tag "item". Next, we read in the document and extract these elements.

Note: `xml_nodes()` was deprecated in rvest 1.0.0 and it is advised to use `html_elements()` instead, also for xml files.

```{r}
# Reading the xml document
rss_xml <- read_xml(url)
Sys.sleep(2) # You should always put delays after sending requests to webservers -- this is good etiquette! 

# Extracting the items in the RSS feed
elements <- html_elements(rss_xml, css = "item")
length(elements)
```

From these elements, we extract titles, description, dates, and urls, and combine everything in a data frame:

```{r}
# Extract titles
title <- elements %>%
  html_element("title") %>%
  xml_text()

# Extract description
description <- elements %>%
  html_element("description") %>%
  xml_text() %>%
  str_replace_all("<.+?>", " ") # removes residual tags
  
# Extract date and process as datetime
datetime <- elements %>%
  html_element("dc[:]date") %>% 
  xml_text() %>%
  parse_datetime() 

# Extract url
article_url <- elements %>%
  html_element("guid") %>%
  xml_text()

# Combine everything in a data frame/tibble
data_articles <- tibble(title,
                        description,
                        datetime,
                        article_url,
                        full_text = "")
```

This is the data frame:

```{r}
data_articles
```

Next, let us prototype how you could scrape the text in the body of each of those URLs. We pick the first URL and write some code to get an object that contains the text of the article. We select all paragraphs in the file.

```{r}
test_url <- article_url[1]

# Note: As the RSS feed changes very frequently, copy the url contained in
# `article_url[1]` into your browser to have a look at the first article in
# your specific list

test_text <- test_url %>%
  read_html() %>%
  html_elements(css = "p") %>% 
  html_text() %>%
  paste(collapse = "\n")

test_text
```

Now that the code works, let us put it into a function that generalises to all URLs in the dataset. This function can be used with a loop or apply later. Recall that we have an empty column in the dataframe called 'text' which we can now fill. Each iteration of e.g. a loop fills the ith element of that vector with the text of the article.

```{r}
text_extractor <- function(current_url, sec = 2){

  current_text <- current_url %>%
    read_html() %>% 
    html_elements(css = "p") %>%
    html_text() %>%
    paste(collapse = "\n") 

  Sys.sleep(sec)
  
  return(current_text)
}
```

Before we begin the loop, let us reduce the dataset to only the first 5 articles to save some time in this illustration:

```{r}
# Storing only the first 5 rows of the dataframe
data_articles_small <- data_articles[1:5,]
```

For loop:

```{r}
for (i in 1:nrow(data_articles_small)) {
  
  # Currently scraping article ...
  print(i)
  
  # Obtain URL of current article
  current_url <- as.character(data_articles[i,"article_url"])
  
  # Obtain full text of article
  data_articles_small[i, "full_text"] <- text_extractor(current_url)
  
}
```

In R, however, we often use apply functions which offer a more compact notation. These functions also often perform computations more quickly. But in this situation, we are purposefully slowing down our code (good scraping etiquette!), so an apply function won't be a big improvement over the loop in terms of speed. 

The function `sapply()` allows to apply another function element-wise to a vector and returns a vector. This vector is assigned to the dataframe as a new column (takes a little time to run):

```{r}
data_articles_small$full_text_2 <- sapply(data_articles_small$article_url, text_extractor)
```

Both the for loop and the sapply approach yielded the same outcome:

```{r}
data_articles_small
```