**Attribution statement:** _The following teaching materials have been iteratively developed by current and former instructors, including: Friedrich Geiecke, Thomas Robinson and Ryan Hübert._

This brief document discusses character encoding and how to perform simple text searches in R using regular expressions.

## Character encoding 

Loading packages (the `tidyverse` includes the `readr` and `stringr` packages that we are using here):

```{r}
library(tidyverse)
```

First, let's examine how special characters can affect file size. Since `readr` encodes all text files with UTF-8, the accents will require more bytes

```{r}
ddir <- "" # insert file path to the directory with the data

# Write a plain text file with my last name
raw <- "Hübert" # how many bytes should you expect?
write_file(raw, paste0(ddir, "hubert1.txt")) # note: default encoding is UTF-8

# Write a plain text file with my last name, but omit the accent
raw <- "Hubert" # how many bytes should you expect?
write_file(raw, paste0(ddir, "hubert2.txt")) # note: default encoding is UTF-8
```

By default `readr` will encode files it saves as UTF-8. See <https://readr.tidyverse.org/reference/locale.html>. As far as I know, you cannot choose a different encoding when you write a file in `readr` using `write_file`. For the most part, this is not only okay, it's preferable. Let's see why through some examples. 

First, I wrote "&uüДሊ🫠" into three text files using different encodings:

- `utf-examples-8.txt` (UTF-8): 13 bytes
- `utf-examples-16.txt` (UTF-16): 16 bytes
- `utf-examples-32.txt` (UTF-32): 28 bytes 

First note that the UTF-32 encoded file is more than double the size of UTF-8 (just to store the same characters!). Next, let's try to open them.

```{r}
## This works
read_file(paste0(ddir, "utf-examples-8.txt"))  # This works (sort of...)
## These do not work
read_file(paste0(ddir, "utf-examples-16.txt"))  # This drops most of the characters
read_file(paste0(ddir, "utf-examples-32.txt"))  # This drops most of the characters
```

The longer the text (i.e., more characters), the worse the "bloat" from UTF-32 becomes. To see this one more time, in week 4 materials, I saved three files starting with "methodology" which contain the text from the Methodology Department's [About us](https://www.lse.ac.uk/Methodology/About-us/About-us) page. Notice that a file encoded as UTF-8 takes up 3,284 bytes of storage, but the file encoded as UTF-32 takes up 13,124 bytes. It's the exact same text, just stored more inefficiently! 

So when would you ever need to use UTF-16 or UTF-32?

- Many Windows applications encode in UTF-16 by default, so it's good to know this.
- It is more efficient to search in UTF-32 encoded documents (see <https://en.wikipedia.org/wiki/UTF-32>)

Finally, let's look at some Chinese characters. We will try to read a file containing the Chinese-language word "上海" (Shanghai), which is encoded with the [GB_18030](https://en.wikipedia.org/wiki/GB_18030) encoding. Let's pretend that we don't know the document's encoding and we are trying to load into R using `read_file`.


This prints hexadecimal code points mapped to approximately "ÉϺ£": 

```{r, error = TRUE}
read_file(paste0(ddir, "shanghai.txt")) 
```

This does not work at all:

```{r, error = TRUE}
read_file(paste0(ddir, "shanghai.txt"), locale = locale(encoding="utf-8"))
```

This does not print the same characters:

```{r, error = TRUE}
read_file(paste0(ddir, "shanghai.txt"), locale = locale(encoding="utf-16")) 
```

Maybe R can help us?? (Nope!)

```{r, error = TRUE}
guess_encoding(paste0(ddir, "shanghai.txt"))
```

Maybe Terminal can help??

```{bash, eval = FALSE}
$ cd <path>
$ file -I shanghai.txt
shanghai.txt: text/plain; charset=iso-8859-1
```

No, it can't... it guesses the encoding is [ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1), which just extended ASCII. Clearly this is not correct, as we expect Chinese characters.

If we just *happen* to know the encoding, we can finally read the file correctly:

```{r}
read_file(paste0(ddir, "shanghai.txt"), locale = locale(encoding="gb18030"))
```

What a nightmare! But if you find yourself in this situation and are lucky enough to figure out a file's encoding, you should always save a new version encoding in a more standard encoding, like UTF-8. You might even consider using a file name indicating it is encoded with UTF-8 so that the next person will thank you!

Since `readr` encodes as UTF-8 by default, we can do the following:

```{r}
shanghai <- read_file(paste0(ddir, "shanghai.txt"), locale = locale(encoding="gb18030"))
print(shanghai)
write_file(shanghai, paste0(ddir, "shanghai_utf-8.txt"))
```

Now that we've saved with UTF-8 encoding, let's see how much easier it is to read the file:

```{r}
read_file(paste0(ddir, "shanghai_utf-8.txt"))
```

Note here that we no longer have to specify an encoding, as R assumes it is UTF-8, and it is. We get the expected text.

## Regular expressions

Here's a text string from a U.S. federal district court case record, which I showed you in the first lecture.

```{r}
sample_text <- "ORDER OF REASSIGNMENT to District Judge Aileen M Cannon for all further proceedings, Judge Cecilia M. Altonaga no longer assigned to case. Signed by Judge Cecilia M. Altonaga on 11/23/2020. See attached document for full details. (yar) (Entered: 11/23/2020)"
```

Helpful functions are e.g. `str_view`, `str_extract`, `str_extract_all`. The `_all` variant process all matches in the string rather than only the first.

The view function allows to view keyword and/or regular expression matches in R similarly to in a text editor:

```{r}
## Find all the mentions of "judge"
str_view(sample_text, "judge") # this doesn't print anything!
## Note that regex searches are case sensitive
str_view(sample_text, "Judge")
```

```{r}
## Find all parenthese
str_view(sample_text, "(|)") ## This isn't what we expected!!
## Parentheses are special characters in regex, need to put in brackets
str_view(sample_text, "[(]|[)]") ## This isn't what we expected!!
```

The extract function on the other hand allows to extract the matching characters. The following regular expression is one possible way to use the options we discussed this week to extract the times from the file (as the backslash is already used as an escape symbol in normal strings in R, we need to use a double backslash for regular expressions):

```{r}
# \d = a digit
# \d{1,2} = 1-2 digits, e.g. "1", "11", "23"
# [:.\\s-]? = optional set of characters (":","."," ","-")
str_extract(sample_text, "[(]Entered[:] *[0-9/]+[)]")
```

```{r}
str_extract_all(sample_text, "Judge [A-z]+ [A-Z].? [A-z]+")
```

Other helpful functions are e.g. `str_detect` (particularly when all words are elements in a long character vector) and `str_count`. For a detailed discussion of strings and regular expressions in R with the `stringr` package, see http://r4ds.had.co.nz/strings.html and the vignette https://cran.r-project.org/web/packages/stringr/vignettes/regular-expressions.html.