## Using AWS S3 to store files

Note: There is some use of S3 included in the free tier, however, check the pricing in order not to incur unwanted costs and delete buckets/files that you do not need anymore.

EC2 instances have some storage included and we copied files to and from an EC2 instance in the coding session this week. Yet, as soon as you shut down your EC2 instance, these files will be lost. EC2 instances are therefore not particularly well suited for storage. Unless you just run a quick computation and download the files with your ftp client or command line afterwards, it can therefore be helpful to additionally store them in a different location. AWS has the Simple Storage Service (S3) which allows to create buckets that can store data, other cloud providers have similar products. The following script shows how you can backup the .csv and .sqlite file from our scraping example in an S3 bucket via R. You could insert such code into your scraping script running in the cloud to load the files from S3 in the beginning when running the script and then store them again in S3 at the end of the script. In between while the script is running, the files would be stored and appended on EC2 just as in the coding session, but could additionally be backed up into S3 in regular intervals. The same could be done e.g. with Dropbox and packages such as [rdrop2](https://cran.r-project.org/web/packages/rdrop2/index.html) which allow to backup/store files in a similar manner.

Loading packages:

```{r}
#install.packages("aws.s3")
library("aws.s3")
```

Under "Services" -> "Storage" -> "S3" you can click on "Create bucket" to create a bucket. Then under "Security, Identity, & Compliance" -> "IAM" click on "Users" -> "Add User" with rights to access S3 (note that the following [tutorial](https://www.gormanalysis.com/blog/connecting-to-aws-s3-with-r/) can be helpful also for this step). You can delete/create a new key under "Security credentials" for this user. Paste the access key ID, secret access key, and the region of the storage you created into the following code chunk:

```{r}
Sys.setenv(
  "AWS_ACCESS_KEY_ID" = "tba",
  "AWS_SECRET_ACCESS_KEY" = "tba",
  "AWS_DEFAULT_REGION" = "eu-west-2"
)
```

The `bucketlist()` function shows all buckets of the user and should display the one you just created:

```{r}
bucketlist()
```

Say you want to save two files which are currently stored in a directory on your EC2 instance also in a bucket within S3. The following code achieves this for the csv file and sqlite file from the lecture (you might need to add the full path to the files into the file = ... argument of the function).

```{r}

# csv file
put_object(
  file = "featured_articles.csv", # change path if contained in a different folder on your computer or EC2
  object = "featured_articles.csv", 
  bucket = "yourbucketname"
)

# SQLite file
put_object(
  file = "wikipedia.sqlite", # change path if contained in a different folder on your computer or EC2
  object = "wikipedia.sqlite", 
  bucket = "yourbucketname"
)

```

In contrast, the following code allows to download the files from the S3 bucket and store them on your computer or the EC2 instance:

```{r}
save_object(object = "s3://yourbucketname/featured_articles.csv", file = "featured_articles_downloaded_from_s3.csv")
save_object(object = "s3://yourbucketname/wikipedia.sqlite", file = "wikipedia_downloaded_from_s3.sqlite")
```


References

- https://www.gormanalysis.com/blog/connecting-to-aws-s3-with-r/
- https://cran.r-project.org/web/packages/aws.s3/readme/README.html
- https://cloudyr.github.io/

