Skip to contents

The pins package provides a robust set of functions to read and write standard types of files using standard tools, e.g. CSV files using read.csv() and write.csv(). However, from time to time, you may wish read or write using other tools. You may want to read and write:

  • CSV files using readr or vroom
  • Arrow files without using compression
  • Parquet files
  • Whole directories that are archived/zipped

An escape hatch for a customized approach is provided: pin_upload() and pin_download(). The goal of this vignette is to show how you can incorporate these into your workflow.

Two points to keep in mind:

We’ll follow an example where we write and read uncompressed Arrow files, starting by creating a temporary board:

library(pins)

board <- board_temp()

Upload a one-off file

If you are writing a one-off file, you can do everything directly:

pin_name <- "mtcars-arrow"

# file name will be `mtcars-arrow.arrow`
path <- fs::path_temp(fs::path_ext_set(pin_name, "arrow"))

arrow::write_feather(mtcars, path, compression = "uncompressed")

pin_upload(board, paths = path, name = pin_name)
#> Creating new version '20230121T182759Z-d9859'

Reading from the downloaded pin is straightforward; pin_download() returns a local path that can be piped to arrow::read_feather():

mtcars_download <- 
  pin_download(board, pin_name) %>%
  arrow::read_feather()

head(mtcars_download)
#>    mpg cyl disp  hp drat    wt  qsec vs am gear carb
#> 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
#> 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
#> 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
#> 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
#> 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
#> 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Function to manage uploading

If you want to write more than one custom file of a certain type, or using a certain tool, you might consider writing a helper function:

pin_upload_arrow <- function(board, x, name, ...) {
  # path deleted when `pin_upload_arrow()` exits
  path <- fs::path_temp(fs::path_ext_set(name, "arrow"))
  withr::defer(fs::file_delete(path))
 
  # custom writer
  arrow::write_feather(x, path, compression = "uncompressed")

  pin_upload(board, paths = path, name = name, ...) 
}

This helper function is designed to work like pin_write():

pin_upload_arrow(board, x = mtcars, name = "mtcars-arrow2")
#> Creating new version '20230121T182800Z-d9859'

As before, you can pipe the result of pin_download() to your reader function:

pin_download(board, name = "mtcars-arrow2") %>%
  arrow::read_feather() %>%
  head()
#>    mpg cyl disp  hp drat    wt  qsec vs am gear carb
#> 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
#> 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
#> 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
#> 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
#> 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
#> 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Another example function

If you want to use this same approach to archive and pin a whole directory, you can write a helper function like:

pin_upload_archive <- function(board, dir, name, ...) {
  path <- fs::path_temp(fs::path_ext_set(name, "tar.gz"))
  withr::defer(fs::file_delete(path))
  archive::archive_write_dir(path, dir)
  pin_upload(b, paths = path, name = name, ...)
}

You can download the compressed archive via pin_download(board, name) and then pipe that path straight to archive::archive_extract() to extract your archive in a new diretory.