plumber R package on R Views

Plumber Logging

Tue, 13 Aug 2019 00:00:00 +0000

The plumber R package is used to expose R functions as API endpoints. Due to plumber’s incredible flexibility, most major API design decisions are left up to the developer. One important consideration to be made when developing APIs is how to log information about API requests and responses. This information can be used to determine how plumber APIs are performing and how they are being utilized.

An example of logging API requests in plumber is included in the package documentation. That example uses a filter to log information about incoming requests before a response has been generated. This is certainly a valid approach, but it means that the log cannot contain details about the response since the response hasn’t been created yet. In this post we will look at an alternative approach to logging plumber APIs that uses preroute and postroute hooks to log information about each API request and its associated response.

Logging

In this example, we will use the logger package to generate the actual log entries. Using this package isn’t required, but it does provide some convenient functionality that we will take advantage of.

Since we will be registering hooks for our API, we will need both a plumber.R file and an entrypoint.R file. The plumber.R file contains the following:

# plumber.R
# A simple API to illustrate logging with Plumber

library(plumber)

#* @apiTitle Logging Example

#* @apiDescription Simple example API for implementing logging with Plumber

#* Echo back the input
#* @param msg The message to echo
#* @get /echo
function(msg = "") {
  list(msg = paste0("The message is: '", msg, "'"))
}

#* Plot a histogram
#* @png
#* @get /plot
function() {
  rand <- rnorm(100)
  hist(rand)
}

Now that we’ve defined two endpoints (/echo and /plot), we can use entrypoint.R to setup logging using preroute and postroute hooks. First, we need to configure the logger package:

# entrypoint.R
library(plumber)

# logging
library(logger)

# Specify how logs are written
log_dir <- "logs"
if (!fs::dir_exists(log_dir)) fs::dir_create(log_dir)
log_appender(appender_tee(tempfile("plumber_", log_dir, ".log")))

The log_appender() function is used to specify which appender method is used for logging. Here we use appender_tee() so that logs will be written to stdout and to a specific file path. We create a directory called logs/ in the current working directory to store the resulting logs. Every log file is assigned a unique name using tempfile(). This prevents errors that can occur if concurrent processes try to write to the same file.

Now, we need to create a helper function that we will use when creating log entries:

convert_empty <- function(string) {
  if (string == "") {
    "-"
  } else {
    string
  }
}

This function takes an empty string and converts it into a dash ("-"). We will use this to ensure that empty log values still get recorded so that it is easy to read the log files. We’re now ready to create our plumber router and register the hooks necessary for logging:

pr <- plumb("plumber.R")

pr$registerHooks(
  list(
    preroute = function() {
      # Start timer for log info
      tictoc::tic()
    },
    postroute = function(req, res) {
      end <- tictoc::toc(quiet = TRUE)
      # Log details about the request and the response
      log_info('{convert_empty(req$REMOTE_ADDR)} "{convert_empty(req$HTTP_USER_AGENT)}" {convert_empty(req$HTTP_HOST)} {convert_empty(req$REQUEST_METHOD)} {convert_empty(req$PATH_INFO)} {convert_empty(res$status)} {round(end$toc - end$tic, digits = getOption("digits", 5))}')
    }
  )
)

pr

We use the $registerHooks() method to register both preroute and postroute hooks. The preroute hook uses the tictoc package to start a timer. The postroute hook stops the timer and then writes a log entry using the log_info() function from the logger package. Each log entry contains the following information:

Log level: This is a distinction made by the logger package, and in this example the value is always INFO
Timestamp: The timestamp for when the response was generated and sent back to the client
Remote Address: The address of the client making the request
User Agent: The user agent making the request
Http Host: The host of the API
Method: The HTTP method attached to the request
Path: The specific API endpoint requested
Status: The HTTP status of the response
Execution Time: The amount of time from when the request received until the response was generated

This log format is loosely inspired by the NCSA Common log format.

Testing

Now that our API is all setup, it’s time to test to make sure logging works as expected. First, we need to start the API. The easiest way to do this is to click the Run API button that appears at the top of the plumber.R file in the RStudio IDE. Once the API is running, you’ll see a message in the console similar to the following:

Running plumber API at http://127.0.0.1:5762

Now that we know the API is running, we need to make a request. One of the easiest ways to make a request in this case is to open a web browser (like Google Chrome) and type the API address in the address bar followed by /plot. In this example, I would type http://127.0.0.1:5762/plot into the address bar of my browser. If all goes well, you should see a plot rendered in the browser. The RStudio console will display the log output:

INFO [2019-08-09 12:30:23] 127.0.0.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" localhost:5762 GET /plot 200 0.158

A new logs/ directory will have been created in the current working directory and it will contain a file with the log entry. You can generate more log entries by refreshing your browser window.

Analyzing

Let’s say that we refreshed the browser window 1,000 times. The log file generated will contain an entry for each request. We can analyze this log file to find helpful information about the API. For example, we could plot a histogram of execution time:

library(ggplot2)

plumber_log <- readr::read_log("logs/plumber_fe3daed895d.log",
                               col_names = c("log_level",
                                             "timestamp",
                                             "remote_address",
                                             "user_agent",
                                             "http_host",
                                             "method",
                                             "path",
                                             "status",
                                             "execution_time"))

ggplot(plumber_log, aes(x = execution_time)) +
  geom_histogram() +
  theme_bw() +
  labs(title = "Execution Times",
       x = "Execution Time")

We could even build a Shiny application to monitor the logs/ directory and provide real-time visibility into API metrics!

The details of this Shiny application go beyond the scope of this post, but the source code is available here.

Plumber APIs published to RStudio Connect can use this pattern to log and monitor API requests. Details on this use case can be found in this repository

Conclusion

Plumber is an incredibly flexible package for exposing R functions as API endpoints. Logging information about API requests and responses provides visibility into API usage and performance. These log files can be manually inspected or used in connection with other tools (like Shiny) to provide real-time metrics around API use. The code used in this example along with additional information is available in this GitHub repository.

If you are interested in learning more about using plumber, logging, Shiny, and RStudio Connect, please visit community.rstudio.com and let us know!

James Blair is a solutions engineer at RStudio who focuses on tools, technologies, and best practices for using R in the enterprise.

Slack and Plumber, Part Two

Tue, 27 Nov 2018 00:00:00 +0000

This is the final entry in a three-part series about the plumber package. The first post introduces plumber as an R package for building REST API endpoints in R. The second post builds a working example of a plumber API that powers a Slack slash command. In this final entry, we will secure the API created in the previous post so that it only responds to authenticated requests, and deploy it using RStudio Connect.

As a reminder, this API is built on top of simulated customer call data. The slash command we create will allow users to view a customer status report within Slack. This status report contains customer name, total calls, date of birth, and a plot of call history for the past 20 weeks. The simulated data, along with the script used to create it, can be found in the GitHub repository for this example.

Setup

Successfully following this example assumes you have created a Slack account and you have followed the instructions for creating an app. The Plumber API as it currently exists is described in detail in the previous post.

This API can be run through the UI as previously described, or by running plumber::plumb("plumber.R")$run(port = 5762) from the directory containing the API defined in plumber.R. As it stands now, this API could be deployed and used by Slack. However, it’s important to remember that we have no control over the request that Slack makes to the API. Because of this, we can’t rely on RStudio Connect’s built-in API authentication mechanism to secure the API because there is no way to submit a key with the request. Our options are either to expose the API with no security, meaning anyone can access the endpoints we’ve defined, or to find some other mechanism for securing the API so that it only responds to authorized requests.

API Security Patterns

The plumber documentation provides a good introduction to API security for the R user:

The majority of R programmers have not been trained to give much attention to the security of the code that they write. This is for good reason since running R code on your own machine with no external input gives little opportunity for attackers to leverage your R code to do anything malicious. However, as soon as you expose an API on a network, your concerns and thought process must adapt accordingly.

API security can be challenging to address. As it stands today, it is the developer’s responsibility to provide proper security on API endpoints, though in the future, there may be additional security features added to plumber or available via other R packages.

As mentioned in the plumber documentation, there are a number of things to consider when designing API security. For example, if the API is deployed on an internal network, securing the API may not be as important as it would be if the API was publicly exposed on the internet. When an API needs to be secured, there are several potential attack vectors that need to be handled. In this specific example, we are exposing a public endpoint that provides access to sensitive customer data. If we are unable to authenticate incoming requests, then we risk exposing sensitive data. To prevent this data from falling into the wrong hands, we will focus on verifying incoming requests so that the API only responds to requests made from Slack.

There are several different methods for authenticating requests made to API endpoints. One common method is the use of API keys, which are cryptographically secure values sent with the request to verify the identity of the client. However, in this case, we have no control over the request Slack sends, so we cannot include such a key in the request. Thankfully, Slack has provided an alternative authentication method using signed secrets. Full details can be read in the Slack documentation, but in essence, each Slack application is assigned a unique secret value that, when used in connection with other request details, can be used to verify that an incoming request is indeed coming from Slack and not an unknown third party.

Securing the API

In order to secure our API so that only requests from Slack are honored, we first need to obtain the signing secret for our application. This value can be found in the Basic Information section of the Slack application settings. Now, it is important to remember that this is called a signing secret for a reason: it should not be shared with anyone. To avoid exposing this secret, we can save it as an environment variable. We add this in a current R session by using Sys.setenv(SLACK_SIGNING_SECRET = <our signing secret>), or we can add it to our .Renviron file so that it is set for every R session. Once this is done, we can access this value in R using Sys.getenv("SLACK_SIGNING_SECRET"). Now we are ready to create a function to verify if incoming requests are from Slack.

Slack provides the following three-step process for verifying requests:

Your app receives a request from Slack
Your app computes a signature based on the request
You make sure the computed signature matches the signature on the request

In order to verify all incoming requests, we can define an additional filter for our API that follows the above recipe.

#* Verify incoming requests
#* @filter verify
function(req, res) {
  # Forward requests coming to swagger endpoints
  if (grepl("swagger", tolower(req$PATH_INFO))) return(forward())

  # Check for X_SLACK_REQUEST_TIMESTAMP header
  if (is.null(req$HTTP_X_SLACK_REQUEST_TIMESTAMP)) {
    res$status <- 401
  }

  # Build base string
  base_string <- paste(
    "v0",
    req$HTTP_X_SLACK_REQUEST_TIMESTAMP,
    req$postBody,
    sep = ":"
  )

  # Slack Signing secret is available as environment variable
  # SLACK_SIGNING_SECRET
  computed_request_signature <- paste0(
    "v0=",
    openssl::sha256(base_string, Sys.getenv("SLACK_SIGNING_SECRET"))
  )

  # If the computed request signature doesn't match the signature provided in the
  # request, set status of response to 401
  if (!identical(req$HTTP_X_SLACK_SIGNATURE, computed_request_signature)) {
    res$status <- 401
  } else {
    res$status <- 200
  }

  if (res$status == 401) {
    list(
      text = "Error: Invalid request"
    )
  } else {
    forward()
  }
}

There are a lot of moving pieces to this filter, but essentially we are following the process outlined by Slack for verifying requests. We also allow Swagger endpoints to be served without verification so that the Swagger UI can still be generated for our API.

Once this filter is in place, all incoming requests will be verified. However, this will create issues with our /plot/history/ endpoint since it is called using a standard GET request without any Slack authentication. To ensure that this endpoint is able to be utilized as we want, we’ll make some small updates to the endpoint and add #* @preempt verify to the plumber comments before the function. This prevents the verify filter from applying to this endpoint.

Now, this prevents the Slack authentication process from applying to our plot endpoint. However, this endpoint, if left unsecured, provides unfiltered access to sensitive customer data. We need an effective way to secure this endpoint so that it only responds to requests generated from Slack.

Since the only thing we control in the request to this endpoint is the URL, we can update our endpoint so that an encrypted parameter is passed as part of the URL. This parameter is a combination of the current datetime and the customer ID that is then encrypted using our Slack signing secret. We can use the encrypt_string() function from the safer package to securely encrypt this string. The following example illustrates this process.

current_time <- Sys.time()
customer_id <- 89
parameter_string <- paste(current_time, customer_id, sep = ";")
safer::encrypt_string(parameter_string, Sys.getenv("SLACK_SIGNING_SECRET"))

## [1] "m7NfMZfpY1n5EuivjuiFQsyKopT68HiX+NIgk5S+VBlDHrVqzRM="

Once we have created this encrypted value, we pass it to the URL of our plot endpoint. Then, within the plot endpoint, we decrypt the string, extract the customer ID, and check to see if the current time is within five seconds of the time encoded in the string. If more than five seconds have passed, we consider the request to be unauthorized. To help with this process, we define two helper functions:

encrypt_string <- function(string) {
  urltools::url_encode(safer::encrypt_string(paste(Sys.time(), string, sep = ";"),
                                             key = Sys.getenv("SLACK_SIGNING_SECRET")))
}

plot_auth <- function(endpoint, time_limit = 5) {
  # Save current time to compare against endpoint time value
  current_time <- Sys.time()

  # Try to decrypt endpoint and extract user id
  tryCatch({
    # Decrypt endpoint using SLACK_SIGNING_SECRET
    decrypted_endpoint <- safer::decrypt_string(endpoint,
                                                key = Sys.getenv("SLACK_SIGNING_SECRET"))
    # Split endpoint on ;
    endpoint_split <- unlist(strsplit(decrypted_endpoint, split = ";"))
    # Convert time
    endpoint_time <- as.POSIXct(endpoint_split[1])
    # Calculate time difference
    time_diff <- difftime(current_time, endpoint_time, units = "secs")

    # If more than 5 seconds have passed since the request was generated, then
    # error
    if (time_diff > time_limit) {
      "Unauthorized"
    } else {
      endpoint_split[2]
    }
  },
  error = function(e) "Unauthorized"
  )
}

Once these helper functions are in place, we can update our /plot/history endpoint as follows:

#* Plot customer weekly calls
#* @png
#* @param cust_secret encrypted value calculated in /status endpoint
#* @response 400 No customer with the given ID was found.
#* @preempt verify
#* @get /plot/history
function(res, cust_secret) {
  # Authenticate that request came from /status
  cust_id <- plot_auth(cust_secret)

  # Return unauthorized error if cust_id is "Unauthorized"
  if (cust_id == "Unauthorized") {
    res$status <- 401
    stop("Unauthorized request")
  } else if (!cust_id %in% sim_data$id) {
    res$status <- 400
    stop("Customer id" , cust_id, " not found.")
  }

  # Filter data to customer id provided
  plot_data <- dplyr::filter(sim_data, id == cust_id)

  # Customer name (id)
  customer_name <- paste0(unique(plot_data$name), " (", unique(plot_data$id), ")")

  # Create plot
  history_plot <- plot_data %>%
    ggplot(aes(x = time, y = calls, col = calls)) +
    ggalt::geom_lollipop(show.legend = FALSE) +
    theme_light() +
    labs(
      title = paste("Weekly calls for", customer_name),
      x = "Week",
      y = "Calls"
    )

  # print() is necessary to render plot properly
  print(history_plot)
}

Now we need to make one small change to our /status endpoint so that we properly build the appropriate URL for our image. We construct the list response to the /status endpoint as follows, where image_url has been updated.

    attachments = list(
      list(
        color = customer_status,
        title = paste0("Status update for ", customer_name, " (", customer_id, ")"),
        fallback = paste0("Status update for ", customer_name, " (", customer_id, ")"),
        # History plot

        image_url = paste0(base_url,
                           "/plot/history?cust_secret=",
                           encrypt_string(customer_id)),
        # Fields provide a way of communicating semi-tabular data in Slack
        fields = list(
          list(
            title = "Total Calls",
            value = sum(customer_data$calls),
            short = TRUE
          ),
          list(
            title = "DoB",
            value = unique(customer_data$dob),
            short = TRUE
          )
        )
      )
    )

Just like that, we have a secure API!

All Together Now

Now, given the authorization pieces we have implemented, it is a bit more difficult to test our API since our endpoints will only respond to authorized requests. However, we can use the free version of Postman to test our API. An in-depth look at the capabilities of Postman is beyond the scope of this post, so hopefully a gif will suffice. Further details about using Postman in this context can be found in the GitHub repository for this example.

It appears that everything is working as expected! Our endpoints fail when the authorization criteria are not met, and otherwise they succeed. Notice that the plot endpoint works when initially called, but when a subsequent call is made it fails since more than five seconds have passed since the /status endpoint was invoked.

Now, the final step in this process is publishing this API so that Slack can properly interact with it. The easiest way to do this is to publish the API to RStudio Connect. Once published, Slack can be updated to point the Slash command to our nice, newly secured API.

Conclusion

This brings us to the conclusion of this series. We’ve discovered the power of plumber in exposing R to downstream consumers via RESTful API endpoints. We built a Slack app powered entirely by R and plumber, and now we have secured the underlying API so that it only responds to authorized requests. As we have seen, plumber provides a powerful and flexible framework for exposing R functions as APIs. These APIs can be safely secured so that only authorized requests are permitted.

James Blair is a solutions engineer at RStudio who focuses on tools, technologies, and best practices for using R in the enterprise.

Slack and Plumber, Part One

Thu, 30 Aug 2018 00:00:00 +0000

In the previous post, we introduced plumber as a way to expose R processes and programs to external systems via REST API endpoints. In this post, we’ll go further by building out an API that powers a Slack slash command, all from within R using plumber. A subsequent post will outline deploying and securing the API.

We will create an API built on top of simulated customer call data that powers a slash command. This command will allows users to view a customer status report within Slack. As shown, this status report contains customer name, total calls, date of birth, and a plot of call history for the past 20 weeks. The simulated data, along with the script used to create it, can be found in the GitHub repository for this example.

Setup

Slack is a commonly used communication tool that’s highly customizable through various integrations. It’s even possible to build your own integrations, which is what we’ll be doing here. In order to build a Slack app, you need to have a Slack account and follow the instructions for creating an app. In this example, we will build an app that includes a slash command that users can access by typing /<command-name> into a Slack message.

The Slack request

In this scenario, we’re building an API that will interact with a known request. This means that we need to understand the nature of the incoming request so that we can appropriately handle it within the API. Slack provides some documentation about the request that is sent when a slash command is invoked. In short, an HTTP POST request is made that contains a URL-encoded data payload. An example data payload looks like:

token=gIkuvaNzQIHg97ATvDxqgjtO
&team_id=T0001
&team_domain=example
&enterprise_id=E0001
&enterprise_name=Globular%20Construct%20Inc
&channel_id=C2147483705
&channel_name=test
&user_id=U2147483697
&user_name=Steve
&command=/weather
&text=94070
&response_url=https://hooks.slack.com/commands/1234/5678
&trigger_id=13345224609.738474920.8088930838d88f008e0

There’s a lot of detail included in the Slack request, and the Slack documentation provides details about each field. We’re mainly interested in the text field, which contains the text entered into Slack after the slash command. In the above example, the user entered /weather 94070 into Slack, so the request indicates that the command was /weather and the text was 94070.

Note that this approach is different from APIs that are not being built around a known request or specification. In such instances, we are free to expose endpoints and return data in whatever method seems most beneficial to downstream consumers. In such a scenario, we would provide downstream API consumers with an understanding of how the API handles requests and what types of responses are generated so that they can appropriately interact with the API. But in this example, the design of our API is, in part, dictated by the specifications Slack provides.

Building the API

Now that we have an understanding of what is included in the incoming request, we can begin to build out the API using plumber. First, we need to set up the global environment for the API by loading necessary packages and global objects, including the simulated data this API is built on. In reality, this data would likely come from an external database accessed via an ODBC connection.

# Packages ----
library(plumber)
library(magrittr)
library(ggplot2)

# Data ----
# Load sample customer data
sim_data <- readr::read_rds("data/sim-data.rds")

The following diagram outlines what we want to build.

In essence, an incoming request will pass through two filters before reaching an endpoint. This first filter is responsible for routing incoming requests to the correct endpoint. This is done so that a single slash command can serve multiple endpoints without the need to create separate commands for each service. The second filter simply logs details about the request for future review.

Filters

The first filter is responsible for parsing the incoming request and ensuring it is assigned to the appropriate endpoint. This is done because when a slash command is created in Slack, there is only one endpoint defined for requests made from the command. This filter enables several endpoints to be utilized by the same slash command by parsing the incoming text of the command and treating the first value of that command as the endpoint to which the request should be routed.

#* Parse the incoming request and route it to the appropriate endpoint
#* @filter route-endpoint
function(req, text = "") {
  # Identify endpoint
  split_text <- urltools::url_decode(text) %>%
    strsplit(" ") %>%
    unlist()
  
  if (length(split_text) >= 1) {
    endpoint <- split_text[[1]]
    
    # Modify request with updated endpoint
    req$PATH_INFO <- paste0("/", endpoint)
    
    # Modify request with remaining commands from text
    req$ARGS <- split_text[-1] %>% 
      paste0(collapse = " ")
  }
  
  # Forward request 
  forward()
}

This filter requires an understanding of the req object. It’s important to note that a few things happen in this filter. First, we parse the text argument and use the first part of text as the req$PATH_INFO, which tells plumber where to route the request. Second, we take anything remaining from text and attach it to the request in req$ARGS. This means that any downstream filters or endpoints will have access to req$ARGS. The second filter is taken straight from the plumber documentation and simply logs details about the incoming request.

#* Log information about the incoming request
#* @filter logger
function(req){
  cat(as.character(Sys.time()), "-", 
      req$REQUEST_METHOD, req$PATH_INFO, "-", 
      req$HTTP_USER_AGENT, "@", req$REMOTE_ADDR, "\n")
  
  # Forward request
  forward()
}

Endpoints

There are a few endpoints we need to define. First, we need to define an endpoint that provides a response Slack can understand and interpret into a message. In this case, we’re going to return a JSON object that Slack interprets into a message with attachments. Slack provides detailed documentation on what fields it accepts in a response. Also note that Slack expects unboxed JSON, while the plumber default is to return boxed JSON. In order to ensure that Slack understands the response, we set the serializer for this response to be unboxedJSON.

#* Return a message containing status details about the customer
#* @serializer unboxedJSON
#* @post /status
function(req, res) {
  # Check req$ARGS and match to customer - if no customer match is found, return
  # an error
  
  customer_ids <- unique(sim_data$id)
  customer_names <- unique(sim_data$name)
  
  if (!as.numeric(req$ARGS) %in% customer_ids & !req$ARGS %in% customer_names) {
    res$status <- 400
    return(
      list(
        response_type = "ephemeral",
        text = paste("Error: No customer found matching", req$ARGS)
      )
    )
  }
  
  # Filter data to customer data based on provided id / name
  if (as.numeric(req$ARGS) %in% customer_ids) {
    customer_id <- as.numeric(req$ARGS)
    customer_data <- dplyr::filter(sim_data, id == customer_id)
    customer_name <- unique(customer_data$name)
  } else {
    customer_name <- req$ARGS
    customer_data <- dplyr::filter(sim_data, name == customer_name)
    customer_id <- unique(customer_data$id)
  }
  
  # Simple heuristics for customer status
  total_customer_calls <- sum(customer_data$calls)
  
  customer_status <- dplyr::case_when(total_customer_calls > 250 ~ "danger",
                                      total_customer_calls > 130 ~ "warning",
                                      TRUE ~ "good")
  
  # Build response
  list(
    # response type - ephemeral indicates the response will only be seen by the
    # user who invoked the slash command as opposed to the entire channel
    response_type = "ephemeral",
    # attachments is expected to be an array, hence the list within a list
    attachments = list(
      list(
        color = customer_status,
        title = paste0("Status update for ", customer_name, " (", customer_id, ")"),
        fallback = paste0("Status update for ", customer_name, " (", customer_id, ")"),
        # History plot
        image_url = paste0("localhost:5762/plot/history/", customer_id),
        # Fields provide a way of communicating semi-tabular data in Slack
        fields = list(
          list(
            title = "Total Calls",
            value = sum(customer_data$calls),
            short = TRUE
          ),
          list(
            title = "DoB",
            value = unique(customer_data$dob),
            short = TRUE
          )
        )
      )
    )
  )
}

There are three main things that happen in this endpoint. First, we check to ensure that the provided customer name or ID appear in the dataset. Next, we create a subset of the data for only the identified customer and use a simple heuristic to determine the customer’s status. Finally, we put a list together that will be serialized into JSON in response to requests made to this endpoint. This list conforms to the standards outlined by Slack.

The second endpoint is used to provide the history plot that is referenced in the first endpoint. When an image_url is provided in a Slack attachment, Slack uses a GET request to fetch the image from the URL. So, this endpoint responds to incoming GET requests with an image.

#* Plot customer weekly calls
#* @png
#* @param cust_id ID of the customer
#* @get /plot/history/<cust_id:int>
function(cust_id, res) {
  # Throw error if cust_id doesn't exist in data
  if (!cust_id %in% sim_data$id) {
    res$status <- 400
    stop("Customer id" , cust_id, " not found.")
  }
  
  # Filter data to customer id provided
  plot_data <- dplyr::filter(sim_data, id == cust_id)
  
  # Customer name (id)
  customer_name <- paste0(unique(plot_data$name), " (", unique(plot_data$id), ")")
  
  # Create plot
  history_plot <- plot_data %>%
    ggplot(aes(x = time, y = calls, col = calls)) +
    ggalt::geom_lollipop(show.legend = FALSE) +
    theme_light() +
    labs(
      title = paste("Weekly calls for", customer_name),
      x = "Week",
      y = "Calls"
    )
  
  # print() is necessary to render plot properly
  print(history_plot)
}

Once these pieces are together, you can run the API either through the UI as described in the previous post, or by running plumber::plumb("plumber.R")$run(port = 5762) from the directory containing the API.

Testing the API

Once the API is up and running, we can test it to make sure it’s behaving as we expect. Since the main point of contact is making a POST request to the /status endpoint, it’s easiest to interact with the API through curl.

$ curl -X POST --data '{"text":"status 1"}' localhost:5762 | jq '.'
{
  "response_type": "ephemeral",
  "attachments": [
    {
      "color": "good",
      "title": "Status update for Rahul Wilderman IV (001)",
      "fallback": "Status update for Rahul Wilderman IV (001)",
      "image_url": "localhost:5762/plot/history/001",
      "fields": [
        {
          "title": "Total Calls",
          "value": 27,
          "short": true
        },
        {
          "title": "DoB",
          "value": "2004-04-01",
          "short": true
        }
      ]
    }
  ]
}

Success! Our API successfully routed our request to the appropriate endpoint and returned a valid JSON response. As a final check, we can visit the image_url in our browser to see if the plot is properly rendered.

Everything appears to be running as expected!

Conclusion

In this post, we used plumber to create an API that can properly interact with the Slack slash command interface. In the next post, we will explore API security and deployment. Continuing with this example, we will secure our API using Slack’s guidelines, deploy the API, and finally connect Slack so that we can use our new slash command. At the conclusion of the next post, we will have a fully functioning Slack slash command, all built using R.

James Blair is a solutions engineer at RStudio who focusses on tools, technologies, and best practices for using R in the enterprise.

REST APIs and Plumber

Mon, 23 Jul 2018 00:00:00 +0000

Moving R resources from development to production can be a challenge, especially when the resource isn’t something like a shiny application or rmarkdown document that can be easily published and consumed. Consider, as an example, a customer success model created in R. This model is responsible for taking customer data and returning a predicted outcome, like the likelihood the customer will churn. Once this model is developed and validated, there needs to be some way for the model output to be leveraged by other systems and individuals within the company.

Traditionally, moving this model into production has involved one of two approaches: either running customer data through the model on a batch basis and caching the results in a database, or handing the model definition off to a development team to translate the work done in R into another language, such as Java or Scala. Both approaches have significant downsides. Batch processing works, but it misses real-time updates. For example, if the batch job runs every night and a customer calls in the next morning and has a heated conversation with support, the model output will have no record of that exchange when the customer calls the customer loyalty department later the same day to cancel their service. In essence, model output is served on a lag, which can sometimes lead to critical information loss. However, the other option requires a large investment of time and resources to convert an existing model into another language just for the purpose of exposing that model as a real-time service. Neither of these approaches is ideal; to solve this problem, the optimal solution is to expose the existing R model as a service that can be easily accessed by other parts of the organization.

plumber is an R package that allows existing R code to be exposed as a web service through special decorator comments. With minimal overhead, R programmers and analysts can use plumber to create REST APIs that expose their work to any number of internal and external systems. This solution provides real-time access to processes and services created entirely in R, and can effectively eliminate the need to perform batch operations or technical hand-offs in order to move R code into production.

This post will focus on a brief introduction to RESTful APIs, then an introduction to the plumber package and how it can be used to expose R services as API endpoints. In subsequent posts, we’ll build a functioning web API using plumber that integrates with Slack and provides real-time customer status reports.

Web APIs

For some, APIs (Application Programming Interface) are things heard of but seldom seen. However, whether seen or unseen, APIs are part of everyday digital life. In fact, you’ve likely used a web API from within R, even if you didn’t recognize it at the time! Several R packages are simply wrappers around popular web APIs, such as tidycensus and gh. Web APIs are a common framework for sharing information across a network, most commonly through HTTP.

HTTP

To understand how HTTP requests work, it’s helpful to know the players involved. A client makes a request to a server, which interprets the request and provides a response. An HTTP request can be thought of simply as a packet of information sent to the server, which the server attempts to interpret and respond to. Every time you visit a URL in a web browser, an HTTP request is made and the response is rendered by the browser as the website you see. It is possible to inspect this interaction using the development tools in a browser.

As seen above, this request is composed of a URL and a request method, which in the case of a web browser accessing a website, is GET.

Request

There are several components of an HTTP request, but here we’ll mention on only a few.

URL: the address or endpoint for the request
Verb / method: a specific method invoked on the endpoint (GET, POST, DELETE, PUT)
Headers: additional data sent to the server, such as who is making the request and what type of response is expected
Body: data sent to the server outside of the headers, common for POST and PUT requests

In the browser example above, a GET request was made by the web browser to www.rstudio.com.

Response

The API response mirrors the request to some extent. It includes headers that contain information about the response and a body that contains any data returned by the API. The headers include the HTTP status code that informs the client how the request was received, along with details about the content that’s being delivered. In the example of a web browser accessing www.rstudio.com, we can see below that the response headers include the status code (200) along with details about the response content, including the fact that the content returned is HTML. This HTML content is what the browser renders into a webpage.

httr

The httr package provides a nice framework for working with HTTP requests in R. The following basic example demonstrates some of what we’ve already learned by using httr and httpbin.org, which provides a playground of sorts for HTTP requests.

library(httr)
# A simple GET request
response <- GET("http://httpbin.org/get")
response

## Response [http://httpbin.org/get]
##   Date: 2018-07-23 14:57
##   Status: 200
##   Content-Type: application/json
##   Size: 266 B
## {"args":{},"headers":{"Accept":"application/json, text/xml, application/...

In this example we’ve made a GET request to httpbin.org/get and received a response. We know our request was successful because we see that the status is 200. We also see that the response contains data in JSON format. The Getting started with httr page provides additional examples of working with HTTP requests and responses.

REST

Representational State Transfer (REST) is an architectural style for APIs that includes specific constraints for building APIs to ensure that they are consistent, performant, and scalable. In order to be considered truly RESTful, an API must meet each of the following six constraints:

Uniform interface: clearly defined interface between client and server
Stateless: state is managed via the requests themselves, not through reliance on an external service
Cacheable: responses should be cacheable in order to improve scalability
Client-Server: clear separation of client and server, each with it’s on distinct responsibilities in the exchange
Layered System: there may be intermediaries between the client and the server, but the client should be unaware of them
Code on Demand: the response can include logic executable by the client

We could spend a lot of time diving further into each of these specifications, but that is beyond the scope of this post. More detail about REST can be found here.

Plumber

Creating RESTful APIs using R is straightforward using the plumber package. Even if you have never written an API, plumber makes it easy to turn existing R functions into API endpoints. Developing plumber endpoints is simply a matter of providing specialized R comments before R functions. plumber recognizes both #' and #* comments, although the latter is recommended in order to avoid potential conflicts with roxygen2. The following defines a plumber endpoint that simply returns the data provided in the request query string.

library(plumber)

#* @apiTitle Simple API

#* Echo provided text
#* @param text The text to be echoed in the response
#* @get /echo
function(text = "") {
  list(
    message_echo = paste("The text is:", text)
  )
}

Here we’ve defined a simple function that takes a parameter, text, and returns it with some additional comments as part of a list. By default, plumber will serialize the object returned from a function into JSON using the jsonlite package. We’ve provided specialized comments to inform plumber that this endpoint is available at api-url/echo and will respond to GET requests.

There are a few ways this plumber script can be run locally. First, assuming the file is saved as plumber.R, the following code would start a local web server hosting the API.

plumber::plumb("plumber.R")$run(port = 5762)

Once the web server has started, the API can be interacted with using any set of HTTP tools. We could even interact with it using httr as demonstrated earlier, although we would need to open a separate R session to do so since the current R session is busy serving the API.

The other method for running the API requires a recent preview build of the RStudio IDE. Recent preview builds include features that make it easier to work with plumber. When editing a plumber script in a recent version of the IDE, a “Run API” icon will appear in the top right hand corner of the source editor. Clicking this button will automatically run a line of code similar to the one we ran above to start a web server hosting the API. A swagger-generated UI will be rendered in the Viewer pane, and the API can be interacted with directly from within this UI.

Now that we have a running plumber API, we can query it using curl from the command line to investigate it’s behavior.

$ curl "localhost:5762/echo" | jq '.'
{
  "message_echo": [
    "The text is: "
  ]
}

In this case, we queried the API without providing any additional data or parameters. As a result, the text parameter is the default empty string, as seen in the response. In order to pass a value to our underlying function, we can define a query string in the request as follows:

$ curl "localhost:5762/echo?text=Hi%20there" | jq '.'
{
  "message_echo": [
    "The text is: Hi there"
  ]
}

In this case, the text parameter is defined as part of the query string, which is appended to the end of the URL. Additional parameters could be defined by separating each key-value pair with &. It’s also possible to pass the parameter as part of the request body. However, to leverage this method of data delivery, we need to update our API definition so that the /echo endpoint also accepts POST requests. We’ll also update our API to consider multiple parameters, and return the parsed parameters along with the entire request body.

library(plumber)

#* @apiTitle Simple API

#* Echo provided text
#* @param text The text to be echoed in the response
#* @param number A number to be echoed in the response
#* @get /echo
#* @post /echo
function(req, text = "", number = 0) {
  list(
    message_echo = paste("The text is:", text),
    number_echo = paste("The number is:", number),
    raw_body = req$postBody
  )
}

With this new API definition, the following curl request can be made to pass parameters to the API via the request body.

$ curl --data "text=Hi%20there&number=42&other_param=something%20else" "localhost:5762/echo" | jq '.'
{
  "message_echo": [
    "The text is: Hi there"
  ],
  "number_echo": [
    "The number is: 42"
  ],
  "raw_body": [
    "text=Hi%20there&number=42&other_param=something%20else"
  ]
}

Notice that we passed more than just text and number in the request body. plumber parses the request body and matches any arguments found in the R function definition. Additional arguments, like other_param in this case, are ignored. plumber can parse the request body if it is URL-encoded or JSON. The following example shows the same request, but with the request body encoded as JSON.

$ curl --data '{"text":"Hi there", "number":"42", "other_param":"something else"}' "localhost:5762/echo" | jq '.'
{
  "message_echo": [
    "The text is: Hi there"
  ],
  "number_echo": [
    "The number is: 42"
  ],
  "raw_body": [
    "{\"text\":\"Hi there\", \"number\":\"42\", \"other_param\":\"something else\"}"
  ]
}

While these examples are fairly simple, they demonstrate the extraordinary facility of plumber. Thanks to plumber, it is now a fairly straightforward process to expose R functions so they can be consumed and leveraged by any number of systems and processes. We’ve only scratched the surface of its capabilities and, as mentioned, future posts will walk through the creation of a Slack app using plumber. Comprehensive documentation for plumber can be found here.

Deploying

Up until now, we’ve just been interacting with our APIs in our local development environment. That’s great for development and testing, but when it comes time to expose an API to external services, we don’t want our laptop held responsible (at least, I don’t!). There are several deployment methods for plumber outlined in the documentation. The most straightforward method of deployment is to use RStudio Connect. When editing a plumber script in recent versions of the RStudio IDE, a blue publish button will appear in the top right-hand corner of the source editor. Clicking this button brings up a menu that enables the user to publish the API to an instance of RStudio Connect. Once published, API access and performance can be configured through RStudio Connect and the API can be leveraged by external systems and processes.

Conclusion

Web APIs are a powerful mechanism for providing systematic access to computational processes. Writing APIs with plumber makes it easy for others to take advantage of the work you’ve created in R without the need to rely on batch processing or code rewriting. plumber is exceptionally flexible and can be used to define a wide variety of endpoints. These endpoints can be used to integrate R with other systems. As an added bonus, downstream consumers of these APIs require no knowledge of R. They only need to know how to properly interact with the API via HTTP. plumber provides a convenient and reliable bridge between R and other systems and/or languages used within an organization.