R Markdown on R Views

An R Community Public Library

Thu, 04 Nov 2021 00:00:00 +0000

If you haven’t recently visited bookdown.org, RStudio’s free site for publishing books written with the bookdown R package, you many be amazed at what is available. Currently, there are over one hundred fifty titles listed under the Books tab. These are written in a panoply of languages including Bulgarian, Chinese, English, French, German, Hindi, Italian, Japanese, Korean, Lithuanian (maybe), Norwegian, Portuguese, Russian, Slovenian (I think) and Vietnamese. The breadth of topics is extraordinary! Most books are concerned with R programming or statistical analyses, but you can find the odd volume of Russian poetry or treatise on Capital. Even with the aid of the bubble chart of tags there are hours of browsing here.

The sophistication of the content ranges from student notes to texts that have been published in hardcover. The latter include:

Happy Reading!! And, please let us know if you would like review one of these books.

Outlier Days with R and Python

Mon, 16 Mar 2020 00:00:00 +0000

Welcome to another installment of Reproducible Finance. Today’s post will be topical as we look at the historical behavior of the stock market after days of extreme returns and it will also explore one of my favorite coding themes of 2020 - the power of RMarkdown as an R/Python collaboration tool.

This post originated when Rishi Singh, the founder of tiingo and one of the nicest people I have encountered in this crazy world, sent over a note about recent market volatility along with some Python code for analyzing that volatility. We thought it would be a nice project to post that Python code along with the equivalent R code for reproducing the same results. For me, it’s a great opportunity to use RMarkdown's R and Python interoperability superpowers, fueled by the reticulate package. If you are an R coder and someone sends you Python code as part of a project, RMarkdown + reticulate makes it quite smooth to incorporate that Python code into your work. It was interesting to learn how a very experienced Python coder might tackle a problem and then think about how to tackle that problem with R. Unsurprisingly, I couldn’t resist adding a few elements of data visualization.

Before we get started, if you’re unfamiliar with using R and Python chunks throughout an RMarkdown file, have a quick look at the reticulate documentation here.

Let’s get to it. Since we’ll be working with R and Python, we start with our usual R setup code chunk to load R packages, but we’ll also load the reticulate package and source a Python script. Here’s what that looks like.

library(tidyverse)
library(tidyquant)
library(riingo)
library(timetk)
library(plotly)
library(roll)
library(slider)
library(reticulate)

riingo_set_token("your tiingo token here")

# Python file that holds my tiingo token
reticulate::source_python("credentials.py")

knitr::opts_chunk$set(message = FALSE, warning = FALSE, comment = NA)

Note that I set my tiingo token twice: first using riingo_set_token() so I can use the riingo package in R chunks and then by sourcing the credentials.py file, where I have put tiingoToken = 'my token'. Now I can use the tiingoToken variable in my Python chunks. This is necessary because we will use both R and Python to pull in data from Tiingo.

Next we will use a Python chunk to load the necessary Python libraries. If you haven’t installed these yet, you can open the RStudio terminal and run pip install. Since we’ll be interspersing R and Python code chunks throughout, I will add a # Python Chunk to each Python chunk and, um, # R Chunk to each R chunk.

# Python chunk
import pandas as pd
import numpy as np
import tiingo

Let’s get to the substance. The goal today is look back at the last 43 years of S&P 500 price history and analyze how the market has performed following a day that sees an extreme return. We will also take care with how we define an extreme return, using rolling volatility to normalize percentage moves.

We will use the mutual fund VFINX as a tradeable proxy for the S&P 500 because it has a much longer history than other funds like SPY or VOO.

Let’s start by passing a URL string from tiingo to the pandas function read_csv, along with our tiingoToken.

# Python chunk 
pricesDF = pd.read_csv("https://api.tiingo.com/tiingo/daily/vfinx/prices?startDate=1976-1-1&format=csv&token=" + tiingoToken)

We just created a Python object called pricesDF. We can look at that object in an R chunk by calling py$pricesDF.

# R chunk
py$pricesDF %>% 
  head()

We just created a Python object called pricesDF. Let’s reformat the date column becomes the index, in date time format.

# Python chunk 
pricesDF = pricesDF.set_index(['date'])
pricesDF.index = pd.DatetimeIndex(pricesDF.index)

Heading back to R for viewing, we see that the date column is no longer a column - it is the index of the data frame and in pandas the index is more like a label than a new column. In fact, here’s what happens when call the row names of this data frame.

# R chunk
py$pricesDF %>% 
  head() %>% 
  rownames()

We now have our prices, indexed by date. Let’s convert adjusted closing prices to log returns and save the results in a new column called returns. Note the use of the shift(1) operator here. That is analogous to the lag(..., 1) function in dplyr.

# Python chunk
pricesDF['returns'] = np.log(pricesDF['adjClose']/pricesDF['adjClose'].shift(1))

Next, we want to calculate the 3-month rolling standard deviation of these daily log returns, and then divide daily returns by the previous rolling 3-month volatility in order to prevent look-ahead error. We can think of this as normalizing today’s return by the previous 3-months’ rolling volatility and will label it as stdDevMove.

# Python chunk
pricesDF['rollingVol'] = pricesDF['returns'].rolling(63).std()
pricesDF['stdDevMove'] = pricesDF['returns'] / pricesDF['rollingVol'].shift(1)

Finally, we eventually want to calculate how the market has performed on the day following a large negative move. To prepare for that, let’s create a column of next day returns using shift(-1).

# Python chunk
pricesDF['nextDayReturns'] = pricesDF.returns.shift(-1)

Now, we can filter by the size of the stdDevMove column and the returns column, to isolate days where the standard deviation move was at least 3 and the returns was less than -3%. We use mean() to find the mean next day return following such large events.

# Python chunk
nextDayPerformanceSeries = pricesDF.loc[(pricesDF['stdDevMove'] < -3) & (pricesDF['returns'] < -.03), ['nextDayReturns']].mean()

Finally, let’s loop through and see how the mean next day return changes as we filter on different extreme negative returns or we can call drop tolerances. We will label the drop tolerance as i, set it at -.03 and then run a while loop that decrements down i by .0025 at each pass. In this way we can look at the mean next return following different levels of negative returns.

# Python chunk
i = -.03
while i >= -.0525:
    nextDayPerformanceSeries = pricesDF.loc[(pricesDF['stdDevMove'] < -3) & (pricesDF['returns'] < i), ['nextDayReturns']]
    print(str(round(i, 5)) + ': ' + str(round(nextDayPerformanceSeries['nextDayReturns'].mean(), 6)))

    i -= .0025

It appears that as the size of the drop gets larger and more negative, the mean bounce back tends to get larger.

Let’s reproduce these results in R.

First, we import prices using the riingo_prices() function from the riingo.

# R chunk
sp_500_prices <- 
  "VFINX" %>% 
  riingo_prices(start_date = "1976-01-01", end_date = today())

We can use mutate() to add a column of daily returns, rolling volatility, standard deviation move and next day returns.

# R chunk
sp_500_returns <- 
  sp_500_prices %>%
  select(date, adjClose) %>% 
  mutate(daily_returns_log = log(adjClose/lag(adjClose)),
         rolling_vol = roll_sd(as.matrix(daily_returns_log), 63),
         sd_move = daily_returns_log/lag(rolling_vol),
         next_day_returns = lead(daily_returns_log))

Now let’s filter() on an sd_move greater than 3 and daily_returns_log less than a drop tolerance of -.03.

# R chunk

sp_500_returns %>% 
  na.omit() %>% 
  filter(sd_move < -3 & daily_returns_log < -.03) %>% 
  select(date, daily_returns_log, sd_move, next_day_returns) %>% 
  summarise(mean_return = mean(next_day_returns)) %>% 
  add_column(drop_tolerance = scales::percent(.03), .before = 1)

# A tibble: 1 x 2
  drop_tolerance mean_return
  <chr>                <dbl>
1 3%                 0.00625

We used a while() loop to iterate across different drop tolerances in Python, let’s see how to implement that using map_dfr() from the purrr package.

First, we will define a sequence of drop tolerances using the seq() function.

# R chunk
drop_tolerance <- seq(.03, .05, .0025)

drop_tolerance

[1] 0.0300 0.0325 0.0350 0.0375 0.0400 0.0425 0.0450 0.0475 0.0500

Next, we will create a function called outlier_mov_fun that takes a data frame of returns, filters on a drop tolerance and gives us the mean return following large negative moves.

# R chunk
outlier_mov_fun <- function(drop_tolerance, returns) {
returns %>%  
  na.omit() %>% 
  filter(sd_move < -3 & daily_returns_log < -drop_tolerance) %>% 
  select(date, daily_returns_log, sd_move, next_day_returns) %>% 
  summarise(mean_return = mean(next_day_returns) %>% round(6)) %>%
  add_column(drop_tolerance = scales::percent(drop_tolerance), .before = 1) %>% 
  add_column(drop_tolerance_raw = drop_tolerance, .before = 1)
}

Notice how that function takes two arguments: a drop tolerance and data frame of returns.

Next, we pass our sequence of drop tolerances, stored in a variable called drop_tolerance to map_dfr(), along with our function and our sp_500_returns object. map_dfr will iterate through our sequence of drops and apply our function to each one.

# R chunk
map_dfr(drop_tolerance, outlier_mov_fun, sp_500_returns) %>% 
  select(-drop_tolerance_raw)

# A tibble: 9 x 2
  drop_tolerance mean_return
  <chr>                <dbl>
1 3%                 0.00625
2 3%                 0.00700
3 3%                 0.00967
4 4%                 0.0109 
5 4%                 0.0122 
6 4%                 0.0132 
7 4%                 0.0149 
8 5%                 0.0149 
9 5%                 0.0162

Have a quick glance up that the results of our Python while() and we should see that the results are consistent.

Alright, let’s have some fun and get to visualizing these results with ggplot and plotly.

# R chunk
(
sp_500_returns %>% 
map_dfr(drop_tolerance, outlier_mov_fun, .) %>% 
  ggplot(aes(x = drop_tolerance_raw, y = mean_return, text = str_glue("drop tolerance: {drop_tolerance}
                                                                      mean next day return: {mean_return * 100}%"))) +
  geom_point(color = "cornflowerblue") +
  labs(title = "Mean Return after Large Daily Drop", y = "mean return", x = "daily drop") +
  scale_x_continuous(labels = scales::percent) +
  scale_y_continuous(labels = scales::percent) + 
  theme_minimal()
) %>% ggplotly(tooltip = "text")

Here’s what happens when we expand the upper bound to a drop tolerance of -2% and make our intervals smaller, moving from .25% increments to .125% increments.

# R chunk
drop_tolerance_2 <- seq(.02, .05, .00125)

(
sp_500_returns %>% 
map_dfr(drop_tolerance_2, outlier_mov_fun, .) %>% 
  ggplot(aes(x = drop_tolerance_raw, y = mean_return, text = str_glue("drop tolerance: {drop_tolerance}
                                                                      mean next day return: {mean_return * 100}%"))) +
  geom_point(color = "cornflowerblue") +
  labs(title = "Mean Return after Large Daily Drop", y = "mean return", x = "daily drop") +
  scale_x_continuous(labels = scales::percent) +
  scale_y_continuous(labels = scales::percent) + 
  theme_minimal()
) %>% ggplotly(tooltip = "text")

Check out what happens when we expand the lower bound, to a -6% drop tolerance.

# R chunk
drop_tolerance_3 <- seq(.02, .06, .00125)

(
sp_500_returns %>% 
map_dfr(drop_tolerance_3, outlier_mov_fun, .) %>% 
  ggplot(aes(x = drop_tolerance_raw, y = mean_return, text = str_glue("drop tolerance: {drop_tolerance}
                                                                      mean next day return: {mean_return * 100}%"))) +
  geom_point(color = "cornflowerblue") +
  labs(title = "Mean Return after Large Daily Drop", y = "mean return", x = "daily drop") +
  scale_x_continuous(labels = scales::percent) +
  scale_y_continuous(labels = scales::percent) + 
  theme_minimal()
) %>% ggplotly(tooltip = "text")

I did not expect that gap upward when the daily drop passes 5.25%.

A quick addendum that if I had gotten my act together and finished this 4 days ago I would not have included, but I’m curious how this last week has compared with other weeks in terms of volatility. I have in mind to visualize weekly return dispersion and that seemed a mighty tall task, until the brand new slider package came to the rescue! slider has a function called slide_period() that, among other things, allows us to break up time series according to different periodicities.

To break up our returns by week, we call slide_period_dfr(., .$date, "week", ~ .x, .origin = first_monday_december, .names_to = "week"), where first_monday_december is a date that falls on a Monday. We could use our eyeballs to check a calendar and find a date that’s a Monday or we could use some good ol’ code. Let’s assume we want to find the first Monday in December of 2016.

We first filter our data with filter(between(date, as_date("2016-12-01"), as_date("2016-12-31"))). Then create a column of weekday names with wday(date, label = TRUE, abbr = FALSE) and filter to our first value of “Monday”.

# R Chunk
first_monday_december <- 
  sp_500_returns %>%
  mutate(date = ymd(date)) %>% 
  filter(between(date, as_date("2016-12-01"), as_date("2016-12-31"))) %>% 
  mutate(day_week = wday(date, label = TRUE, abbr = FALSE)) %>% 
  filter(day_week == "Monday") %>% 
  slice(1) %>% 
  pull(date)

Now we run our slide_period_dfr() code and it will start on the first Monday in December of 2016, and break our returns into weeks. Since we set .names_to = "week", the function will create a new column called week and give a unique number to each of our weeks.

# R chunk

sp_500_returns %>%
  select(date, daily_returns_log) %>%
  filter(date >= first_monday_december) %>%
  slide_period_dfr(.,
                   .$date,
                   "week",
                   ~ .x,
                   .origin = first_monday_december,
                   .names_to = "week") %>% 
  head(10)

# A tibble: 10 x 3
    week date                daily_returns_log
   <int> <dttm>                          <dbl>
 1     1 2016-12-05 00:00:00           0.00589
 2     1 2016-12-06 00:00:00           0.00342
 3     1 2016-12-07 00:00:00           0.0133 
 4     1 2016-12-08 00:00:00           0.00226
 5     1 2016-12-09 00:00:00           0.00589
 6     2 2016-12-12 00:00:00          -0.00105
 7     2 2016-12-13 00:00:00           0.00667
 8     2 2016-12-14 00:00:00          -0.00810
 9     2 2016-12-15 00:00:00           0.00392
10     2 2016-12-16 00:00:00          -0.00172

From here, we can group_by that week column and treat each week as a discrete time period. Let’s use ggplotly to plot each week on the x-axis and the daily returns of each week on the y-axis, so that the vertical dispersion shows us the dispersion of weekly returns. Hover on the point to see the exact date of the return.

# R chunk
(
sp_500_returns %>%
  select(date, daily_returns_log) %>%
  filter(date >= first_monday_december) %>%
  slide_period_dfr(.,
                   .$date,
                   "week",
                   ~ .x,
                   .origin = first_monday_december,
                   .names_to = "week") %>%
  group_by(week) %>%
  mutate(start_week = ymd(min(date))) %>%
  ggplot(aes(x = start_week, y = daily_returns_log, text = str_glue("date: {date}"))) +
  geom_point(color = "cornflowerblue", alpha = .5) +
  scale_y_continuous(labels = scales::percent,
                     breaks = scales::pretty_breaks(n = 8)) +
  scale_x_date(breaks = scales::pretty_breaks(n = 10)) +
  labs(y = "", x = "", title = "Weekly Daily Returns") +
  theme_minimal()
) %>% ggplotly(tooltip = "text")

We can also plot the standard deviation of returns for each week.

# R chunk

(
sp_500_returns %>%
  select(date, daily_returns_log) %>%
  filter(date >= first_monday_december) %>%
  slide_period_dfr(.,
                   .$date,
                   "week",
                   ~ .x,
                   .origin = first_monday_december,
                   .names_to = "week") %>%
  group_by(week) %>%
  summarise(first_of_week = first(date),
            sd = sd(daily_returns_log)) %>%
  ggplot(aes(x = first_of_week, y = sd, text = str_glue("week: {first_of_week}"))) +
  geom_point(aes(color = sd)) +
  labs(x = "", title = "Weekly Standard Dev of Returns", y = "") +
  theme_minimal()
) %>% ggplotly(tooltip = "text")

That’s all for today! Thanks for reading and stay safe out there.

Productionizing Shiny and Plumber with Pins

Thu, 17 Oct 2019 00:00:00 +0000

Producing an API that serves model results or a Shiny app that displays the results of an analysis requires a collection of intermediate datasets and model objects, all of which need to be saved. Depending on the project, they might need to be reused in another project later, shared with a colleague, used to shortcut computationally intensive steps, or safely stored for QA and auditing.

Some of these should be saved in a data warehouse, data lake, or database, but write access to an appropriate database isn’t always available. In other cases, especially with models, it may not be clear where they should be saved at all.

Enter pins, a new R package written by Javier Luraschi. pins makes it easy to save (pin) R objects including datasets, models, and plots to a central location (board), and access them easily from both R and Python. Pins make it much easier to create production-ready R assets by simplifying the storage and updating of intermediate data artifacts.

Problems you can put a pin in

In general, pins are a good substitute for saving objects alongside analysis code as .csv or .rds objects. Especially when the object is reused several times or updated independently from the rest of the analysis, a pin is probably a better solution than saving a file with your code.

In this article, I’ll create a predictive model, programmatically serve predictions via a Plumber API, and visualize those predictions in a Shiny app. Along the way, I’ll make extensive use of pins for important parts of my workflow.

The model will predict future availability of bicycles at Capital Bikeshare docks, which provide short-term bicycle rentals in and around Washington DC. Capital Bikeshare makes data on the current availability of bikes at each station available via a public API.

I’m going to make model predictions available in production by providing programmatic access to the model via an API and to humans via a Shiny app. All of the code for this demo is available on Github.

To get there, I’m going to follow this analysis workflow:

Ingest metadata about the stations, like name and location, from the bike data API.
Combine the station metadata with raw data on bike availability from the data lake to create an analysis dataset.
Train and deploy a model of future bike availability.
Serve model predictions via a Plumber API.
Visualize model predictions via a Shiny app.

Along the way, here are three specific times that a pin is going to come in handy:

Maintaining the metadata table of station IDs and details. Especially since I’m reusing this table in multiple assets in this project, having it in a pin is a sure way to know it’s up-to-date.
Saving the final analysis dataset. In this case, the raw Capitol Bikeshare data is being imported with a completely separate ETL script, and I don’t want to write my analysis dataset into a data lake. Without a separate database for analysis data, a pin is my best option.
Deploying the model to serve the predictions. Saving the model separately from the API makes it easy to decouple API and model versions and to retrain the model and redeploy seamlessly when needed.

In all of these cases, pins drastically simplify my workflow, improve discoverability of the objects my analysis has created, and makes me more confident that I’m always using the newest version.

Where to pin?

Before getting started describing exactly how this analysis project works, let’s dive a little deeper into the pins package itself.

Pins live on boards. A board is a set of content names and the associated files. The magic of the pins package is that with only two commands and the name of some content, you can upload and download your R objects without having to worry about how how the content is stored.

By default, there are two boards you can use immediately: the packages board of the datasets from R packages that are installed, and the local board, which caches datasets for quick loading later.

The real power of pins is unlocked with remote boards. pins supports Kaggle, Github, website, and RStudio Connect boards, and also supports building custom extensions. By using a remote board, you can use pins to make your R objects accessible to others on your team in a central location.

How it works

Using a pin works like this:

Register the board with the the pins::board_register function. You’ll need to provide the proper authentication mechanism like a Kaggle token, Github Personal Access Token (PAT), or RStudio Connect API key if you are using a remote board.

For GitHub, you need a repo that you have write access to, as well as a token:

pins::board_register(board = "github", 
                     repo = "akgold/pins_demo", 
                     branch = "master",
                     token = Sys.getenv("GITHUB_PAT"))

For an RStudio Connect board, you need the server URL and an API key:

pins::board_register(board = "rsconnect", 
                     server = "https://colorado.rstudio.com/rsc", 
                     key = Sys.getenv("RSTUDIOCONNECT_API_KEY"))

At that point, your connections pane in RStudio will show the content available in the board.

Once you’ve registered the board, your interactions are exactly the same no matter which board type you’re using.

Pin an object to the board.

pins::pin(
  x = mtcars, 
  name = "mtcars_pin", 
  description = "A pin of the mtcars dataset.", 
  board = "rsconnect"
)

Download the object later.

cars_data <- pins::pin_get(
  name = "mtcars_pin"
  board = "rsconnect"
)

Production Apps with Pins

In order to create, serve, and visualize my bike-availability predictions, I’m going to use RStudio’s publishing and scheduling platform, RStudio Connect.

As of RStudio Connect 1.7.8, you can publish pins to RStudio Connect, and pins of datasets provide a nice preview of the pin, as well as code to retrieve the pin in both R and Python.

The advantage of using RStudio Connect is that I can deploy R Markdown documents, Shiny apps, and Plumber APIs that create, use, and update the pins in addition to storing the pins themselves. I can also use the permissions and security of RStudio Connect to make sure that my pins are viewable only by those with the proper permissions.

Here’s how the process works:

4.

5.

The three times that a pin was useful here turn out to represent three of the most compelling reasons to use a pin.

A small dataset that gets reused. By accessing the station metadata dataset in a pin, I know I’m always getting the latest version regardless of which asset is using it, and it’s also accessible for other analyses in the future.

An analysis dataset when you can’t write back to the database. In this case, I don’t want to write an analysis dataset back to the raw data lake, so it’s easier to store it as a pin.

A model in production. By using a pin to store my model, it’s easy to update the version that’s in production by running the R Markdown document that trains the model. It’s also conceptually simple to update the model independently from the API that serves predictions or the Shiny app that visualizes the predictions.

Pins can be a fantastic way to enable Shiny and Plumber in production. By giving data scientists a place to save and deploy the output of their projects, pins make it easier to create, deploy, and update models, datasets, and other production-ready R objects.

How to share R visualizations in Microsoft PowerPoint

Thu, 04 Apr 2019 00:00:00 +0000

Hadrien Dykiel is an RStudio Customer Success Engineer

Microsoft PowerPoint is often the de facto choice for creating presentation slides, especially at larger companies. In many organizations, it comes pre-installed on workstations and pretty much everybody knows how to use it. This can make it an effective medium for sharing information, since most folks are comfortable with it. Unfortunately, valuable time is often lost manually creating slides. R developers often find themselves copying and pasting their results into presentation decks. Moreover, results may change or over time, requiring analysts and data scientists to manually update their slides with the latest results. So, in addition to being a time-consuming task, copying and pasting also introduces a big reproducibility problem. R can help solve these problems by programmatically exporting your results to PowerPoint for you.

Let’s say you are collaborating on a project in which members of your team will use other tools like SAS and Excel to perform their analyses. At the end of the day, you plan to combine all of your work together into a single presentation deck that will be shared with various business stakeholders. You boot up RStudio and open an R Markdown file and produce a correlation plot for the presentation.

Rather than manually copying and pasting your corrplot into the final PowerPoint deck, you can update the output document type in your document’s YAML header to powerpoint_presentation. Optionally, you may also want to customize the appearance of your slides by passing a custom reference document via the reference_doc option. This is a nice option to use if you want your slides to match your company’s color schemes, for example. The snippet below shows what the code for a typical rmarkdown file with the output format set to PowerPoint might look like. Like all .Rmd files, it contains three elements: a YAML header that contains the metadata for your RMD file, narrative in simple markdown syntax, and code.

As soon as you hit the Knit button (or use the keyboard shortcut Cmd/Ctrl + Shift + K), RStudio initiates the knitting process. The rmarkdown package transforms your R script into markdown, and the pandoc package converts it to the PowerPoint output format, as specified in your YAML header. This process happens all underneath the hood, so as a user, the only thing you see is the final PowerPoint output file, which automatically opens as soon as your document finishes knitting. Because you created your PowerPoint slides programmatically, you can easily update them in the future, such as if new data becomes available and you wish to refresh your results.

The rmarkdown package offers a fair amount of flexibility for customizing your PowerPoint slides, such as having the ability to include R code, images, R visualizations, speaker notes, and customized column layout.

To learn more about creating PowerPoint presentations with R, Yihui’s RMD: The Definitive Guide and RStudio’s article Rendering Powerpoint Presentations with RStudio are both great resources.

Enterprise Dashboards with R Markdown

Wed, 16 May 2018 00:00:00 +0000

This is a second post in a series on enterprise dashboards. See our previous post, Enterprise-ready dashboards with Shiny Databases.

We have been living with spreadsheets for so long that most office workers think it is obvious that spreadsheets generated with programs like Microsoft Excel make it easy to understand data and communicate insights. Everyone in a business, from the newest intern to the CEO, has had some experience with spreadsheets. But using Excel as the de facto analytic standard is problematic. Relying exclusively on Excel produces environments where it is almost impossible to organize and maintain efficient operational workflows. In addition to fostering low productivity, organizations risk profits and reputations in an age where insightful analyses and process control translate to a competitive advantage. Most organizations want better control over accessing, distributing, and processing data. You can use the R programming language, along with with R Markdown reports and RStudio Connect, to build enterprise dashboards that are robust, secure, and manageable.

This Excel dashboard attempts to function as a real application by allowing its users to filter and visualize key metrics about customers. It took dozens of hours to build. The intent was to hand off maintenance to someone else, but the dashboard was so complex that the author was forced to maintain it. Every week, the author copied data from an ETL tool and pasted it into the workbook, spot checked a few cells, and then emailed the entire workbook to a distribution list. Everyone on the distribution list got a new copy in their inbox every week. There were no security controls around data management or data access. Anyone with the report could modify its contents. The update process often broke the brittle cell dependencies; or worse, discrepancies between weeks passed unnoticed. It was almost impossible to guarantee the integrity of each weekly report.

Why coding is important

Excel workbooks are hard to maintain, collaborate on, and debug because they are not reproducible. The content of every cell and the design of every chart is set without ever recording the author’s actions. There is no simple way to recreate an Excel workbook because there is no recipe (i.e., set of instructions) that describes how it was made. Because Excel workbooks lack a recipe, they tend to be hard to maintain and prone to errors. It takes care, vigilance, and subject-matter knowledge to maintain a complex Excel workbook. Even then, human errors abound and changes require a lot of effort.

A better approach is to write code. There are many reasons to start programming. When you create a recipe with code, anyone can reproduce your work (including your future self). The act of coding implicitly invites others to collaborate with you. You can systematically validate and debug your code. All of these things lead to better code over time. Coding in R has particular advantages given its vast ecosystem of packages, its vibrant community, and its powerful tool chain.

Using R Markdown

There are many tools for replacing complex Excel dashboards with R code. One of these tools is R Markdown, an open-source R package that turns your analyses into high quality documents, reports, presentations and dashboards. R Markdown documents are fully reproducible and support dozens of output formats including HTML, PDF, and Microsoft Word documents.

Here is the same Excel dashboard translated to an R Markdown report. Because this report is written in code, it is vastly simpler and easier to maintain. Like the Excel dashboard above, this R Markdown report is designed to take user inputs so that it could render custom report versions.

Many people are already aware that R Markdown reports combine narrative, code, and output in a single document. What is less commonly known is that you can generalize any R Markdown report by declaring parameters in the document header. R Markdown documents with parameters are known as parameterized reports. In the Excel dashboard users can select segment, group, and period. In a parameterized R Markdown document, you would specify these inputs with the following YAML header:

---
title: Customer Tracker Report
output: html_notebook
params:
  seg: 
    label: "Segment:"
    value: Total
    input: select
    choices: [Total, Heavy, Mainstream, Focus1, Focus2, 
              Specialty, Diverse1, Diverse2, Other, New]
  grp: 
    label: "Group:"
    value: Total
    input: select
    choices: [Total, Core, Extra]
  per: 
    label: "Period:"
    value: Week
    input: radio
    choices: [Week, YTD]
---

You can then call the parameters you declare in the YAML header from your R code chunks.

```{r}
params$segment
params$grp
params$per
```

You can render the document with different inputs by selecting knit with parameters in RStudio. This option will open a user interface that allows you to select the parameters you want.

If you want to automate the process of creating custom report versions, you can render these documents programmatically with the rmarkdown::render() function.

rmarkdown::render(
  input = "tracker-report.Rmd", 
  params = list(seg = "Focus1", grp = "Core", per = "Weekly")
)

Publishing to RStudio Connect

Managing access and permissions for an ocean of Excel files is painful. Data in Excel spreads through an organization without controls like a virus spreads through a body without disease prevention. There are better ways to secure the operation, access, and distribution of information.

RStudio Connect is a server product from RStudio that is designed for secure sharing of R content. It is on-premises software you run behind your firewall. You keep control of your data and of who has access. With RStudio Connect, you can see all your content, decide who should be able to view and collaborate on it, tune performance, schedule updates, and view logs. You can schedule your R Markdown reports to run automatically or even distribute the latest version by email.

When you publish a parameterized R Markdown report to RStudio Connect, an interface appears for selecting inputs. Viewers can create new report versions, then email themselves a copy. Collaborators can save and schedule new report versions, then email others a copy. You can even attach output files to these versions. Using parameterized R Markdown documents in RStudio Connect is a powerful way to communicate information.

You can publish content from the RStudio IDE by clicking the Publish button that looks like a blue Eye of Horus. Pressing this button will begin the publishing process. First, it creates a set of instructions for recreating your content. Second, it deploys your content bundle to the server. Third, it recreates your content on RStudio Connect. Push-button publishing has a long history of being used with RStudio. In 2012, RStudio enabled push-button publishing of R Markdown documents to RPubs. In 2014, RStudio enabled push-button publishing of Shiny apps to shinyapps.io. In 2016, RStudio enabled push-button publishing to RStudio Connect.

Adding Shiny

R Markdown documents are rendered with batch processing. That makes them ideal for automation, long running workflows, and custom report versions. However, if you want your documents to be immediately reactive to user input, then you can add a Shiny runtime. These interactive documents behave like a Shiny application in that they must be hosted. You can host interactive documents and Shiny applications with RStudio Connect. Deciding when to choose between R Markdown, interactive documents, and Shiny applications is a subject for a later post.

Summary

Reproducible code in R leads to better analysis and collaboration. You can use parameterized R Markdown reports to create complex, interactive dashboards. Hosting these dashboards securely in RStudio Connect gives you control over accessing, distributing, and processing data. You can use the R programming language, along with with R Markdown reports and RStudio Connect, to build enterprise dashboards that are robust, secure, and manageable.

Click here for source code.

How to Show R Inline Code Blocks in R Markdown

Mon, 04 Dec 2017 00:00:00 +0000

Inline code with R Markdown

R Markdown is a well-known tool for reproducible science in R. In this article, I will focus on a few tricks with R inline code.

Some time ago, I was writing a vignette for my package WordR. I was using R Markdown. At one point I wanted to show `r expression` in the output, exactly as it is shown here, as an inline code block.

In both R Markdown and Markdown, we can write `abc` to show abc. What is not obvious is that you can use double backticks to escape single backticks in the code block. So code like this: `` `abc` `` (mind the spaces!) produces this `abc`.

Now as an exercise, you can guess how I produced the `` `abc` `` block above. Yes, indeed, I have ``` `` `abc` `` ``` in the Rmd source file. And we can go on like this ad infinitum (can we?).

OK, but I wanted to produce `r expression`. Learning the lesson above, we can try `` `r expression` ``. But trying this, I was getting an error:

processing file: codeBlocks.Rmd
Quitting from lines 12-22 (codeBlocks.Rmd) 
Error in vapply(x, format_sci_one, character(1L), ..., USE.NAMES = FALSE) : 
  values must be length 1,
 but FUN(X[[1]]) result is length 0
Calls: <Anonymous> ... paste -> hook -> .inline.hook -> format_sci -> vapply
Execution halted

Obviously, the R Markdown renderer is trying to evaluate the expression. So it seems that R Markdown renderer does not know that it should (should it?) skip R inline code blocks which are enclosed by double backticks.

Solution

Making a long (and yes, I spent some time to find a solution) story short. The correct code block to produce `r expression` is `` `r "\u0060r expression\u0060"` ``.

Short explanation how it works: \\u0060 is an Unicode representation of the backtick (`). So first, the R Markdown renderer finds the R expression within the double backticks and it evaluates it. Important here is the usage of the Unicode for backtick, since using backtick within the expression would result in an error. (We are lucky, that the R Markdown renderer is not running recursively, finding again the R code block and evaluating it again.) So once the R Markdown is done, the Markdown is just seeing `` `r expression` `` in the temporary .md file, and it evaluates it correctly to `r expression` in the HTML output.

If you want to see (much) more, just look at the source R Markdown file for this article here. Do you know a better, more elegant solution? If you do, please use the discussion below.

Epilogue

Some time after I sent the draft of this blog to the RViews admin, I got a reply (thank you!) which pointed to the knitr FAQ page, specifically question number 7 (and a new post from author of knitr package explaining it a little further). It suggests probably more elegant solution of using

Some text before inline code `` `r
expression` `` and some code after

(mind the newline!) that will produce Some text before inline code `r expression` and some text after or use `` `r knitr::inline_expr("expression")` `` which produces similarly `r expression`.

But, I believe this post (especially its source) might still help someone to understand how the R inline code is evaluated.

Using Shiny with Scheduled and Streaming Data

Wed, 15 Nov 2017 00:00:00 +0000

Note: This article is now several years old. If you have RStudio Connect, there are more modern ways of updating data in a Shiny app.

Shiny applications are often backed by fluid, changing data. Data updates can occur at different time scales: from scheduled daily updates to live streaming data and ad-hoc user inputs. This article describes best practices for handling data updates in Shiny, and discusses deployment strategies for automating data updates.

This post builds off of a 2017 rstudio::conf talk. The recording of the original talk and the sample code for this post are available.

The end goal of this example is a dashboard to help skiers in Colorado select a resort to visit. Recommendations are based on:

Snow reports that provide useful metrics like number of runs open and amount of new snow. Snow reports are updated daily.
Weather data, updated in near real-time from a live stream.
User preferences, entered in the dashboard.

The backend for the dashboard looks like:

Automate Scheduled Data Updates

The first challenge is preparing the daily data. In this case, the data preparation requires a series of API requests and then basic data cleansing. The code for this process is written into an R Markdown document, alongside process documentation and a few simple graphs that help validate the new data. The R Markdown document ends by saving the cleansed data into a shared data directory. The entire R Markdown document is scheduled for execution.

It may seem odd at first to use a R Markdown document as the scheduled task. However, our team has found it incredibly useful to be able to look back through historical renderings of the “report” to gut-check the process. Using R Markdown also forces us to properly document the scheduled process.

We use RStudio Connect to easily schedule the document, view past historical renderings, and ultimately to host the application. If the job fails, Connect also sends us an email containing stdout from the render, which helps us stay on top of errors. (Connect can optionally send the successfully rendered report, as well.) However, the same scheduling could be accomplished with a workflow tool or even CRON.

Make sure the data, written to shared storage, is readable by the user running the Shiny application - typically a service account like rstudio-connect or shiny can be set as the run-as user to ensure consistent behavior.

Alternatively, instead of writing results to the file system, prepped data can be saved to a view in a database.

Using Scheduled Data in Shiny

The dashboard needs to look for updates to the underlying shared data and automatically update when the data changes. (It wouldn’t be a very good dashboard if users had to refresh a page to see new data.) In Shiny, this behavior is accomplished with the reactiveFileReader function:

daily_data <- reactiveFileReader(
  intervalMillis = 100,
  filePath       = 'path/to/shared/data',
  readFunc       = readr::read_cs
)

The function checks the shared data file’s update timestamp every intervalMillis to see if the data has changed. If the data has changed, the file is re-read using readFunc. The resulting data object, daily_data, is reactive and can be used in downstream functions like render***.

If the cleansed data is stored in a database instead of written to a file in shared storage, use reactivePoll. reactivePoll is similar to reactiveFileReader, but instead of checking the file’s update timestamp, a second function needs to be supplied that identifies when the database is updated. The function’s help documentation includes an example.

Streaming Data

The second challenge is updating the dashboard with live streaming weather data. One way for Shiny to ingest a stream of data is by turning the stream into “micro-batches”. The invalidateLater function can be used for this purpose:

liveish_data <- reactive({
  invalidateLater(100)
  httr::GET(...)
})

This causes Shiny to poll the streaming API every 100 milliseconds for new data. The results are available in the reactive data object liveish_data. Picking how often to poll for data depends on a few factors:

Does the upstream API enforce rate limits?
How long does a data update take? The application will be blocked while it polls data.

The goal is to pick a polling time that balances the user’s desire for “live” data with these two concerns.

Conclusion

To summarize, this architecture provides a number of benefits: No more painful, manual running of R code every day! Dashboard code is isolated from data prep code. There is enough flexibility to meet user requirements for live and daily data, while preventing un-necessary number crunching on the backend.

Growth of DataFest over the years

Wed, 24 May 2017 00:00:00 +0000

In a previous post, I introduced DataFest and how one can streamline the organization of this event using Google Forms and tools from the tidyverse. In this post, I’ll walk through building a Shiny app that demonstrates the growth of DataFest over the years, both in terms of host locations and participating institutions, as well as in terms of the number of students who participated in each event.

Here is a list of all packages used in this article:

library(tidyverse)
library(googlesheets)
library(devtools)
library(ggmap)
library(stringr)
library(leaflet)

The data were contributed by the event organizers, and were collected using a Google Form.

To begin, the data are read using the googlesheets package.

datafest_wide <- gs_title("DataFest over the years (Responses)") %>%
  gs_read()

Data prep

Then minimal manipulation is applied to column names, and a new column concatenating city, state, and country is added to be used in geocoding.

# rename columns
yrs <- sort(rep(2011:2017, 3))
cols <- c("df_", "num_part_", "other_inst_")

names(datafest_wide) <- c("timestamp", "host", "city", "state", "country", "url",
                     paste0(cols, yrs))

# geocode host location
datafest_wide <- datafest_wide %>%
  mutate(address = paste(city, state, country)) %>% 
  mutate_geocode(address)

Note that we need to use the development version of the ggmap package for mutate_geocode() to play nicely with a tbl_df. You can install this version with install_github("dkahle/ggmap").

Next, we convert the data from wide to long format using functionality from the tidyr package. First, we gather the columns that contain yearly information (for each year, we have an indicator for whether an event was hosted at the location, the number of students that participated, and other participating institutions, if any). Then, we strip the year information from variable names, and instead save it as a variable in the dataset. Finally, we spread the key-value pair across three columns.

datafest_long <- datafest_wide %>% 
  gather(key, value, df_2011:other_inst_2017) %>%
  mutate(year = as.numeric(str_match(key, "[0-9]+"))) %>%
  mutate(key = str_replace(key, "_[0-9]+", "")) %>%
  spread(key, value) %>%
  mutate(num_part = as.numeric(num_part))

Map of 2017 ASA DataFests

The eventual goal of this post is to make a Shiny app that maps DataFest spread and growth over the years; however, I’ll start by making a map for just one year, 2017, to develop the code for the map, and then use this code within a Shiny app.

Going forward, I’ll refer to the long dataset as datafest.

datafest <- datafest_long

First, I take a subset of the data for hosts that held an event in 2017:

datafest_2017 <- filter(datafest, year == 2017 & df == "Yes")

Then, I set a few colors to be used in the plot,

href_color <- "#A7C6C6"
marker_color <- "black"
part_color <- "#89548A"

as well as the bounds of the plot based on the min/max longitude/latitude.

left <- floor(min(datafest$lon))
right <- ceiling(max(datafest$lon))
bottom <- floor(min(datafest$lat))
top <- ceiling(max(datafest$lat))

I will be making the map using the leaflet package, as this package allows for easily overlaying markers and popups to maps. The popups are text bubbles that appear when a point is clicked, and that contain additional information about that data point. This is a good place to add some event-specific information, such as name of host, and link to their event homepage, other participating institutions (if any), and number of participants.

host_text <- paste0(
  "<b><a href='", datafest_2017$url, "' style='color:", 
  href_color, "'>", datafest_2017$host, "</a></b>"
)

other_inst_text <- paste0(
  ifelse(is.na(datafest_2017$other_inst), 
         "", 
         paste0("<br>", "with participation from ", datafest_2017$other_inst))
)

part_text <- paste0(
  "<font color=", part_color,">", datafest_2017$num_part, 
  " participants</font>"
)

popups <- paste0(
  host_text, other_inst_text, "<br>", part_text
)

We’re finally ready to make our map! Note that the radii of the points are proportional to the log of the number of participants (times an arbitrary factor for visual appeal).

leaflet() %>%
  addTiles() %>%
  fitBounds(lng1 = left, lat1 = bottom, lng2 = right, lat2 = top) %>%
  addCircleMarkers(lng = datafest_2017$lon, lat = datafest_2017$lat,
                   radius = log(datafest_2017$num_part) * 1.2, 
                   fillColor = marker_color,
                   color = marker_color,
                   weight = 1,
                   fillOpacity = 0.5,
                   popup = popups)

Shiny app

Next, we build upon our earlier plot to create a Shiny app that has the following three components:

A slider input with animation for values between 2011 and 2017 (DataFest years, so far)
A line plot that shows the increase in the number participants over the year
A map that shows the spread of DataFest geographically over the years

You can find and interact with the app at https://gallery.shinyapps.io/datafest-map-all-years/, and the code for the app, as well as all steps up to this point, can be found at this GitHub repo.

Copper, Gold and Ten-Year Treasury Notes

Wed, 12 Apr 2017 00:00:00 +0000

Today, we will continue getting familiar with data from Quandl, but will also devote more time to expanding our dygraphs toolkit. We will be building up a data visualization in discrete pieces, which isn’t very efficient, but will make things easier when we move this project into production as a Shiny app. From a substantive perspective, we will examine the relationship between the price ratio of copper-gold and 10-year Treasure yields.

Why do we care about the copper-gold price ratio and Treasury yields? First, both Jeff Gundlach (in this webcast) and Adam Robinson say so, and that’s probably good enough. The theory goes like this:

Copper is a useful industrial metal whose price tends to rise when the global economy expands. As firms produce more goods that require copper as an input, the increased demand for copper drives the price higher. Gold, on the other hand, is a somewhat less useful metal whose prices tends to rise when investors are fearful about a contracting global economy. Gold is a safe-haven investment, and a rising gold price signals either a contracting economy, investor fears of a contracting economy, or both. Gold prices tend to fall when the economy is humming along nicely. Thus, the copper-gold price ratio tends to be increasing when the economy is expanding.

The yield on 10-year Treasury Notes also tends to rise during economic expansion because investors’ inflation expectations are on the rise. When investors expect inflation to increase, they anticipate an uptick in interest rates (for those of you who are too young to remember what an interest rate is, take a look at rates in the mid-1980s) and start to seek higher yields today. That can drive down Treasury prices and increase yields.

Thus, we should observe a positive relationship between the copper-gold price ratio and 10-year yields. Both should be rising and falling based on the state of the world economy. There’s nothing too crazy here, but it’s an interesting relationship to think about and investigate. That’s what we’ll do today!

First, let’s import the relevant time series data from Quandl. We will specify type = "xts" in order to create xts objects and collapse = "daily" because we want daily prices. Note in particular our data sources: CME for copper and gold, and FRED for the 10-year yield. But, we just need the Quandl codes and to be careful about consistent start/end dates for each data set. It’s a liberating feeling to know before starting a project that we’ll be able to find whatever data we need in one source.

Let’s get to it.

library(Quandl)
library(dplyr)
library(xts)
library(lubridate)
library(dygraphs)

copper <- Quandl("CHRIS/CME_HG1", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

gold <- Quandl("CHRIS/CME_GC1", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

ten_year <- Quandl("FRED/DGS10", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

Now we want to combine these into one xts object, which normally would be an easy invocation of merge.xts, but there’s a slight wrinkle. After creating one xts object, we know we need to calculate a ratio of copper/gold. That means that the presence of NAs will be a problem (I found this out the old-fashioned way, by running code that threw an error). Let’s take care of that by prepending the na.locf() function to our merge operation. That function will replace all NAs with the previous day’s value. Why might one of our time series have an NA when another doesn’t? Maybe one of the instruments has a trading holiday, or FRED publishes data for days that are general market holidays.

The downside, of course, is that we are adding observed values that are fictional. I can live with that here since we’re creating a visualization to help understand and gain an intuition about the hypothesized copper-gold yield relationship. If we were using this data to create a trading strategy or algorithm, our NA replacement would be unacceptably fuzzy and a more rigorous decision-making process would be needed to synchronize the data sets.

No matter which approach is taken, the most crucial thing is to explain and make easily reproducible whatever process is used for handling NAs or any data cleaning such as this. Your colleagues, future self, clients and research audience can scrutinize, discount, ignore or applaud it accordingly.

# We're going to merge our 3 xts objects into on xts objects. This would normally be
# very simple with merge.xts but we want to eliminat NAs with na.locf().

copper_gold_tenYear_merged <- na.locf(merge.xts(copper$Settle, gold$Settle, ten_year),
                                      formLast = TRUE)

colnames(copper_gold_tenYear_merged ) <- c("Copper", "Gold", "TenYear")

Now we create a new column to store the ratio of copper gold prices.

# Create the ratio of prices. I multiply copper by 100 to synch with the scale used by 
# Gundlach in his presentation.
copper_gold_tenYear_merged$ratio <- (copper_gold_tenYear_merged$Copper*100)/copper_gold_tenYear_merged$Gold

We have an xts object that holds four time series, but I want to chart only the copper-gold price ratio and 10-year yields. This next step is not necessary but to be extra clear, I am going to create a new xts object to hold only those two time series.

tenYear_ratio  <- merge(copper_gold_tenYear_merged$ratio, copper_gold_tenYear_merged$TenYear)

We use dygraphs to chart that one xts object, and call dySeries() for each of the columns to be included.

dygraph(tenYear_ratio) %>%
  dySeries("ratio") %>% 
  dySeries("TenYear")

That chart is completely unhelpful, of course, because our time series have different scales. I also don’t love the choppiness of the blue 10-year chart. Let’s address these two issues by adding a right-hand side scale and a dyRoller().

The dyRoller() will help smooth out our chart because each plotted point will be an average of the number of periods specified with rollPeriod = X. This won’t affect our xts object, where we store the data, it just makes the chart more readable.

Adding the right-hand-side y-axis requires a few more lines of code. First we need to invoke dyAxis() for the left-hand axis, called “y”. Then we invoke dyAxis() for the right-hand axis, called “y2”. We also need to set independentTicks = TRUE so that we can use a unique, independent value scale for the right-hand side. Next, in our dySeries() call for each time series, we assign each one to an axis. Here we assign “ratio” with axis = 'y', so that the copper-gold price ratio will be on the left-hand scale, and we assign “TenYear” with axis = 'y2', so the 10-year yield will be on the right-hand scale. I also like to include a label with LHS and RHS for each time series.

dygraph(tenYear_ratio) %>% 
  # Add the rollPeriod for smoothing.
  dyRoller(rollPeriod = 3) %>% 
  # Create two independent axes.
  dyAxis("y", label = "USD") %>%
  dyAxis("y2", label = "Percent (%)", independentTicks = TRUE) %>%
  # Assign each time series to an axis.
  dySeries("ratio", axis = 'y', label = "Copper/Gold (LHS)") %>% 
  dySeries("TenYear", axis = 'y2', label = "10-Year % Yield (RHS)")

Alright, that chart looks better now (though I encourage you to play with different settings for rollPeriod to compare different smoothness levels) and, incidentally, seems supportive of our hypothesized relationship between copper-gold and yields. There did seem to be a few periods of divergence, which would be a ripe area for further research. But we will move on to calculate and chart the rolling correlation between these two time series.

Recall way back to this post from January where we examined rolling correlations between sector ETFs. We’ll port our work from that project and use rollyapply to calculate the 90-day rolling correlation between copper-gold and yields. We will also rename the column since it will be displayed on the dygraph.

# Calculate the rolling correlation between copper-gold and Treasury yields. 

rolling_cor <- rollapply(tenYear_ratio, 90, 
                                       function(x) cor(x[, 1], x[, 2], use = "pairwise.complete.obs"), 
                                       by.column = FALSE)

names(rolling_cor) <- "Copper/Gold 10-Year Correlation"

Let’s graph that rolling correlation.

dygraph(rolling_cor, main = "Rolling 90-day Correlation Copper-Gold & 10-Year Yield")

It looks like the correlation has been above .4 for the last year and on the increase since the election in November. Peeking back up at the chart of the copper-gold ratio and yields, both those time series have been on the rise since the election as well. Investors seem bullish on the economy and an increase in interest rates.

We can add further context with the minimum, maximum and mean rolling correlations and an event label for the election. Nothing complicated here but it’s a good use of the dyLimit() and dyEvent() functions.

  avg <- round(mean(rolling_cor,  na.rm = T), 2)

  mini <- round(min(rolling_cor,  na.rm = T), 2)
 
  maxi <- round(max(rolling_cor,  na.rm = T), 2)
  
dygraph(rolling_cor, main = "Rolling 90-day correlations Copper-Gold & 10-Year Yield") %>% 
  dyLimit(avg, color = 'purple') %>% 
  dyLimit(mini, color = 'red') %>% 
  dyLimit(maxi, color = 'blue') %>% 
  dyEvent("2016-11-08", "Trump!", labelLoc = "bottom")

That’s all for today. We have examined a hypothesized relationship and found some confirmatory evidence, or at least visual indicators. We have also explored some new tools in the dygraphs world by using dyRoll() and dyAxis(). From a workflow and reproducibility perspective, I like this as a template for importing multiple time series and visualizing the relationship between them in different combinations. When we wrap this into a Shiny app next time, we will see how this can be useful for further exploration. Thanks and see you next time!

Organizing DataFest the tidy way

Wed, 05 Apr 2017 00:00:00 +0000

Organizing an event can be a full-time task in and of its own. I have been organizing ASA DataFest for six years at Duke, and over this time, the number of participants has grown from 23 students from Duke only, to 360 students from seven area schools this year!

First, a bit about ASA DataFest: ASA DataFest is a data “hackathon” for students around the U.S., Canada, and Germany (for now; this list has been growing each year). Students spend a weekend working in small teams, around the clock, to find insight and meaning in a large, messy, and rich data set. For almost all students, it is the most complex data they have encountered, and they push themselves to master new skills, resurrect forgotten knowledge, and bring everything they’ve got to compete for the honor of being declared the best by a panel of expert judges.

As an educator, statistician, and data scientist, growth in student interest in this event sounds fantastic to me. However, as the person responsible for running the event at Duke, it has also meant that for the couple months leading up to DataFest, I have almost an additional full-time job dealing with everything from student registrations to promoting the event to putting in food orders. While I have not found an R-based solution for ordering food (yet!), this year I incorporated R and R Markdown in my organization workflow for grabbing, processing, and reporting registration information.

This post highlights using Google Forms for data collection (e.g., registration), the googlesheets package to pull that data into R, and packages from the tidyverse to manipulate, summarise, and visualize that data. Then, we use RPubs for publishing documents to be shared with participants and other constituents.

Here is a list of all packages used in this article:

library(googlesheets)
library(tidyverse)
library(stringr)
library(DT)
library(knitr)

In an effort to make it easier for others organizing DataFests to replicate my workflow, I have created a Google Drive containing all forms needed for registering participants and collecting information from consultants (mentors), and judges. I have also populated these forms with randomly generated names to showcase how these data are processed to yield the rosters and reports that are useful for organizing the event and disseminating registration information. All Google Forms mentioned can be found in the DataFest Organization Google Drive, which is available for public viewing. You can make a copy for your own use.

Additionally, all R scripts and R Markdown documents used to process these data are available on the datafest GitHub repo.

Team sign ups

If a group of students has already formed a team, it makes sense for them to sign up as a team to ensure that they use the same team name and that everyone registers at once. This Google Form is used to sign such students up.

One issue with registering each team as a single entry is that we end up with what we call “wide” data: each row represents a team, and within that row we have information on all students in that team. However for most practical purposes (counting participants, plotting distributions of years and majors, figuring out how many of each size t-shirt to order, etc.) we need the data to be in “long” format, where each row represents a student.

To accomplish this transformation, we first read the data in using the googlesheets package:

part_wide <- gs_title("DataFest [YEAR] @ [HOST] - Team Sign up (Responses)") %>%
  gs_read()

Then, we realize that the variable names are a mess since they come directly from questions on the Google form!

names(part_wide)

##  [1] "timestamp"       "team_name"       "last_name_1"    
##  [4] "first_name_1"    "school_1"        "tshirt_size_1"  
##  [7] "class_year_1"    "major_1"         "email_1"        
## [10] "participation_1" "diet_1"          "last_name_2"    
## [13] "first_name_2"    "school_2"        "tshirt_size_2"  
## [16] "class_year_2"    "major_2"         "email_2"        
## [19] "participation_2" "diet_2"          "last_name_3"    
## [22] "first_name_3"    "school_3"        "tshirt_size_3"  
## [25] "class_year_3"    "major_3"         "email_3"        
## [28] "participation_3" "diet_3"          "last_name_4"    
## [31] "first_name_4"    "school_4"        "tshirt_size_4"  
## [34] "class_year_4"    "major_4"         "email_4"        
## [37] "participation_4" "diet_4"          "last_name_5"    
## [40] "first_name_5"    "school_5"        "tshirt_size_5"  
## [43] "class_year_5"    "major_5"         "email_5"        
## [46] "participation_5" "diet_5"          "photo"

Using stringr, we can get these variable names in concise snake_case shape:

names(part_wide) <- names(part_wide) %>%
  str_replace(" of team member", "") %>%
  str_replace(" in DataFest before", "") %>%
  str_replace(" Check all that apply.", "") %>%
  str_replace("Email address", "email") %>%
  str_replace("Dietary restrictions", "diet") %>%
  str_replace("Check if you agree", "photo") %>%
  str_replace("\\:", "") %>%
  str_replace("-", "") %>%
  str_replace_all(" ", "_") %>%
  tolower()

We can see that things look a lot better now:

names(part_wide)

##  [1] "timestamp"       "team_name"       "last_name_1"    
##  [4] "first_name_1"    "school_1"        "tshirt_size_1"  
##  [7] "class_year_1"    "major_1"         "email_1"        
## [10] "participation_1" "diet_1"          "last_name_2"    
## [13] "first_name_2"    "school_2"        "tshirt_size_2"  
## [16] "class_year_2"    "major_2"         "email_2"        
## [19] "participation_2" "diet_2"          "last_name_3"    
## [22] "first_name_3"    "school_3"        "tshirt_size_3"  
## [25] "class_year_3"    "major_3"         "email_3"        
## [28] "participation_3" "diet_3"          "last_name_4"    
## [31] "first_name_4"    "school_4"        "tshirt_size_4"  
## [34] "class_year_4"    "major_4"         "email_4"        
## [37] "participation_4" "diet_4"          "last_name_5"    
## [40] "first_name_5"    "school_5"        "tshirt_size_5"  
## [43] "class_year_5"    "major_5"         "email_5"        
## [46] "participation_5" "diet_5"          "photo"

Finally, then we use dplyr and tidyr to transform the data from wide to long:

participants <- part_wide %>%
  select(-photo) %>%
  gather(column, entry, last_name_1:diet_5, -timestamp, -team_name) %>%
  mutate(person_in_team = str_match(column, "[0-9]")) %>%
  mutate(column = str_replace(column, "_[0-9]", "")) %>%
  spread(column, entry) %>%
  filter(!is.na(last_name)) %>%
  arrange(team_name, last_name, first_name) %>%
  select(-person_in_team) %>%
  select(timestamp, team_name, first_name, last_name, email, school, 
         class_year, major, participation, diet, tshirt_size)    # reorder

Let’s take a peek:

participants

## # A tibble: 16 x 11
##             timestamp          team_name first_name last_name
##                 <chr>              <chr>      <chr>     <chr>
## 1   4/2/2017 22:20:05      Bae's Theorem   Adrienne    Fuller
## 2   4/2/2017 22:20:05      Bae's Theorem     Sylvia     Hicks
## 3   4/2/2017 22:20:05      Bae's Theorem       Toni   Simpson
## 4   4/2/2017 22:20:05      Bae's Theorem      Vicky     Water
## 5    4/4/2017 1:03:26      Bayes Anatomy   Meredith      Gray
## 6    4/4/2017 1:03:26      Bayes Anatomy      Derek  Shepherd
## 7   4/3/2017 16:14:00         Fake iid's    Carolyn      Byrd
## 8   4/3/2017 16:14:00         Fake iid's     Gordon   Hawkins
## 9   4/3/2017 16:14:00         Fake iid's    Cecilia   Pittman
## 10  4/3/2017 16:14:00         Fake iid's       Paul      Rios
## 11  4/3/2017 16:14:00         Fake iid's       Ryan      Rose
## 12 3/31/2017 23:55:00 Passive Regression     Amanda      Boyd
## 13 3/31/2017 23:55:00 Passive Regression       Rosa       Fox
## 14 3/31/2017 23:55:00 Passive Regression      Lucas  Gonzales
## 15  4/1/2017 20:14:05            The Pit      James   Andrews
## 16  4/1/2017 20:14:05            The Pit        Tom  Lawrence
## # ... with 7 more variables: email <chr>, school <chr>, class_year <chr>,
## #   major <chr>, participation <chr>, diet <chr>, tshirt_size <chr>

We can now easily look for duplicates (sometimes students sign up twice or more times) or use these data to explore the various features of participants.

Individual sign-ups

If a student is wanting to participate in DataFest but they don’t have a team in mind, we ask them to fill out a brief survey where they answer questions about their background as well as how much time they are wanting to commit to DataFest, ranging from “I’m in it to win it” to “I’m more interested in the experience, and am not really sure if I’ll submit a final presentation.”

Sometimes students find a team and register with that team after having filled out this survey. These students should be removed from the list of those looking for teammates, though there is no easy way for them to do so in Google Forms (they can’t go back and remove their response).

However we can easily do this with an anti_join. Suppose this data frame is called looking, and remember that the earlier data frame of students registering with teams was called participants.

looking <- anti_join(looking, participants, by = "email")

Then, the survey results are made available to the same students who are looking for teammates so that they can match up with others and form a team.

looking %>%
  select(first_name, last_name, participation_level, class_year, major, school, participation_before, email) %>%
  arrange(participation_level, class_year, major, school) %>%
  datatable()

Here we make use of the datatable function in the DT package to display the list of students in a pretty and easily sortable and searchable format.

Consultants and judges

Using a similar approach we can also grab, organize, and report lists of consultants and judges. All relevant code for this can be found in the datafest GitHub repo.

Participant summary

Now that we have our participant data in a tidy format, we can visualize distributions of majors, years, previous participation etc.

For example, we can count how many teams are participating from each school:

participants %>%
  distinct(team_name, .keep_all = TRUE) %>%
  count(school) %>%
  arrange(desc(n))

## # A tibble: 3 x 2
##                    school     n
##                     <chr> <int>
## 1           Faber College     2
## 2 Port Chester University     2
## 3     Harrison University     1

or visualize the distribution of class years per school:

ggplot(data = participants, aes(x = school, fill = class_year)) +
  geom_bar(position = "fill") +
  labs(title = "Schools and class years")

Information guides

We can also use R Markdown to create documents that are mostly text, that introduce the event to the participants, consultants, and judges. Then, summary statistics and visualizations of the participants can easily be included in these guides.

Sample guides for participants and consultants/judges can also be found on the GitHub repo.

And finally, all of these can be published on RPubs. However, note that these documents will be publicly available.

Quandl and Forecasting

Fri, 17 Mar 2017 00:00:00 +0000

Welcome to another installment of Reproducible Finance with R. Today we are going to shift focus in recognition of the fact that there’s more to finance than stock prices, and there’s more to data download than quantmod/getSymbols. In this post, we will explore oil prices using data from Quandl, a repository for both free and paid data sources. We will also get into the forecasting game a bit and think about how best to use dygraphs when visualizing predicted time series as an extension of historical data. We are not going to do anything too complex, but we will expand our toolkit by getting familiar with Quandl, commodity prices, the forecast package, and highcharter. Our ultimate goal is to build a tool where an end user can explore and forecast commodities prices via a Shiny app. The final app can be viewed here.

Before we dive in, a few thoughts to frame this post.

We are using oil data from Quandl, but the original data is from FRED. There’s nothing wrong with grabbing the data directly from FRED, of course, and I browse FRED frequently to check out economic data, but I tend to download the data into my RStudio environment using Quandl. I wanted to introduce Quandl today because it’s a nice resource and it’s gaining in popularity. If you work in the financial industry, you might start to encounter it in your work.
This post marks our first foray into the world of predictive modeling, albeit in a very simple way. But the complexity and accuracy of the forecasting methodology we use here is almost irrelevant since I expect that most R coders, whether in industry or otherwise, will have their own proprietary models. Rather, what I want to accomplish here is a framework where models can be inserted, visualized, and scrutinized in the future. I harp on reproducible workflows a lot, and that’s not going to change today because one goal of this Notebook is to house a forecast that can be reproduced in the future (at which point, we will know if the forecast was accurate or not), and then tweaked/criticized/updated/heralded.

Let’s get to the data download! In the chunk below, as we import WTI oil prices, notice that Quandl makes it easy to choose types of objects (raw/dataframe, xts, or zoo), periods (daily, weekly, or monthly) and start/end dates.

library(Quandl)
library(tidyverse)
library(tidyquant)
library(timetk)
library(forecast)
library(highcharter)

# You might want to supply an API key. It's free to sign up.
# Quandl.api_key("your API key here")

# Start with daily data. Note that "type = raw" will download a data frame.
oil_daily <- Quandl("FRED/DCOILWTICO", 
                    type = "raw", 
                    collapse = "daily",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# Now weekly and let's use xts as the type.
oil_weekly <- Quandl("FRED/DCOILWTICO", 
                     type = "xts", 
                     collapse = "weekly",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# And monthly using xts as the type.
oil_monthly <- Quandl("FRED/DCOILWTICO", 
                      type = "xts", 
                      collapse = "monthly",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# Have a quick look at our three  objects. 
head(oil_daily)

##         Date Value
## 1 2017-12-29 60.46
## 2 2017-12-28 59.84
## 3 2017-12-27 59.67
## 4 2017-12-26 59.55
## 5 2017-12-22 58.25
## 6 2017-12-21 58.34

head(oil_weekly)

##             [,1]
## 2008-01-06 97.90
## 2008-01-13 92.74
## 2008-01-20 90.55
## 2008-01-27 90.37
## 2008-02-03 89.03
## 2008-02-10 91.77

head(oil_monthly)

##            [,1]
## Jan 2008  91.67
## Feb 2008 101.78
## Mar 2008 101.54
## Apr 2008 113.70
## May 2008 127.35
## Jun 2008 139.96

Note that we specified the start date as January 1, of 2008 and end date of January 1, 2018 so we will be working with 10 years of data.

Each of the oil data objects we created would work well for the rest of this project, but let’s stick with the monthly data. I don’t love the formate of the column so let’s use the seq() function and mdy() from the lubridate package to put the date into a nicer format.

index(oil_monthly) <- seq(mdy('01/01/2008'), mdy(last(index(oil_monthly))), by = 'months')

head(index(oil_monthly))

## [1] "2008-01-01" "2008-02-01" "2008-03-01" "2008-04-01" "2008-05-01"
## [6] "2008-06-01"

Now we have a cleaner date format and our price data object is in good shape. Let’s fire up highcharter and visualize our price history. Since we imported an xts object directly from Quandl, we can plug it straight into the hchart() function.

hchart(oil_monthly)

Or we can use the highchart(type = "stock") code flow to produce the same chart. Let’s also add a $ label to the y-axis with hc_yAxis.

highchart(type = "stock") %>% 
  hc_add_series(oil_monthly, color = "cornflowerblue") %>% 
  hc_yAxis(title = list(text = "monthly price"),
           labels = list(format = "${value}"),
           opposite = FALSE) %>% 
  hc_add_theme(hc_theme_flat())

Nothing too shocking here. We see a peak in mid-2008, followed by a precipitous decline through the beginning of 2009.

Now we’ll make things a bit more interesting and try to extract some meaning from that data. First, let’s fit an arima model to our time series using the auto.arima() function from the forecast package. This is for illustrative purposes only and probably isn’t the best model for oil prices.

auto.arima(oil_monthly)

## Series: oil_monthly 
## ARIMA(1,1,0) 
## 
## Coefficients:
##          ar1
##       0.3218
## s.e.  0.0871
## 
## sigma^2 estimated as 47.84:  log likelihood=-398.55
## AIC=801.1   AICc=801.2   BIC=806.65

Next we can use the forecast() function to predict what oil prices will look like over the next six months, based on the arima model we just fit.

auto.arima(oil_monthly) %>% 
  forecast(h = 6)

##     Point Forecast    Lo 80    Hi 80    Lo 95     Hi 95
## 121       61.44478 52.58047 70.30909 47.88799  75.00157
## 122       61.76170 47.06935 76.45406 39.29169  84.23172
## 123       61.86370 42.48557 81.24182 32.22741  91.49998
## 124       61.89652 38.60035 85.19270 26.26809  97.52495
## 125       61.90709 35.21664 88.59754 21.08757 102.72661
## 126       61.91049 32.19775 91.62322 16.46878 107.35219

The point forecast is around $62. It looks like the 95% confidence interval 6 months out has a high of $16 and a low of $107. We won’t dwell on these numbers because I imagine you will want to use your own model here - this code flow is more of a skeleton where other models can be inserted and then tested or evaluated at a later date.

Let’s move on to visualizing the results of the forecast along with the historical data. A great feature of highcharter is that it accepts the results of forecast directly. We can pipe the model and forecast results directly to hchart().

auto.arima(oil_monthly) %>% 
  forecast(h = 6) %>% 
  hchart() %>% 
  hc_title(text = "Oil historical and forecast") %>% 
  hc_yAxis(title = list(text = "monthly price"),
           labels = list(format = "${value}"),
           opposite = FALSE) %>% 
  hc_add_theme(hc_theme_flat()) %>% 
  hc_navigator(enabled = TRUE)

That’s all for today. We have gotten some familiarity with Quandl, used forecast() to predict the next six months of oil prices, and seen how smooth it is to pass forecasts to highcharter. Next time, we will wrap this into a Shiny app so that users can choose their own parameters, and choose different commodities. See you then!

Note: this post was updated on August 2, 2018.

Why I love R Notebooks

Wed, 15 Mar 2017 00:00:00 +0000

Note: R Notebooks requires RStudio Version 1.0 or later

I’m a big fan of the R console. During my early years with R, that’s all I had, so I got very comfortable with pasting my code into the console. Since then I’ve used many code editors for R, but they all followed the same paradigm – script in one window and get output in another window. Notebooks on the other hand combine code, output, and narrative into a single document. Notebooks allow you to interactively build narratives around small chunks of code and then publish the complete notebook as a report.

R Notebooks is a new feature of RStudio that combines the benefits of other popular notebooks (such as Jupyter, Zeppelin, and Beaker) with the benefits of R Markdown documents. As a long-time R user, I was skeptical that I would like this new paradigm, but after a few months I became a big fan. Here are my top three reasons why I love R Notebooks.

Number 3: Notebooks are for doing science

If scripting is for writing software, then notebooks are for doing data science. In my high school science class I used a laboratory notebook that contained all my experiments. When I conducted an experiment, I drew sketches and wrote down my results. I also wrote down my ideas and thoughts. The process had a nice flow which helped me improve my thinking. Doing science with physical notebooks is an idea that is centuries old.

Electronic notebooks follow the same pattern as physical notebooks, but apply the pattern to code. With notebooks, you break your script into manageable code chunks. You add narrative and output around the code chunk, which puts it into context and makes it reproducible. When you are done, you have an elegant report that can be shared with others. Here is the thought process for doing data science with notebooks:

I have a chunk of code that I want to tell you about.
I am going to execute this chunk of code and show you the output.
I am going to share all chunks of code with you in a single, reproducible document.

If you do data science with R scripts, on the other hand, you develop your code as a single script. You add comments to the code, but the comments tend to be terse or nonexistent. Your output may or may not be captured at all. Sharing your results in a report requires a separate, time consuming process. Here is the thought process for doing data science with scripts:

I have a thousand lines of code and you get to read my amazing comments!
Hold onto your hats while I batch execute this entire script!
You can find my code and about 50 plots under the project directory (I hope you have permissions).

Number 2: R Notebooks have great features

R Notebooks are based on R Markdown documents. That means they are written in plain text and work well with version control. They can be used to create elegantly formatted output in multiple document types (e.g. HTML, PDF, and Word).

R Notebooks have some features that are not found in traditional notebooks. These are not necessarily inherent differences, but differences of emphasis. For example, R Markdown documents give you many options when selecting graphics, templates, and formats for your output.

Feature	R Notebooks	Traditional Notebooks
Plain text representation	✓
Same editor/tools used for R scripts	✓
Works well with version control	✓
Create elegantly formatted output	✓
Output inline with code	✓	✓
Output cached across sessions	✓	✓
Share code and output in a single file	✓	✓
Emphasized execution model	Interactive & Batch	Interactive

When you execute a code chunk in an R Notebook, the output is cached and rendered inside the IDE. When you save the notebook, the same cache is rendered inside a document. The HTML output of R Notebooks is a dual-file format that contains both the HTML and the R Markdown source code. The dual format gives you a single file that can be viewed in a browser or opened in the RStudio IDE.

R Notebooks for data science

The following table summarizes the differences between notebooks and scripts.

Activity	R Notebook	R Script
Building a narrative	Rich text is added throughout the analytic process to describe the motivation and the conclusions for each chunk of the code.	Comments are added to the script, and a report that describes the entire analysis is drafted separately after the script is completed.
Organizing plots, widgets, and tables	All output is embedded in a single document and collocated with the narrative and code chunk to which it belongs.	Each individual output is sent to file and is collected later into a report.
Creating reports	Rendering the final report is instant. The same document can be published to multiple formats (e.g. HTML, PDF, Word). Since the document is based on code, future changes are easy to implement and the document is reproducible by others.	Creating a report is a separate, time-consuming step. Any changes to the report can be time-consuming and prone to error. Since the report is not tied to code, it is not reproducible.

R Notebooks are not designed for all the work you do in R. If you are writing software for a new package or building a Shiny app, you will want to use an R script. However, if you are doing data science you might try R Notebooks. They are great for tasks like exploratory data analysis, model building, and communicating insights. Notebooks are useful for data science because they organize narrative, code, and text around manageable code chunks; and creating quality, reproducible reports is easy.

References: For an introduction to R Notebooks see this video or blog post. For more detailed information, see this workflow presentation or the reference site.

R Markdown for the Enterprise

Wed, 25 Jan 2017 00:00:00 +0000

In the corporate world, spreadsheets and PowerPoint presentations still dominate as the tools used for analyzing and sharing information. So, it is not at all surprising that even when business analysts use R for the analytical heavy lifting, they frequently revert to using spreadsheets and slide decks to share their results. This may seem like the easiest way to communicate with colleagues, but any modestly complicated project is likely to be error-prone and generate hours of unnecessary rework.

An R-savvy analyst can harness R Markdown to develop reproducible business reporting and information sharing workflows in any business organization; all it takes is a little effort to master some basic R document preparation tools.

In this post, I would like to examine a scenario that represents some experiences I had as an analytics professional.

“The report is great but…” Scenario

A new R analysis is delivered in a PowerPoint presentation, and everyone thinks that the insights are very valuable. They all want more associates to see it, so almost immediately, the following three requests are made:

“…we need it broken out by” - The presentation needs to be split by a specific segment. The segment is normally geographical or managerial in nature.
“…they shouldn’t see each others data” - Since the results are not published in a central publishing platform, it is necessary to create multiple versions of the same report in order to secure the contents.
“…we need it every” - Satisfying requests 1 and 2 may not be too overwhelming if this were meant as a one-time analysis, but usually the analysis and its distribution need to be repeated on a regular interval.

Because we exported the findings into a presentation, sharing results becames more complex and time-consuming if we wish to satisfy the new requirements.

How can R Markdown help?

R Markdown combines the creation and sharing steps. The three requests can be satisfied using the following features of R Markdown:

Break out the reports - Using R Markdown’s Parameterized Reports feature, we can easily create documents for each required segment.
Automate the file creation - R Markdown can be run from code, so a separate R script can iteratively run the R Markdown and pass a different parameter for each iteration.
Create the slides inside R - Take advantage of R Markdown Presentation output to create a slide deck. Without having to learn a new scripting language, we can code the slide deck and use the same Parameter feature to automate its creation.
Keep the interactivity - In many cases, the end user needs a level of interactivity with the report. This interactivity can be achieved by using htmlwidgets inside the R Markdown document. For example, the Leaflet widget can be used for interactive maps, the Data Table widget for interactive tables, and the dygraphs widget for interactive time series charting.

Additional benefits

Accessible and easy to open - Any alternative tool needs to be as accessible as the current spreadsheet and presentation tool. R Markdown can output results in HTML, PDF, and Word. Additionally, the Presentation output uses the highly accessible HTML5 format.
Reproducibility - Copying-and-pasting files, text, or images inevitably introduces human error. In R, data import, wrangling and modeling are already automated, so why not take it to its natural conclusion by using R Markdown to automate the presentation end of the process, as well?
Creating a dashboard is easy - In a spreadsheet, this is normally accomplished with a combination of pivot tables and graphs. R Markdown uses flexdashboard to create visually striking dashboards that are self-contained. By using this in combination with htmlwidgets, the audience gains access to a very powerful tool.

Here is an example of a live parameterized R Markdown flexdashboard based on stock data:

How to get started

R Markdown is a free package, so if you have R (and ideally RStudio), you can start using it today. Also, there are a lot of resources available for learning how to use R Markdown; the package’s official website is a good place to start.

Here is a sample script that uses Parameterized R Markdown to create a slide deck based on a selected stock. In this case we used Google:

And here is the resulting deck. Press the left arrow key to see the next slide:

This simple script creates an nice-looking and interactive deck that needs no manual intervention if the data needs to be refreshed, and one small parameter change if a different stock is to be selected.

Final thought

We encourage you to try R Markdown yourself. The “start small and then build big” strategy rarely fails, so you could begin by automating a simple report first, and then start taking advantage of more advanced features as you grow comfortable with the tool.