R on R Views

Multiple Versions of R

Wed, 21 Mar 2018 00:00:00 +0000

Data scientists prefer using the latest R packages to analyze their data. To ensure a good user experience, you will need a recent version of R running on a modern operating system. If you run R on a production server – and especially if you use RStudio Connect – plan to support multiple versions of R side by side so that your code, reports, and apps remain stable over time. You can support multiple versions of R concurrently by building R from source. Plan to install a new version of R at least once per year on your servers.

A solid foundation for R

Administering R on the desktop is relatively easy, because desktops are designed for a single user at a specific time. Desktop users upgrade R versions and R packages as new software becomes available, leaving old versions and packages behind. Servers, on the other hand, are designed to support multiple people who want to access content across time. Servers are increasingly used for building data science labs in R, deploying R in production, and running R in the cloud. You may find that the same strategies you use to administer R on your desktop do not work as well on a server. In particular, upgrading your version of R must be handled differently.

If you upgrade R on your server as you do on your desktop, you could easily break some apps and disrupt your teams. Administrators should exercise caution when upgrading to a new version of R on a Linux server. Consider the following situations:

You are hosting apps on RStudio Connect and Shiny Server for more than a year. When you upgrade R, you break many of your older apps.
Your team is developing code on a shared instance of RStudio Server. When you upgrade R, you disrupt people’s work and break their code.

Instead of upgrading your existing version of R, a better solution to these problems is to run multiple versions of R side by side. This strategy preserves past versions of R so you can manage upgrades and keep your code, apps, and reports stable over time.

Building R from source

The best way to run multiple versions of R side by side is to build R from source. If you are running R on a Linux server – and particularly in the enterprise – you should always build R from source, because it will help you:

Run multiple versions of R side by side
Guarantee that R will work on your unique server configuration
Potentially speed up certain low-level computations used by R
Build technical expertise that will help you administer R at scale

Most enterprise IT departments will be comfortable building software from source. If you have never built R from source, it is very straightforward. First, you need the build dependencies for R. If you’ve already installed R from a binary source like CRAN or EPEL, you may already have these dependencies installed; otherwise, you can run sudo yum-builddep R on RedHat or sudo apt-get build-dep r-base on Ubuntu. Second, you should obtain and unpack the source tarball for the version of R you want to install from CRAN. Third, from within the extracted source directory, build R from source using configure, make, and make install commands. For example:

# BUILD R FROM SOURCE ON REDHAT LINUX
# R-3.4.3

# Install Linux dependencies
$ sudo yum-builddep R

# Download and extract source code
$ wget https://cran.r-project.org/src/base/R-3/R-3.4.3.tar.gz
$ tar -xzvf R-3.4.3.tar.gz
$ cd R-3.4.3

# Build R from source
$ ./configure --prefix=/opt/R/$(cat VERSION) --enable-R-shlib --with-blas --with-lapack
$ make
$ sudo make install

This script installs R version 3.4.3 into /opt/R/3.4.3, but you can install R into any of the recommended directories. The --enable-R-shlib option is required to make the shared libraries known to RStudio. The --with-blas and --with-lapack options are not required, but are commonly included. These options install the system BLAS and LAPACK libraries, which are used to speed up certain low-level math computations (e.g., multiplying and inverting matrices). These libraries will not speed up R itself, but can significantly speed up the underlying code execution.

If you run into problems installing R from source, you can always remove the installation directory and start over. However, once the installation succeeds, you should never move the installation directory – in other words, always install into the final destination directory. If you run into problems with dependencies, make sure you are able to identify and install all of the required Linux libraries (e.g., the X11 library is commonly overlooked). Building R from source will be much easier with a modern operating system that is connected to the Internet.

For further details about building R from source, see the RStudio Server Admin Guide.

RStudio professional products

RStudio professional products automatically support multiple versions of R and provide additional features, such as having administrators control access to multiple versions, or allowing users to choose for themselves. RStudio Connect automatically provides R version matching. Running multiple versions of R side by side with RStudio Connect will ensure that your content persists over time.

References

A Data Science Lab for R

Wed, 20 Dec 2017 00:00:00 +0000

In a previous post I described the role of analytic administrator as a data scientist who: onboards new tools, deploys solutions, supports existing standards, and trains other data scientists. In this post I will describe how someone in that role might set up a data science lab for R.

Architecture

A data science lab is an environment for developing code and creating content. It should enhance the productivity of your data scientists and integrate with your existing systems. Your data science lab might live on your premises or in the cloud. It might be built with hardware, virtual machines, or containers. You may use it to support a single data scientist or hundreds of R developers. Here is one reference architecture of a data science lab based on server instances.

Key components of this setup include: authentication; load balancing; a testing environment; data connectivity; and a publishing platform. In this server-based architecture, data scientists use a web browser to access the data science lab. High performance compute and live data reside securely behind a firewall.

Instance Sizing

The size of your server instance depends on how many concurrent sessions you run and how large your sessions are. Keep in mind that R is single threaded by default and holds data in memory. Here is a list of example server sizes:

Instance Size	Cores	RAM	Description
Minimum recommended	2	4G	This server will be for lightweight jobs, testing, and sandboxing.
Small	4	8G	This server will support one or two analysts with small data.
Large	16	256G	This server will support 15 analysts with a blend of large and small sessions. Alternatively, it will support dozens of analysts with small sessions.
Jumbo	32+	1T+	May be useful for heavier workloads.

Open-Source R

If you haven’t done so already, I recommend you make R a legitimate part of your organization by officially recognizing it as an analytic standard. You should be familiar with installing and managing R and its packages.

You can install R as a pre-compiled binary from a repository, or you can install R from source. Installing R from source allows you to install multiple versions of R side by side. If you compile R from source, I recommend you link to the BLAS libraries so that you can speed up certain low-level math computations.

Data science labs tend to require a modern toolkit. You should expect to upgrade R at least once a year. You should also keep your operating system up to date. New and improved R packages tend to work better when you use them with recent versions of R and updated system libraries.

RStudio Server Pro

Building a data science lab involves installing, configuring, and managing tools. In this section I will describe how to administer RStudio Server Pro which has features for authentication, security, and admin controls.

1. Installation

Once you have installed R, you can install RStudio Server Pro by downloading the binaries and following the instructions. You will need root privileges to install and run the software. You will also need to create local system accounts for all of your R developers.

2. Configuration

Authentication. The first thing you will want to do after you install RStudio Server Pro is to configure it with your authentication system. RStudio Server Pro supports LDAP via PAM sessions. If you use single sign on or another system, you can configure RStudio Server Pro to work in proxied auth mode. You can also authenticate via Google accounts and local system accounts.

Data Connectivity. Most data scientists use R with databases. The RStudio Pro Drivers are ODBC drivers that will connect R to some of the most popular databases today. These drivers are a free add-on for RStudio Server Pro. If you are using a data source that is not supported, or if you are using the open source version of RStudio Server, you can bring your own ODBC driver.

Load Balancing. If you want to load balance your server instances, you can use the load balancer that is built into RStudio Server Pro or you can bring your own load balancer. Load balancing is designed to balance user sessions seamlessly across the cluster and provide high availability. It requires a shared home drive that is mounted to each one of the instances.

More Features. RStudio Server Pro has a list of features that you can configure. You should decide which features you want to enable or disable. For more information on configuring each of these features, see the admin guide.

Feature	Description
Authentication	LDAP, Active Directory, Google Accounts and system accounts Full support for Pluggable Authentication Modules, Kerberos via PAM, and custom authentication via proxied HTTP header
Data Connectivity	RStudio Professional Drivers are ODBC data connectors that help you connect to some of the most popular databases.
Load Balancing	Load balance R sessions across two or more servers Ensure high availability using multiple masters
Enhanced security	Encrypt traffic using SSL and restrict client IP addresses
Administrative dashboard	Monitor active sessions and their CPU and memory utilization Suspend, forcibly terminate, or assume control of any active session Review historical usage and server logs
Auditing and monitoring	Monitor server resources (CPU, memory, etc.) on both a per-user and system-wide basis Send metrics to external systems with the Graphite/Carbon plaintext protocol Health check with configurable output (custom XML, JSON) Audit all R console activity by writing input and output to a central location
Advanced R session management	Tailor the version of R, reserve CPU, prioritize scheduling and limit resources by User and Group Provision accounts and mount home directories dynamically via the PAM Session API Automatically execute per-user profile scripts for database and cluster connectivity
Project sharing	Share projects & edit code files simultaneously with others

3. Management

Once RStudio Server Pro is installed and configured, you’ll need to manage it over time. RStudio Server Pro comes with a variety of tools for workspace and server management that will help keep your environment organized. For example, you can kill sessions, set session timeouts, and broadcast notifications to user sessions in real-time. You can manage product licenses for both online and offline environments. If your instances start and stop frequently you can opt for using a floating license manager.

Next Steps

Your data science lab for R should be designed to scale. That might mean adding more people, more systems, or more tools. It also might mean creating more content. Shiny is an R package that makes it easy to build interactive web apps straight from R. R Markdown is an R package that makes it easy to author reports and build dashboards. You can publish your Shiny apps or R Markdown reports with the push of a button to RStudio Connect. RStudio Connect lets you share and manage content in one convenient place. You can also publish Shiny apps to shinyapps.io, which allows you to share your Shiny apps online.

WordR - A New R Package for Rendering Documents in MS Word Format

Wed, 04 Oct 2017 00:00:00 +0000

Motivation

One day earlier this year, I was faced with the challenge of creating a report for management. It had to be an MS Word document (corporate requirement, you know). It was supposed to be polished and use many of the standard MS Word features like headers, footers, table of contents, and styles.

I am not a Word guy, and besides, I wanted to make a reproducible document that would make it easy for me to include R code and plots in the text. Having no idea what tools were available, I revved up Google and found several packages including R2wd, officer and ReporteRs. They are all very good packages, but for different reasons, none of them covered a hundred percent of my needs. Also, I thought that they make some things unnecessarily complicated (again, for my needs).

For example, I wanted to be able to create the document template that included a significant amount of text and styles. I also wanted to be able to mark a place to add a table or plot and mainly, I wanted to have something like R inline code in R markdown language (link).

I found out that with combination of officer and ReporteRs packages and some effort, I could achieve what I needed. (It could have been done solely with officer, but the tables and plots creation/insertion capability in ReporteRs serves me better).

Package WordR, in version 0.2.2 on CRAN now, is the result.

How to use it

This is, I hope, pretty simple. Details are described in the package vignette. The text below should provide a basic understanding.

MS Word part

As a first step, prepare a Word document with static text, images, headers, footers, styles, etc. In your text, if you need, you can add an R inline code part. It needs to be separated from the rest of the text by style separators at the beginning and the end (insert style separator by pressing Ctrl+Alt+Enter (it is preferable to have formatting symbols visible by clicking the “new line” sign button on Home>Paragraph panel in MS Word)), and it should have a form of `r expression`, where expression is an R expression like 1+1. So the result should look something like this, in the MS Word editor:

At a place where you want to render a table, insert a bookmark named t_XYZ, where XYZ will be the name of a FlexTable (ReporteRs::FlexTable) table which you want to insert.

At a place where you want to render a plot/image, insert a bookmark named p_XYZ, where XYZ will be the name of plot function (ReporteRs::addPlot.docx) you want to be used to render the plot/image.

Save the prepared Word template.

Example

The resulting document may look like this (print screen; original document is here):

If you don’t see the file in the MS Word as in the print screen above, please refer to the package vignette on how to make the bookmarks and non-printable signs visible.

Please mind a few things in the example:

R inline code in the header
two inline code snippets in the text, the second one formatted in a custom style red style
two bookmarks, one for plot and one for table

R part

Now, let’s switch to R. Install the WordR package (and all the dependencies; Java is required by the ReporteRs package, so make sure you have it working for table and plot rendering):

install.packages("WordR")

Then you have three functions from the WordR package at your disposal:

renderInlineCode for rendering the R inline code
addFlexTables for inserting FlexTables at the locations defined by the bookmarks (for more about FlexTables and their wide variety of formatting possibilities, look at ReporteRs::FlexTable)
addFlexPlots for inserting plots/images at the locations defined by the bookmarks

Please refer to the help of each function for details, but in general, you provide the path of your prepared Word file as an argument (and the tables or plots respectively), and also the desired path where you want to have your result file saved. The function returns that path upon successful execution.

Back to our example

Assume we saved our file as templateMTcars.docx in sub-directory /resources located in our working directory:

dir(path = paste0(getwd(), "/resources/"), pattern = ".docx")

## character(0)

Then the rest of the code is simple:

library(WordR)
library(ReporteRs)
library(ggplot2)
# prepare outputs (Tables and Plots)
ft_mtcars <- vanilla.table(mtcars)
FT <- list(mtcars = ft_mtcars)
Plots <- list(mtcars1 = function() print(ggplot(mtcars, aes(x = wt, y = hp, col = as.factor(cyl))) + geom_point()))

# render docx
renderInlineCode("resources/templateMTcars.docx","resources/templateMTcarsOutput.docx")
addFlexTables("resources/templateMTcarsOutput.docx","resources/templateMTcarsOutput.docx", FT)
addPlots("resources/templateMTcarsOutput.docx","resources/templateMTcarsOutput.docx", Plots, height = 4)

This should result in the rendered file resources/templateMTcarsOutput.docx looking like this (printscreen with non-printable characters and bookmarks off; in docx format here):

A few comments

In my workflow, I let people who want the report prepare the Word template (without the bookmarks and R code). This is better both for me - I can do other things - and for them - they can format the report the way they want, without iterating with me. Then I take that file, put the R inline code in place, add the bookmarks, and the template is done.

Additionally, when the result is created, it can be further changed. For example, the person who receives the report from me may add some interpretation based on the current results, which would be difficult with the traditional Rmd/LaTeX -> PDF way of reporting.

Summary

I believe that the WordR package offers something new. I admit that it is just a small modification of much more powerful packages officer and ReporteRs, but it simplified things for me a lot. Obviously it will never be the first choice for reproducible research, but I believe it can be useful for few corporate rats and other people who want to produce fully formatted reports but do not want to go deep into LaTeX.

Survival Analysis with R

Mon, 25 Sep 2017 00:00:00 +0000

With roots dating back to at least 1662 when John Graunt, a London merchant, published an extensive set of inferences based on mortality records, survival analysis is one of the oldest subfields of Statistics [1]. Basic life-table methods, including techniques for dealing with censored data, were discovered before 1700 [2], and in the early eighteenth century, the old masters - de Moivre working on annuities, and Daniel Bernoulli studying competing risks for the analysis of smallpox inoculation - developed the modern foundations of the field [2]. Today, survival analysis models are important in Engineering, Insurance, Marketing, Medicine, and many more application areas. So, it is not surprising that R should be rich in survival analysis functions. CRAN’s Survival Analysis Task View, a curated list of the best relevant R survival analysis packages and functions, is indeed formidable. We all owe a great deal of gratitude to Arthur Allignol and Aurielien Latouche, the task view maintainers.

Looking at the Task View on a small screen, however, is a bit like standing too close to a brick wall - left-right, up-down, bricks all around. It is a fantastic edifice that gives some idea of the significant contributions R developers have made both to the theory and practice of Survival Analysis. As well-organized as it is, however, I imagine that even survival analysis experts need some time to find their way around this task view. Newcomers - people either new to R or new to survival analysis or both - must find it overwhelming. So, it is with newcomers in mind that I offer the following narrow trajectory through the task view that relies on just a few packages: survival, ggplot2, ggfortify, and ranger

The survival package is the cornerstone of the entire R survival analysis edifice. Not only is the package itself rich in features, but the object created by the Surv() function, which contains failure time and censoring information, is the basic survival analysis data structure in R. Dr. Terry Therneau, the package author, began working on the survival package in 1986. The first public release, in late 1989, used the Statlib service hosted by Carnegie Mellon University. Thereafter, the package was incorporated directly into Splus, and subsequently into R.

ggfortify enables producing handsome, one-line survival plots with ggplot2::autoplot.

ranger might be the surprise in my very short list of survival packages. The ranger() function is well-known for being a fast implementation of the Random Forests algorithm for building ensembles of classification and regression trees. But ranger() also works with survival data. Benchmarks indicate that ranger() is suitable for building time-to-event models with the large, high-dimensional data sets important to internet marketing applications. Since ranger() uses standard Surv() survival objects, it’s an ideal tool for getting acquainted with survival analysis in this machine-learning age.

Load the data

This first block of code loads the required packages, along with the veteran dataset from the survival package that contains data from a two-treatment, randomized trial for lung cancer.

library(survival)
library(ranger)
library(ggplot2)
library(dplyr)
library(ggfortify)

#------------
data(veteran)
head(veteran)

##   trt celltype time status karno diagtime age prior
## 1   1 squamous   72      1    60        7  69     0
## 2   1 squamous  411      1    70        5  64    10
## 3   1 squamous  228      1    60        3  38     0
## 4   1 squamous  126      1    60        9  63    10
## 5   1 squamous  118      1    70       11  65    10
## 6   1 squamous   10      1    20        5  49     0

The variables in veteran are: * trt: 1=standard 2=test * celltype: 1=squamous, 2=small cell, 3=adeno, 4=large * time: survival time in days * status: censoring status * karno: Karnofsky performance score (100=good) * diagtime: months from diagnosis to randomization * age: in years * prior: prior therapy 0=no, 10=yes

Kaplan Meier Analysis

The first thing to do is to use Surv() to build the standard survival object. The variable time records survival time; status indicates whether the patient’s death was observed (status = 1) or that survival time was censored (status = 0). Note that a “+” after the time in the print out of km indicates censoring.

# Kaplan Meier Survival Curve
km <- with(veteran, Surv(time, status))
head(km,80)

##  [1]  72  411  228  126  118   10   82  110  314  100+  42    8  144   25+
## [15]  11   30  384    4   54   13  123+  97+ 153   59  117   16  151   22 
## [29]  56   21   18  139   20   31   52  287   18   51  122   27   54    7 
## [43]  63  392   10    8   92   35  117  132   12  162    3   95  177  162 
## [57] 216  553  278   12  260  200  156  182+ 143  105  103  250  100  999 
## [71] 112   87+ 231+ 242  991  111    1  587  389   33

To begin our analysis, we use the formula Surv(futime, status) ~ 1 and the survfit() function to produce the Kaplan-Meier estimates of the probability of survival over time. The times parameter of the summary() function gives some control over which times to print. Here, it is set to print the estimates for 1, 30, 60 and 90 days, and then every 90 days thereafter. This is the simplest possible model. It only takes three lines of R code to fit it, and produce numerical and graphical summaries.

km_fit <- survfit(Surv(time, status) ~ 1, data=veteran)
summary(km_fit, times = c(1,30,60,90*(1:10)))

## Call: survfit(formula = Surv(time, status) ~ 1, data = veteran)
## 
##  time n.risk n.event survival std.err lower 95% CI upper 95% CI
##     1    137       2    0.985  0.0102      0.96552       1.0000
##    30     97      39    0.700  0.0392      0.62774       0.7816
##    60     73      22    0.538  0.0427      0.46070       0.6288
##    90     62      10    0.464  0.0428      0.38731       0.5560
##   180     27      30    0.222  0.0369      0.16066       0.3079
##   270     16       9    0.144  0.0319      0.09338       0.2223
##   360     10       6    0.090  0.0265      0.05061       0.1602
##   450      5       5    0.045  0.0194      0.01931       0.1049
##   540      4       1    0.036  0.0175      0.01389       0.0934
##   630      2       2    0.018  0.0126      0.00459       0.0707
##   720      2       0    0.018  0.0126      0.00459       0.0707
##   810      2       0    0.018  0.0126      0.00459       0.0707
##   900      2       0    0.018  0.0126      0.00459       0.0707

#plot(km_fit, xlab="Days", main = 'Kaplan Meyer Plot') #base graphics is always ready
autoplot(km_fit)

Next, we look at survival curves by treatment.

km_trt_fit <- survfit(Surv(time, status) ~ trt, data=veteran)
autoplot(km_trt_fit)

And, to show one more small exploratory plot, I’ll do just a little data munging to look at survival by age. First, I create a new data frame with a categorical variable AG that has values LT60 and GT60, which respectively describe veterans younger and older than sixty. While I am at it, I make trt and prior into factor variables. But note, survfit() and npsurv() worked just fine without this refinement.

vet <- mutate(veteran, AG = ifelse((age < 60), "LT60", "OV60"),
              AG = factor(AG),
              trt = factor(trt,labels=c("standard","test")),
              prior = factor(prior,labels=c("N0","Yes")))

km_AG_fit <- survfit(Surv(time, status) ~ AG, data=vet)
autoplot(km_AG_fit)

Although the two curves appear to overlap in the first fifty days, younger patients clearly have a better chance of surviving more than a year.

Cox Proportional Hazards Model

Next, I’ll fit a Cox Proportional Hazards Model that makes use of all of the covariates in the data set.

# Fit Cox Model
cox <- coxph(Surv(time, status) ~ trt + celltype + karno                   + diagtime + age + prior , data = vet)
summary(cox)

## Call:
## coxph(formula = Surv(time, status) ~ trt + celltype + karno + 
##     diagtime + age + prior, data = vet)
## 
##   n= 137, number of events= 128 
## 
##                         coef  exp(coef)   se(coef)      z Pr(>|z|)    
## trttest            2.946e-01  1.343e+00  2.075e-01  1.419  0.15577    
## celltypesmallcell  8.616e-01  2.367e+00  2.753e-01  3.130  0.00175 ** 
## celltypeadeno      1.196e+00  3.307e+00  3.009e-01  3.975 7.05e-05 ***
## celltypelarge      4.013e-01  1.494e+00  2.827e-01  1.420  0.15574    
## karno             -3.282e-02  9.677e-01  5.508e-03 -5.958 2.55e-09 ***
## diagtime           8.132e-05  1.000e+00  9.136e-03  0.009  0.99290    
## age               -8.706e-03  9.913e-01  9.300e-03 -0.936  0.34920    
## priorYes           7.159e-02  1.074e+00  2.323e-01  0.308  0.75794    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
##                   exp(coef) exp(-coef) lower .95 upper .95
## trttest              1.3426     0.7448    0.8939    2.0166
## celltypesmallcell    2.3669     0.4225    1.3799    4.0597
## celltypeadeno        3.3071     0.3024    1.8336    5.9647
## celltypelarge        1.4938     0.6695    0.8583    2.5996
## karno                0.9677     1.0334    0.9573    0.9782
## diagtime             1.0001     0.9999    0.9823    1.0182
## age                  0.9913     1.0087    0.9734    1.0096
## priorYes             1.0742     0.9309    0.6813    1.6937
## 
## Concordance= 0.736  (se = 0.03 )
## Rsquare= 0.364   (max possible= 0.999 )
## Likelihood ratio test= 62.1  on 8 df,   p=2e-10
## Wald test            = 62.37  on 8 df,   p=2e-10
## Score (logrank) test = 66.74  on 8 df,   p=2e-11

cox_fit <- survfit(cox)
#plot(cox_fit, main = "cph model", xlab="Days")
autoplot(cox_fit)

Note that the model flags small cell type, adeno cell type and karno as significant. However, some caution needs to be exercised in interpreting these results. While the Cox Proportional Hazard’s model is thought to be “robust”, a careful analysis would check the assumptions underlying the model. For example, the Cox model assumes that the covariates do not vary with time. In a vignette [12] that accompanies the survival package Therneau, Crowson and Atkinson demonstrate that the Karnofsky score (karno) is, in fact, time-dependent so the assumptions for the Cox model are not met. The vignette authors go on to present a strategy for dealing with time dependent covariates.

Data scientists who are accustomed to computing ROC curves to assess model performance should be interested in the Concordance statistic. The documentation for the survConcordance() function in the survival package defines concordance as “the probability of agreement for any two randomly chosen observations, where in this case agreement means that the observation with the shorter survival time of the two also has the larger risk score. The predictor (or risk score) will often be the result of a Cox model or other regression” and notes that: “For continuous covariates concordance is equivalent to Kendall’s tau, and for logistic regression is is equivalent to the area under the ROC curve.”

To demonstrate using the survival package, along with ggplot2 and ggfortify, I’ll fit Aalen’s additive regression model for censored data to the veteran data. The documentation states: “The Aalen model assumes that the cumulative hazard H(t) for a subject can be expressed as a(t) + X B(t), where a(t) is a time-dependent intercept term, X is the vector of covariates for the subject (possibly time-dependent), and B(t) is a time-dependent matrix of coefficients.”

The plots show how the effects of the covariates change over time. Notice the steep slope and then abrupt change in slope of karno.

aa_fit <-aareg(Surv(time, status) ~ trt + celltype +
                 karno + diagtime + age + prior , 
                 data = vet)
aa_fit

## Call:
## aareg(formula = Surv(time, status) ~ trt + celltype + karno + 
##     diagtime + age + prior, data = vet)
## 
##   n= 137 
##     75 out of 97 unique event times used
## 
##                       slope      coef se(coef)      z        p
## Intercept          0.083400  3.81e-02 1.09e-02  3.490 4.79e-04
## trttest            0.006730  2.49e-03 2.58e-03  0.967 3.34e-01
## celltypesmallcell  0.015000  7.30e-03 3.38e-03  2.160 3.09e-02
## celltypeadeno      0.018400  1.03e-02 4.20e-03  2.450 1.42e-02
## celltypelarge     -0.001090 -6.21e-04 2.71e-03 -0.229 8.19e-01
## karno             -0.001180 -4.37e-04 8.77e-05 -4.980 6.28e-07
## diagtime          -0.000243 -4.92e-05 1.64e-04 -0.300 7.65e-01
## age               -0.000246 -6.27e-05 1.28e-04 -0.491 6.23e-01
## priorYes           0.003300  1.54e-03 2.86e-03  0.539 5.90e-01
## 
## Chisq=41.62 on 8 df, p=1.6e-06; test weights=aalen

#summary(aa_fit)  # provides a more complete summary of results
autoplot(aa_fit)

Random Forests Model

As a final example of what some might perceive as a data-science-like way to do time-to-event modeling, I’ll use the ranger() function to fit a Random Forests Ensemble model to the data. Note however, that there is nothing new about building tree models of survival data. Terry Therneau also wrote the rpart package, R’s basic tree-modeling package, along with Brian Ripley. See section 8.4 for the rpart vignette [14] that contains a survival analysis example.

ranger() builds a model for each observation in the data set. The next block of code builds the model using the same variables used in the Cox model above, and plots twenty random curves, along with a curve that represents the global average for all of the patients. Note that I am using plain old base R graphics here.

# ranger model
r_fit <- ranger(Surv(time, status) ~ trt + celltype + 
                     karno + diagtime + age + prior,
                     data = vet,
                     mtry = 4,
                     importance = "permutation",
                     splitrule = "extratrees",
                     verbose = TRUE)

# Average the survival models
death_times <- r_fit$unique.death.times 
surv_prob <- data.frame(r_fit$survival)
avg_prob <- sapply(surv_prob,mean)

# Plot the survival models for each patient
plot(r_fit$unique.death.times,r_fit$survival[1,], 
     type = "l", 
     ylim = c(0,1),
     col = "red",
     xlab = "Days",
     ylab = "survival",
     main = "Patient Survival Curves")

#
cols <- colors()
for (n in sample(c(2:dim(vet)[1]), 20)){
  lines(r_fit$unique.death.times, r_fit$survival[n,], type = "l", col = cols[n])
}
lines(death_times, avg_prob, lwd = 2)
legend(500, 0.7, legend = c('Average = black'))

The next block of code illustrates how ranger() ranks variable importance.

vi <- data.frame(sort(round(r_fit$variable.importance, 4), decreasing = TRUE))
names(vi) <- "importance"
head(vi)

##          importance
## karno        0.0903
## celltype     0.0323
## diagtime    -0.0012
## trt         -0.0013
## prior       -0.0027
## age         -0.0037

Notice that ranger() flags karno and celltype as the two most important; the same variables with the smallest p-values in the Cox model. Also note that the importance results just give variable names and not level names. This is because ranger and other tree models do not usually create dummy variables.

But ranger() does compute Harrell’s c-index (See [8] p. 370 for the definition), which is similar to the Concordance statistic described above. This is a generalization of the ROC curve, which reduces to the Wilcoxon-Mann-Whitney statistic for binary variables, which in turn, is equivalent to computing the area under the ROC curve.

cat("Prediction Error = 1 - Harrell's c-index = ", r_fit$prediction.error)

## Prediction Error = 1 - Harrell's c-index =  0.3087233

An ROC value of .68 would normally be pretty good for a first try. But note that the ranger model doesn’t do anything to address the time varying coefficients. This apparently is a challenge. In a 2011 paper [16], Hamad observes:

However, in the context of survival trees, a further difficulty arises when time–varying effects are included. Hence, we feel that the interpretation of covariate effects with tree ensembles in general is still mainly unsolved and should attract future research.

I believe that the major use for tree-based models for survival data will be to deal with very large data sets.

Finally, to provide an “eyeball comparison” of the three survival curves, I’ll plot them on the same graph.The following code pulls out the survival data from the three model objects and puts them into a data frame for ggplot().

# Set up for ggplot
kmi <- rep("KM",length(km_fit$time))
km_df <- data.frame(km_fit$time,km_fit$surv,kmi)
names(km_df) <- c("Time","Surv","Model")

coxi <- rep("Cox",length(cox_fit$time))
cox_df <- data.frame(cox_fit$time,cox_fit$surv,coxi)
names(cox_df) <- c("Time","Surv","Model")

rfi <- rep("RF",length(r_fit$unique.death.times))
rf_df <- data.frame(r_fit$unique.death.times,avg_prob,rfi)
names(rf_df) <- c("Time","Surv","Model")

plot_df <- rbind(km_df,cox_df,rf_df)

p <- ggplot(plot_df, aes(x = Time, y = Surv, color = Model))
p + geom_line()

For this data set, I would put my money on a carefully constructed Cox model that takes into account the time varying coefficients. I suspect that there are neither enough observations nor enough explanatory variables for the ranger() model to do better.

This four-package excursion only hints at the Survival Analysis tools that are available in R, but it does illustrate some of the richness of the R platform, which has been under continuous development and improvement for nearly twenty years. The ranger package, which suggests the survival package, and ggfortify, which depends on ggplot2 and also suggests the survival package, illustrate how open-source code allows developers to build on the work of their predecessors. The documentation that accompanies the survival package, the numerous online resources, and the statistics such as concordance and Harrell’s c-index packed into the objects produced by fitting the models gives some idea of the statistical depth that underlies almost everything R.

Some Tutorials and Papers

For a very nice, basic tutorial on survival analysis, have a look at the Survival Analysis in R [5] and the OIsurv package produced by the folks at OpenIntro.

Look here for an exposition of the Cox Proportional Hazard’s Model, and here [11] for an introduction to Aalen’s Additive Regression Model.

For an elementary treatment of evaluating the proportional hazards assumption that uses the veterans data set, see the text by Kleinbaum and Klein [13].

For an exposition of the sort of predictive survival analysis modeling that can be done with ranger, be sure to have a look at Manuel Amunategui’s post and video.

See the 1995 paper [15] by Intrator and Kooperberg for an early review of using classification and regression trees to study survival data.

References

For convenience, I have collected the references used throughout the post here.

[1] Hacking, Ian. (2006) The Emergence of Probability: A Philosophical Study of Early Ideas about Probability Induction and Statistical Inference. Cambridge University Press, 2nd ed., p. 11
[2] Andersen, P.K., Keiding, N. (1998) Survival analysis Encyclopedia of Biostatistics 6. Wiley, pp. 4452-4461 [3] Kaplan, E.L. & Meier, P. (1958). Non-parametric estimation from incomplete observations, J American Stats Assn. 53, pp. 457–481, 562–563. [4] Cox, D.R. (1972). Regression models and life-tables (with discussion), Journal of the Royal Statistical Society (B) 34, pp. 187–220.
[5] Diez, David. Survival Analysis in R, OpenIntro
[6] Klein, John P and Moeschberger, Melvin L. Survival Analysis Techniques for Censored and Truncated Data, Springer. (1997)
[7] Wright, Marvin & Ziegler, Andreas. (2017) ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R, JSS Vol 77, Issue 1.
[8] Harrell, Frank, Lee, Kerry & Mark, Daniel. Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Statistics in Medicine, Vol 15 (1996), pp. 361-387 [9] Amunategui, Manuel. Survival Ensembles: Survival Plus Classification for Improved Time-Based Predictions in R
[10] NUS Course Notes. Chapter 3 The Cox Proportional Hazards Model
[11] Encyclopedia of Biostatistics, 2nd Edition (2005). Aalen’s Additive Regression Model [12] Therneau et al. Using Time Dependent Covariates and Time Dependent Coefficients in the Cox Model
[13] Kleinbaum, D.G. and Klein, M. Survival Analysis, A Self Learning Text Springer (2005) [14] Therneau, T and Atkinson, E. An Introduction to Recursive Partitioning Using RPART Routines
[15] Intrator, O. and Kooperberg, C. Trees and splines in survival analysis Statistical Methods in Medical Research (1995)
[16] Bou-Hamad, I. A review of survival trees Statistics Surveys Vol.5 (2011)

Authors’s note: this post was originally published on April 26, 2017 but was subsequently withdrawn because of an error spotted by Dr. Terry Therneau. He observed that the Cox Portional Hazards Model fitted in that post did not properly account for the time varying covariates. This revised post makes use of a different data set, and points to resources for addressing time varying covariates. Many thanks to Dr. Therneau. Any errors that remain are mine.

Writing and Publishing my first R package

Tue, 05 Sep 2017 00:00:00 +0000

Inspired by the Community

One of the themes at useR 2017 in Brussels was “Get involved”. People were encouraged to contribute to the community, even when they did not consider themselves R specialists (yet). This could be by writing a package or a blog post, but also by simply correcting typos through pull requests, or sending a tweet about a successful analysis. Bottom line: get your stuff out in the open. Share your work!

I felt this urge of getting involved already a year ago, at the useR conference in 2016. Hearing all these people speak about the great work they had done was really inspiring, and I wanted to be a part of it. I wanted to find out what it was like to develop a package. I knew that what I could do was only minor; I have a full-time job as a data scientist of which contributing to the community is not part of the job description. However, I could free up one day a week by working a few hours more on the other days of the week and I could do a little on the train to work every day. I was not sure if I was up to developing a full R package, but I could at least try.

Writing `padr`

The idea for the padr package came at work, where I was dealing with time-stamped data a lot. Aggregating data to a higher level and filling missing values took me a quite some time each time I did it. I thought there must be a better way for doing this. I did not have a clear end product in mind. I just started trying stuff and stitched small ideas together to do more complex operations. In the meantime, I worked my way through Hadley’s R Packages, and learned about the fundamentals of software design in R, picking up skills while I went along.

This past January, I published the first version of padr on CRAN, and I must say that it was quite scary. Putting your product out in the open comes with a lot of impostor feelings. I had thoroughly tested the functions, but still was afraid all kinds of bugs would pop up. Well, bugs did pop up and this appeared to be just fine. I had very nice comments by people suggesting improvements or who even fixed bugs themselves. If there is one lesson I learned from writing a package, it is that software does not have to be perfect to be useful. Of course, you don’t bring junk that is full of errors to CRAN, but once you have tried your best, you should just get it in the open and wait for feedback. This would improve the software much faster than spending endless evenings trying to find bugs.

How `padr` works

The central concept for the padr package is that time-stamped data are associated with an interval. This is the heartbeat of the data. Observed data points are separated by multiples of the interval. But, there can be observations missing. We compute the largest, evenly spaced grid that fits all our observations. See the following example.

library(padr)
library(dplyr)
as.Date(c("2017-01-01","2017-01-03","2017-01-07")) %>% get_interval()

## [1] "2 day"

Now, the first main thing that padr does is thicken() the datetime variable in a data frame. It adds a variable of a higher interval to the data frame. We can subsequently aggregate to this higher interval with dplyr. We use the coffee data set, that comes with padr as an example. This contains four hypothetical purchases in a coffee shop.

coffee

##            time_stamp amount
## 1 2016-07-07 00:11:21   3.14
## 2 2016-07-07 00:46:48   2.98
## 3 2016-07-09 04:25:17   4.11
## 4 2016-07-10 01:45:11   3.14

Now, to get the total amount spent per day, we run the following code.

(coffee_day <- coffee %>% 
  thicken("day") %>% 
  group_by(time_stamp_day) %>% 
  summarise(day_amnt = sum(amount)))

## # A tibble: 3 x 2
##   time_stamp_day day_amnt
##           <date>    <dbl>
## 1     2016-07-07     6.12
## 2     2016-07-09     4.11
## 3     2016-07-10     3.14

You will notice that on 2016-07-08 there was no visit to the store. However, this is implicit in our data. For visualisation or time-series analysis, it is useful to have a row that explicitly states there is no money spent on this day. Here comes pad() into play. It detects the interval of the datetime variable and inserts a record.

(coffee_day_padded <- 
  coffee_day %>% 
  pad())

## # A tibble: 4 x 2
##   time_stamp_day day_amnt
##           <date>    <dbl>
## 1     2016-07-07     6.12
## 2     2016-07-08       NA
## 3     2016-07-09     4.11
## 4     2016-07-10     3.14

Finally, we need to fill the missing value in day_amnt with 0.

coffee_day_padded %>% fill_by_value()

## # A tibble: 4 x 2
##   time_stamp_day day_amnt
##           <date>    <dbl>
## 1     2016-07-07     6.12
## 2     2016-07-08     0.00
## 3     2016-07-09     4.11
## 4     2016-07-10     3.14

Lessons Learned

I benefit greatly from the skills I picked up in the process in my day job. My code in data science projects is cleaner now. I try to write as many functions as I can. Data scientists that have no background in computer science, like myself, can become better analysts by acquainting themselves with principles of software design. R’s package structure is not only great for building packages. You can also benefit from it when building software tools for specific data analyses, making your work more reproducible and shareable over projects.

There is still a lot of stuff for me to learn about writing better software and gaining a deeper understanding of the language. However, I no longer think this is intimidating. Rather, it is a great opportunity to improve and improve the software I am writing. You don’t need complete mastery to start developing. On the contrary, mastery seems to come through developing. You only truly learn R by trying things you have not tried before.

July 2017 New Package Picks

Mon, 28 Aug 2017 00:00:00 +0000

Two hundred and twenty-four new packages were added to CRAN in July. Below are my picks for the “Top 40” packages arranged in eight categories: Machine Learning, Science, Statistics, Numerical Methods, Statistics, Time Series, Utilities and Visualizations. Science and Numerical Methods are categories that I have not used before. The idea behind the Science category is to find a place for packages that appear to have been created with some particular scientific investigation or problem in mind. The Numerical Methods category is reserved for packages that, while they may be targeted to some general form of statistical analysis, emphasize numerical considerations and carefully constructed algorithms.

As always, my selections are heavily weighted by the availability of documentation beyond what is included in the package PDF. I rarely select packages that do not have a vignette or some other source of documentation about how the package can be used, for example, README files or a referenced URL. I almost never select “professional” packages, which I define as packages that are devoted to esoteric topics that either include no documentation beyond the PDF, or exclusively refer to papers that are protected by a paywall. While these packages usually comprise serious, valuable contributions to R, they also appear to have been written for very small audiences.

Finally, before listing this month’s Top 40, I would like to call attention to an awesome display of productivity by Kevin R. Coombes, who had fourteen packages on various topics accepted by CRAN in July: BimodalIndex, ClassComparison, ClassDiscovery, CrossValidate, GeneAlgo, IntegIRTy, Modeler, NameNeedle, oompaBase, oompaData, PreProcess, SIBERG, TailRank and Umpire.

The July 2017 Top 40

Machine Learning

autoBagging v0.1.0: Implements a framework for automated machine learning that focuses on the optimization of bagging workflows. See the paper by Pinto et al. for details.

grf v0.9.3: Provides methods for non-parametric least-squares regression, quantile regression, and treatment effect estimation (optionally using instrumental variables).

iRF v2.0.0: Provides functions to iteratively grow feature-weighted random forests and finds high-order feature interactions in a stable fashion. Look here for the details.

keras v2.0.5: Implements an interface to Keras, a high-level neural networks API that runs on top of TensorFlow. There is an Overview of the Keras backend, and a number of vignettes including Keras Layers, Writing Custom Keras Layers, Keras Models, Using Pre-Trained Models, Sequential Models and more.

randomForestExplainer v0.9: Provide set of tools to help explain which variables are most important in a random forests. The vignette provides examples of visualizing multiple performance measures.

sgmcmc v0.1.0: Provides functions to implement stochastic gradient Markov chain Monte Carlo (SGMCMC) methods for user-specified models. TensorFlow is used to calculate the gradients. There is a Getting Started Guide and vignettes for Simulating from a Gaussian Mixture, a Multivariate Gaussian Mixture and Logistic Regression.

Numerical Methods

mcMST v1.0.0: Provides algorithms to approximate the Pareto-front of multi-criteria minimum spanning tree problems, along with a toolbox for generating multi-objective benchmark graph problems. There is an Introduction and a vignette on benchmarking optimization problems.

mize v0.1.1: Provides optimization algorithms, including conjugate gradient (CG), Broyden-Fletcher-Goldfarb-Shanno (BFGS), and the limited memory BFGS (L-BFGS) methods. There is an introduction and vignettes on Convergence, Metric MDS, and Stateful Optimization.

SuperGauss v1.0: Provides a fast C++ based algorithm for the evaluation of Gaussian time series, along with efficient implementations of the score and Hessian functions. The vignette shows an example of inference for the Hurst parameter.

Science

GROAN v1.0.0: Is a workbench for testing genomic regression accuracy on noisy phenotypes. It contains a noise-generator function. There is a vignette on Genomic Regression in Noisy Scenarios.

noaastormevents v0.1.0: Allows users to explore and plot data from the National Oceanic and Atmospheric Administration (NOAA) Storm Events database for United States counties through R. There is an Overview and a vignette providing details.

swmmr v0.7.0: Provides functions to connect the Storm Water Management Model (SWMM) of the United States Environmental Protection Agency (US EPA).

Statistics

blandr v0.4.3: Contains functions to carry out Bland Altman analyses (also known as a Tukey mean-difference plot) as described by JM Bland and DG Altman. See the vignette for details.

cnbdistr v1.0.1: Provides the distribution functions for the Conditional Negative Binomial distribution. The vignette shows the math.

diffpriv v0.4.2: Provides an implementation of major general-purpose mechanisms for privatizing statistics, models, and machine learners, within the framework of differential privacy of Dwork et al. (2006). There is a vignette on The Bernstein Mechanism and an Introduction

fence v1.0: Implements a new class of model-selection strategies for mixed-model selection, which includes linear and generalized linear mixed models. The idea involves a procedure to isolate a subgroup of what are known as correct models (of which the optimal model is a member). The package points to several references in the literature including papers by Jiang et al. 2008, Jiang et al. 2010, Jiang et al. 2011, Nguyen et al. 2012, and Jiang 2014.

llogistic v1.0.0: Provides density, distribution, quantile and random generation functions for the L-Logistic distribution with parameters m (median) and b.

metaBMA v0.3.9: Provides functions to compute the posterior model probabilities the meta-analysis models assuming either fixed or random effects. See the paper by Gronau et al. and the vignette for details.

MFKnockoffs v0.9: Provides functions to create model-free knockoffs, a general procedure for controlling the false discovery rate FDR when performing variable selection. There are vignettes on using the the Model-Free Knockoff Filter Basic and Advanced, and Using the Filter with a Fixed Design Matrix.

Modeler v3.4.2: Provides tools to define classes and methods to learn models and use them to predict binary outcomes. There is a vignette on Learning and Predicting.

msde v1.0: Implements an MCMC sampler for the posterior distribution of arbitrary, time-homogeneous, multivariate stochastic differential equation (SDE) models with possibly latent components. There is a vignette with Sample Models and another for Inference.

RBesT v1.2-3: Provides a tool set to support Bayesian evidence synthesis, including meta-analysis, prior derivation from historical data, operating characteristics, and analysis. There is an Introduction and vignettes on Customizing Plots, Normal Endpoints, and Robust Priors.

RcppTN v0.2-1: Provides R and C++ functions to generate random deviates from and calculate moments of a Truncated Normal distribution using the algorithm of Robert (1995). There is a vignette showing how to use the package, and one for Performance.

rsample v0.0.1: Provides classes and functions to create and summarize different kinds of resampling objects. There is a vignette on the Basics, and another for Working with rsets.

SMM v1.0: Provides functions to simulate and estimate of Multi-State Discrete-Time Semi-Markov and Markov Models. The implementation details are described in two papers by Barbu, Limnios one and two, and one paper by Trevezas and Limnios. The vignette also provides considerable detail.

treeDA v0.02: Provides functions to perform sparse discriminant analysis on a combination of node and leaf predictors, when the predictor variables are structured according to a tree. There is a vignette.

vennLasso v0.1: Provides variable selection and estimation routines for models stratified by binary factors. There is a vignette.

Time Series

sweep v0.2.0: Provides tools for bringing tidyverse organization to time series forecasting. There is an Introduction, as well as vignettes on Forecasting and Forecasting with Mutiple Models.

timetk v0.1.0: Implements a toolkit for working with time series, including functions to interrogate time series objects and tibbles, and coerce between time-based tibbles (‘tbl’) and ‘xts’, ‘zoo’, and ‘ts’. There is an Introduction and vignettes on Working with time series index, Making a Future Index, and Forecasting.

Utilities

dataCompareR v0.1.0: Contains functions to compare two tabular data objects with the specific intent of showing differences in a way that should make it easier to understand the differences. The vignette shows how to use the package.

dataPreparation v0.2: Provides functions for data preparation that take advantage of data.table efficiencies. There is a tutorial.

datastructures v0.2.0: Provides implementations of advanced data structures such as hashmaps, heaps, and queues. There is a Tutorial.

docker v0.0.2: Provides access to the Docker SDK from R via python, using the reticulate package. There is a vignette to get you started.

seplyr v0.1.4: Supplies standard evaluation adapter methods for important common dplyr methods that currently have a non-standard programming interface. There is an Introduction, as well as vignettes for Using seplyr with dplyr, the operator named map builder, and the operator rename_se.

vetr v0.1.0: Provides a declarative template-based framework for verifying that objects meet structural requirements, and auto-composing error messages when they do not. There is a vignette on Alikeness and one on Trust, but Verify.

Visualization

ggjoy v0.3.0: Joyplots provide a convenient way of visualizing changes in distributions over time or space. ggjoy enables the creation of such plots in ggplot2. There is an Introduction and a Gallery of examples.

ggplotgui v1.0.0: Implements a shiny app for creating and exploring ggplot2 graphs that also generates the required R code. Look here for examples.

loon v1.1.0: Is an extensible toolkit for interactive data visualization and exploration. There are two vignettes containing examples: Visible minorities in Canadian cities and Smoothers and Bone Mineral Density

sugrrants v0.1.0: Provides ggplot2 graphics for analyzing time series data with the goal of fitting into the tidyverse and grammar of graphics framework. The vignette provides examples.

tidygraph v1.0.0: A graph, while not “tidy” in itself, can be thought of as two tidy data frames describing node and edge data respectively. tidygraph provides functions to manipulate these virtual data frames using the dplyr package. Look here for some details.

visdat v0.1.0: provides functions to create preliminary exploratory data visualizations of an entire dataset using ggplot2. There is a vignette to get you started.

Control Systems Toolbox – System Interconnection

Thu, 24 Aug 2017 00:00:00 +0000

Introduction

Dynamic systems are usually represented by a model before they can be analyzed computationally. These dynamic systems are systems that change, evolve or have their states altered or varied with time based on a set of defined rules. Dynamic systems could be mechanical, electrical, electronic, biological, sociological, and so on. Many such systems are usually defined by a set rules that are represented as a set of nonlinear differential equations. A generic first-order differential form is as shown below: \[ \frac{dx}{dt} = f ( x(t), u(t), t ) ........... (1) \]

where x(t) is the state vector representing the configuration (or state) of the system at the time, t, u(t) represents the external control input at time, t, and f is a function that gives the rate of change of the state vector for a specific input, state and time.

The system of (1) could be said to be time invariant if we assume that f does not explicitly depend on time. This implies that the relation of (1) becomes:

\[ \frac{dx}{dt} = f (x, u) ........... (2) \]

For such time-invariant systems, the control input may remain time-variant, but the coefficients of the function f are constant since the underlying physical laws themselves are not usually time-dependent. Over a sufficiently small operating range, the dynamics of most systems are approximately linear, and the following relation entails a linear time-invariant (LTI) system:

\[ \frac{dx}{dt} = Ax + Bu ........... (3) \]

Several engineering challenges till today are being solved using LTI techniques, while most results obtained from control theory are based on the described assumptions associated with LTI systems.

Laplace Transform

For those systems that are described by differential equations, it is possible to convert the system’s time-domain model into a frequency-domain (s-domain) representation. Two striking benefits for doing this are:

the differential equations that define the system are transformed into algebraic equations, which are often easier to analyze
The frequency response of the system can be obtained

This frequency-domain output / input representation is called the transfer function. Many control system models are available in transfer function representation, and this model is mostly used in the block diagram representation of control systems.

Figure 1 shows the block diagram for the pitch-control system of an Unmanned Free Swimming Submersible (UFSS) vehicle. This system and its block diagram shall be detailed in the Case Study section of this article.

Transfer Function Expressions

A transfer function expression is an algebraic expression in the frequency-domain. It is usually the ratio of the output to the input of a control system. Evaluating such expressions while retaining their mathematical form (as in symbolic mathematics) should be possible using a scientific computing language. An example of such expression is shown below:

\[ G(s) = \frac{Output}{Input} = \frac{s^2 + 3s + 4}{2s^3 + 4s^2 + 5s + 1} \]

Models of physical systems often appear as transfer functions with the coefficients of state as electrical, mechanical, chemical, or other physical quantities. Users would find it convenient to simply input these transfer functions as-is into a computer, and quickly analyze the system.

In the following mechanical example of a mass-spring-damper system of [1], the transfer function contains symbols m, b and k, which represent mass, damping constant, and spring constant variables respectively, as shown below:

Figure 2: Mass-Spring-Damper diagram \[ \frac{X(s)}{F(s)} = \frac{1}{ms^2+bs+k} \]

where m = 1; k = 1; b = 0.2

In this Google Summer of Code (GSoC) 2017 project, we have tried to include such capability within the control library, making it possible to create a transfer function model from the evaluation of a rational expression (the transfer function). A function named TF() was created to accept transfer function expressions, and return a transfer function model after evaluation. Therefore, to create this mass-spring-damper model within R, use the control library as shown in the snippet below:

 m <- 1;  k <- 1;  b <- 0.2
TF("1/(m*s^2 + b *s + k)")

## 
## y1:
##                 1 
##   - - - - - - - - - - -
##      s^2 + 0.2 s + 1 
## 
## 
## Transfer Function: Continuous time model

Electric Motor Example

Consider the following example from a University of Michigan course page on Control Systems using MATLAB – an example of an electric motor [2]

Figure 3: showing electric motor physical model [2]

Wikipedia: A DC motor is any class of rotary electric machines that converts direct current electrical energy into mechanical energy. It can provide translational motion when coupled with wheel, drums and cables. DC motors are used in propulsion of electric vehicles, elevator and hoists, or in drives for steel rolling mills. Several applications are: air compressors, vacuum cleaner, hair drier, sewing machine, and even toys or any system that requires variable speed.

The physical parameters for this example are [3]:

Moment of inertia of the rotor, J = 3.2284E-6 kg.m^2

Motor viscous friction constant, b = 3.5077E-6 N.m.s

Electromotive force constant, Ke = 0.0274 V/rad/sec

Motor torque constant, Kt = 0.0274 N.m/Amp

Electric resistance, R = 4 Ohm

Electric inductance, L = 2.75E-6 H

where, Kt = Ke = K

The input of the system is the voltage source (V) applied to the motor’s armature, while the output is the rotational speed of the shaft.

Open-Loop Response of the electric motor

The open-loop transfer function for the DC motor described above in the s-domain is:

\[ P(s) = \frac{K}{(s(Js + b)(Ls + R) + K^2)} \]

The step function is usually used to analyze the system’s step response, so it is applied here to analyze the open-loop response of the system. Using the control library, this model in the s-domain could be created and analyzed in R as follows:

 J <- 3.2284E-6;
b <- 3.5077E-6;
K <- 0.0274;
R <- 4;
L <- 2.75E-6;
P_motor <- TF("K/(s*((J*s + b)*(L*s + R) + K^2))")
time <- seq(0,0.2,0.001)
stepplot(P_motor, t = time)

Figure 1: Figure 4: Electric Motor Open-loop Step Response

From the above plot showing the system’s step response, it is clear that when a step input (of 1 volt) is applied to the system, the electric motor’s position grows unbounded - system instability.

The stability of the system can also be determined from the poles of the transfer function using the pole function:

pole(P_motor)

##               [,1]
## [1,]  0.000000e+00
## [2,] -1.454487e+06
## [3,] -5.922604e+01

As shown above, one of the poles of the open-loop transfer function is on the imaginary axis while the other two poles are in the left half of the complex s-plane. A pole on the imaginary axis indicates that the free response of the system will not only grow unbounded, but also will not decay to zero [3].

Closed-Loop Response of the electric motor

We can add feedback the motor model with a controller set to 1, and then investigate the system’s closed-loop step response for stability. This can be obtained using the following R commands:

P_motor2 <- feedback(P_motor, 1)
print(P_motor2)

## 
## y1:
##                                                     0.0274 
##   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
##     8.8781e-12 s^3 + 1.291361e-05 s^2 + 0.0007647908 s + 0.0274 
## 
## 
## Transfer Function: Continuous time model

# OR

P_motor2 <- cloop(P_motor, -1)
print(P_motor2)

## 
## y1:
##                                                     0.0274 
##   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
##     8.8781e-12 s^3 + 1.291361e-05 s^2 + 0.0007647908 s + 0.0274 
## 
## 
## Transfer Function: Continuous time model

stepplot(P_motor2, t = time)

Figure 2: Figure 5: Electric Motor Closed-loop Step Response

The above closed-loop step response shows that the addition of feedback has stabilized the system.

The damping and natural frequencies associated with the poles of this closed-loop system could be obtained using the following damp function:

damp(P_motor2)

## 
##      Eigenvalue            Damping     Freq. (rad/s)    Freq. (Hz). 
##      ----------            -------     -------------    -----------
## -29.61     +     35.28j   0.64           46.06           0.46      
## -29.61     -     35.28j   0.64           46.06           0.46      
## -1454487.32 +      0.00j   1.00           1454487.32      0.00

## $omega
##              [,1]
## [1,] 4.606386e+01
## [2,] 4.606386e+01
## [3,] 1.454487e+06
## 
## $zeta
## [1] 0.642853 0.642853 1.000000

Furthermore, using the TF() function, transfer function models could be summed, subtracted, multiplied, and divided. This is exemplified in the following section.

System Interconnections

In many cases, systems or subsystems have to be interconnected to obtain a particular controlled or regulated activity. Interconnecting models of components allows you to easily construct large control system models.

Analyzing systems that are modeled in a block diagram should be easily done using a scientific computing tool like R. The time response or frequency response of a system (such as shown in Figure 1), can be obtained after the block diagram is simplified. Simplification means that the several components (blocks described by different transfer functions) that make up the complete system have to be algebraically reduced to one transfer function to represent the system. This mathematical reduction is done based on the rules of control theory.

Figure 6: Block Diagram Interconnection

For an example, Figure 6 shows the interconnection of dynamic system models of a plant P(s), a controller C(s), sensor dynamics S(s), and a filter F(s). To analyze the system requires constructing a single (equivalent) model that represents the entire closed-loop control system.

In the second coding phase of the GSoC 2017, the following model interconnection functions have been produced as shown in Figure 7 below:

Figure 7: System Model Interconnections

Append Connection using the `append` function

A number of systems, whether in transfer-function, state-space or zero-pole model, could be appended into one state-space model using the append function. The block diagram arrangement is found in Figure 7 (e).

The syntax is: append(sys1, sys2, …, sysN). The resulting model is in state-space form.

For example:

 sys1 <- ss(1,2,3,4)
 sys2 <- ss(2,3,4,5)
 sys3 <- ss(6,7,8,9)
 sys4 <- tf(1, c(1,2,5))
 append(sys1, sys2, sys3)
 append(sys1, sys2, sys4, sys3)

Series Connection using the `series` function

This function connects two systems in series, in transfer-function, state-space or zero-pole model, into one model. The block diagram arrangement is found in Figure 7 (a).

Syntax: series(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1*sys2”)

For now, TF(“sys1*sys2”) is only possible when sys1 and sys2 are transfer-function models

Parallel Connection using the `parallel` function

The parallel function connects two systems in parallel, in transfer-function, state-space or zero-pole model, into one model. The block diagram arrangement is found in Figure 7 (c).

Syntax: parallel(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1 + sys2”)

For now, TF(“sys1+ sys2”) is only possible when sys1 and sys2 are transfer-function models

Feedback Connection using the `feedback` function

This function sets up a closed-loop transfer function from output to input based on the standard configuration shown in Figure 7 (b).

Syntax: feedback(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1 / (1 + sys2*sys1)”), but this is not recommended in terms of accuracy. feedback(sys1, sys2) is preferred.

General block diagram building using `connect` function

Using the connect function, more complicated (especially multiple-input multiple-output (MIMO) ) block diagrams could be set up. For this case, all systems that are to be connected are first appended, then connected using their inputs and outputs.

Syntax: connect(append(sys1, sys2, sys3), connections, inputs, outputs)

Case Study – Unmanned Free Swimming Submersible (UFSS) Vehicle

The Unmanned Free-Swimming Submersible (UFSS) vehicle was one of the first autonomous underwater vehicles (AUVs). It was developed in the late 1970s as a test vehicle to demonstrate the potential of autonomous vehicles over long distances, and to study laminar flow [5]. A picture showing an Unmanned Free-Swimming Submersible (UFSS) vehicle is shown in Figure 8 below.

Figure 8: Unmanned Free Swimming Submersible Vehicle (UFSS). Source: [5]

Nise [4] relates the following concerning the UFSS:

The depth of the vehicle is controlled as follows. During forward motion, an elevator surface on the vehicle is deflected by a selected amount. This deflection causes the vehicle to rotate about the pitch axis. The pitch of the vehicle creates a vertical force that causes the vehicle to submerge or rise.

A detailed block diagram for the pitch control system is shown in Figure 9 below.

Figure 9: Block diagram for the pitch control system of UFSS

Rationale:

The subsystems of the UFSS vehicle are developed independently, and studied using computer simulation before integration into the vehicle [5]. Thus, for this reason, we are examining the pitch control system independently.

For this case study, we shall use the developing control library in R to:

Obtain the closed-loop UFSS pitch control system by block diagram reduction of the system in Figure 9 above, using series, parallel and feedback.
Obtain the step response of the pitch control system simplified by series, parallel and feedback
Obtain the closed-loop UFSS pitch control system by block diagram reduction of the system in Figure 9 above, using the algebraic operations provided by TF().
Obtain the step response of the pitch control system from the method of algebraic operations.

Solution using `series`, `parallel` and `feedback`

K1 <- 1
K2  <- 1
G1 <- tf(-K1,1)
G2 <- tf(c(0,2), c(1,2))

numg3 <- -0.125*c(1, 0.435)
deng3 <- pracma::polymul(c(1, 1.23), c(1, 0.226, 0.0169))
G3 <- tf(numg3, deng3)

H1 <- tf(c(-K2, 0), c(0, 1))

G4 <- series(G2, G3)
G5 <- feedback(G4, H1)
Ge <- series(G1, G5)
T1 <- feedback(Ge, 1)
stepplot(T1, t = seq(0,40,0.1))

Figure 3: Figure 10: R plot showing UFSS vehicle system response - Block diagram reduction by series, parallel and feedback

Solution via algebraic operations

G4 <- TF("G3*G2")
G5 <- TF("G4/(1+ G4*H1)")
Ge <- TF("G5*G1")
T2 <- TF("Ge/(1+Ge)")
stepplot(T2, t = seq(0,40,0.1))

Figure 4: Figure 11: R plot showing UFSS vehicle system response - Block diagram reduction by Algebraic Operations

Clearly, the step responses (Figure 10 and Figure 11) of the pitch control system using the two methods are identical to each other - a justification to the framework of the control systems toolbox.

Feedback

Interested users are welcome to install the development version of this package using devtools::install_github("benubah/control"), and feel free to get in touch for any related issues.

Conclusion

In this article, we have discussed on our progress with the Control Systems Toolbox in R as a GSoC 2017 project. We have highlighted the use of this toolbox to carry out the following tasks:

Create Transfer function models by evaluating transfer function expressions in the s-domain
Obtain the poles and their associated damping and natural frequencies of a system
Open-loop and closed-loop responses
System Model Interconnections using series, parallel and feedback and algebraic functions in the UFSS vehicle case study.

Other functions that have been developed in this GSoC phase are: givens_rot, ordschur, care, ctrb, acker, place.

You may want to read about what was achieved in the first GSoC coding phase here.

References

http://ctms.engin.umich.edu/CTMS/index.php?example=Introduction&section=SystemModeling
http://ctms.engin.umich.edu/CTMS/index.php?example=MotorSpeed&section=SystemModeling
http://ctms.engin.umich.edu/CTMS/index.php?example=MotorPosition&section=SystemAnalysis
Norman, S. Nise, Control Systems Engineering, 6th. Ed, 2011.
http://www.navalunderseamuseum.org/ufss/

Learning things we already know about stocks

Tue, 22 Aug 2017 00:00:00 +0000

This example groups stocks together in a network that highlights associations within and between the groups using only historical price data. The result is far from ground-breaking: you can already guess the output. For the most part, the stocks get grouped together into pretty obvious business sectors.

Despite the obvious result, the process of teasing out latent groupings from historic price data is interesting. That’s the focus of this example. A central idea of the approach taken here comes from the great paper of Ledoit and Wolf, “Honey, I Shrunk the Sample Covariance Matrix” (http://www.ledoit.net/honey.pdf). This example employs an alternative approach based on a matrix eigenvalue decomposition, but it’s the same general idea.

This note follows an informal, how-to format. Rather than focus on mathematical analysis, which is well-detailed in the references, I try to spell out the how’s and why’s: how to do things step by step (using R) and a somewhat non-rigorous rationale for each step that’s hopefully at least convincing and intuitive.

For emphasis, allow me to restate the first sentence as an objective:

Group stocks together in a network that highlights associations within and between the groups using only historical price data.

That’s what the rest of this example will do, hopefully illuminating some key ideas about regularization along the way.

Software used in the example

The example uses R of course, and the following R packages, all available on CRAN (some of the packages themselves have dependencies):

quantmod (at least version 0.4-10)
igraph (at least version 1.1.2)
threejs (at least version 0.3.1)

Getting data

NOTE: You can skip ahead to the Sample correlation section by simply downloading a sample copy of processed log(return) data as follows:

library(quantmod)
load(url("http://illposed.net/logreturns.rdata"))

Otherwise, follow the next two sections to download the raw stock daily price data and process those data into log(returns).

Download daily closing price data from Google Finance

The quantmod package (Ulrich and Ryan, http://www.quantmod.com/) makes it ridiculously easy to download (and visualize) financial time series data. The following code uses quantmod to download daily stock price data for about 100 companies with the largest market capitalizations listed on the Standard & Poor’s 500 index at the time of this writing. The code downloads daily closing prices from 2012 until the present. Modify the code to experiment with different time periods or stocks as desired!

Because stock symbol names may change and companies my come and go, it’s possible that some of the data for some time periods are not available. The tryCatch() block in the code checks for a download error and flags problems by returning NA, later removed from the result. The upshot is that the output number of columns of stock price time series may be smaller than the input list of stock symbols.

The output of the following code is an xts time series matrix of stock prices called prices whose rows correspond to days and columns to stock symbols.

library(quantmod)
from="2012-05-17"
sym = c("AAPL", "ABBV", "ABT", "ACN", "AGN", "AIG", "ALL", "AMGN", "AMZN", "AXP",
        "BA", "BAC", "BIIB", "BK", "BLK", "BMY", "BRK.B", "C", "CAT", "CELG", "CL",
        "CMCSA", "COF", "COP", "COST", "CSCO", "CVS", "CVX", "DD", "DHR", "DIS", "DOW",
        "DUK", "EMR", "EXC", "F", "FB", "FDX", "FOX", "FOXA", "GD", "GE", "GILD", "GM",
        "GOOG", "GOOGL", "GS", "HAL", "HD", "HON", "IBM", "INTC", "JNJ", "JPM", "KHC",
        "KMI", "KO", "LLY", "LMT", "LOW", "MA", "MCD", "MDLZ", "MDT", "MET", "MMM",
        "MO", "MON", "MRK", "MS", "MSFT", "NEE", "NKE", "ORCL", "OXY", "PCLN", "PEP",
        "PFE", "PG", "PM", "PYPL", "QCOM", "RTN", "SBUX", "SLB", "SO", "SPG", "T",
        "TGT", "TWX", "TXN", "UNH", "UNP", "UPS", "USB", "UTX", "V", "VZ", "WBA",
        "WFC", "WMT", "XOM")

prices = Map(function(n)
             {
               print(n)
               tryCatch(getSymbols(n, src="google", env=NULL, from=from)[, 4], error = function(e) NA)
             }, sym)
N = length(prices)
# identify symbols returning valid data
i = ! unlist(Map(function(i) is.na(prices[i]), seq(N)))
# combine returned prices list into a matrix, one column for each symbol with valid data
prices = Reduce(cbind, prices[i])
colnames(prices) = sym[i]

Clean up and transform data

Not every stock symbol may have prices available for every day. Trading can be suspended for some reason, companies get acquired or go private, new companies form, etc.

Let’s fill in missing values going forward in time using the last reported price (piecewise constant interpolation)–a reasonable approach for stock price time series. After that, if there are still missing values, just remove those symbols that contain them, possibly further reducing the universe of stock symbols we’re working with.

for(j in 1:ncol(prices)) prices[, j] = na.locf(prices[, j])       # fill in
prices = prices[, apply(prices, 2, function(x) ! any(is.na(x)))]  # omit stocks with missing data

Now that we have a universe of stocks with valid price data, convert those prices to log(returns) for the remaining analysis (by returns I mean simply the ratio of prices relative to the first price).

Why log(returns) instead of prices?

The log(returns) are closer to normally distributed than prices especially in the long run. Pat Burns wrote a note about this (with a Tom Waits soundrack): http://www.portfolioprobe.com/2012/01/23/the-distribution-of-financial-returns-made-simple/.

But why care about getting data closer to normally distributed?

That turns out to be important to us because later we’ll use a technique called partial correlation. That technique generally works better for normally distributed data than otherwise, see for example a nice technical discussion about this by Baba, Shibata, and Sibuya here: https://doi.org/10.1111%2Fj.1467-842X.2004.00360.x

The following simple code converts our prices matrix into a matrix of log(returns):

log_returns = apply(prices, 2, function(x) diff(log(x)))

Sample correlation matrix

It’s easy to convert the downloaded log(returns) data into a Pearson’s sample correlation matrix X:

X = cor(log_returns)

The (i, j)th entry of the sample correlation matrix X above is a measurement of the degree of linear dependence between the log(return) series for the stocks in columns i and j.

There exist at least two issues that can lead to serious problems with the interpretation of the sample correlation values:

As Ledoit and Wolf point out, it’s well-known that empirical correlation estimates may contain lots of error.

Correlation estimates between two stock log(return) series can be misleading for many reasons, including spurious correlation or existence of confounding variables related to both series (http://www.tylervigen.com/spurious-correlations).

A Nobel-prize winning approach to dealing with the second problem considers cointegration between series instead of correlation, see for example notes by Eric Zivot (https://faculty.washington.edu/ezivot/econ584/notes/cointegrationslides.pdf), or Bernhard Pfaff’s lovely book “Analysis of Integrated and Cointegrated Time Series with R” (http://www.springer.com/us/book/9780387759661), or Wikipedia (https://en.wikipedia.org/wiki/Cointegration). (I also have some weird technical notes on the numerics of cointegration at http://illposed.net/cointegration.html.)

Cointegration is a wonderful but fairly technical topic. Instead, let’s try a simpler approach.

We can try to address issue 2 above by controlling for confounding variables, at least partially. One approach considers partial correlation instead of correlation (see for example the nice description in Wikipedia https://en.wikipedia.org/wiki/Partial_correlation). That approach works best in practice with approximately normal data–one reason for the switch to log(returns) instead of prices. We will treat the entries of the precision matrix as measures of association in a network of stocks below.

It’s worth stating that our simple approach basically treats the log(returns) series as a bunch of vectors and not so much bona fide time series, and can’t handle as many pathologies that might occur as well as cointegration can. But as we will see, this simple technique is still pretty effective at finding structure in our data. (And indeed related methods as discussed by Ledoit and Wolf and elsewhere are widely used in portfolio and risk analyses in practice.)

The partial correlation coefficients between all stock log(returns) series are the entries of the inverse of the sample correlation matrix (https://www.statlect.com/glossary/precision-matrix).

Market trading of our universe of companies, with myriad known and unknown associations between them and the larger economy, produced the stock prices we downloaded. Our objective is a kind of inverse problem: given a bunch of historical stock prices, produce a network of associations.

You may recall from some long ago class that, numerically speaking, inverting matrices is generally a bad idea. Even worse, issue 1 above says that our estimated correlation coefficients contain error (noise). Even a tiny amount noise can be hugely amplified if we invert the matrix. That’s because, as we will soon see, the sample correlation matrix contains tiny eigenvalues and matrix inversion effectively divides the noise by those tiny values. Simply stated, dividing by a tiny number returns a big number–that is, matrix inversion tends to blow the noise up. This is a fundamental issue (in a sense, the fundamental issue) common to many inverse problems.

Ledoit and Wolf’s sensible answer to reducing the influence of noise is regularization. Regularization replaces models with different, but related, models designed to reduce the influence of noise on their output. LW use a form of regularization related to ridge regression (a. k. a. Tikhonov regularization) with a peculiar regularization operator based on a highly structured estimate of the covariance. We will use a simpler kind of regularization based on an eigenvalue decomposition of the sample correlation matrix X.

Regularization

Here is an eigenvalue decomposition of the sample correlation matrix:

L = eigen(X, symmetric=TRUE)

Note that R’s eigen() function takes care to return the (real-valued) eigenvalues of a symmetric matrix in decreasing order for us. (Technically, the correlation matrix is symmetric positive semi-definite, and will have only nonnegative real eigenvalues.)

Each eigenvector represents an orthogonal projection of the sample correlation matrix into a line (a 1-d shadow of the data); The first two eigenvectors define a projection of the sample correlation matrix into a plane (2-d), and so on. The eigenvalues estimate the proportion of information (or variability if you prefer) from the original sample correlation matrix contained in each eigenvector. Because the eigenvectors are orthogonal, these measurements of projected information are additive.

Here is a plot of all the sample correlation matrix eigenvalues (along with a vertical line that will be explained in a moment):

plot(L$values, ylab="eigenvalues")
abline(v=10)

The eigenvalues fall off rather quickly in our example! That means that a lot of the information in the sample correlation matrix is contained in the first few eigenvectors.

Let’s assume, perhaps unreasonably, that the errors in our estimate of the correlation matrix are equally likely to occur in any direction (that the errors are white noise, basically). As we can see above, most of the information is concentrated in the subspace corresponding to the first few eigenvectors. But white noise will have information content in all the dimensions more or less equally.

One regularization technique replaces the sample correlation matrix with an approximation defined by only its first few eigenvectors. Because they represent a large amount of the information content, the approximation can be pretty good. More importantly, because we assumed noise to be more or less equally represented across the eigenvector directions and we’re cutting most of those off, this approximation tends to damp the noise more than the underlying information. Most importantly, we’re cutting off the subspace associated with tiny eigenvalues, avoiding the problem of division by tiny values and significantly reducing amplified noise in the inverse of the sample correlation matrix (the precision matrix).

The upshot is, we regularize the sample correlation matrix by approximating it by a low-rank matrix that substantially reduces the influence of noise on the precision matrix. See Per Christian Hansen’s classic paperback “Rank-Deficient and Discrete Ill-Posed Problems” (http://epubs.siam.org/doi/book/10.1137/1.9780898719697) for insight into related topics.

But how to choose a cut-off rank?

There is a substantial mathematical literature for just this topic (regularization parameter choice selection), complete with deep theory as well as lots of heuristics. Let’s keep things simple for this example and form our approximation by cutting off eigenvectors beyond where the eigenvalue plot starts to flatten out – close to the vertical line in the above plot.

Alternatively consider the lovely short 2004 paper by Chris Ding and Xiaofeng He (http://dl.acm.org/citation.cfm?id=1015408) that illuminates connections (that I happen to find fascinating) between k-means clustering and projections like truncated eigenvalue expansions. Although we aren’t interested in k-means clustering per se, our objective is connected to clustering. Ding and He show that we can find at least k (k-means) clusters using the first k - 1 eigenvectors above. This gives us another heuristic way to choose a projection dimension, at least if we have an idea about the number of clusters to look for.

A precision matrix, finally

Finally, we form the precision matrix P from the regularized sample correlation matrix. The inversion is less numerically-problematic now because of regularization. Feel free to experiment with the projected rank N below!

N = 10  # (use 1st 10 eigenvectors, set N larger to reduce regularization)
P = L$vectors[, 1:N] %*% ((1 / L$values[1:N]) * t(L$vectors[, 1:N]))
P = P / tcrossprod(sqrt(diag(P)))

Other approaches

I’m not qualified to write about them, but you should be aware that Bayesian approaches to solving problems like this are also effectively (and effective!) regularization methods. I hope to someday better understand the connections between classical inverse problem solution methods that I know a little bit about, and Bayesian methods that I know substantially less about.

Put a package on it

There is a carefully written R package to construct regularized correlation and precision matrices: the corpcor package (https://cran.r-project.org/package=corpcor, and also see http://strimmerlab.org/software/corpcor/) by Juliane Schafer, Rainer Opgen-Rhein, Verena Zuber, Miika Ahdesmaki, A. Pedro Duarte Silva, and Korbinian Strimmer. Their package includes the original Ledoit Wolf-like regularization method, as well as refinements to it and many other methods. The corpcor package, like Ledoit Wolf, includes ways to use sophisticated regularization operators and can apply more broadly than the simple approach taken in this note.

You can use the corpcor package to form a Ledoit-Wolf-like regularized precision matrix P, and you should try it! The result is pretty similar to what we get from our simple truncated eigenvalue decomposition regularization in this example.

Networks and clustering

The (i, j)th entry of the precision matrix P is a measure of association between the log(return) time series for the stocks in columns i and j, with larger values corresponding to more association.

An interesting way to group related stocks together is to think of the precision matrix as an adjacency matrix defining a weighted, undirected network of stock associations. Thresholding entries of the precision matrix to include, say, only the top ten per cent results in a network of only the most strongly associated stocks.

Thinking in terms of networks opens up a huge and useful toolbox: graph theory. We gain access to all kinds of nifty ways to analyze and visualize data, including methods for clustering and community detection.

R’s comprehensive igraph package by Gábor Csárdi (https://cran.r-project.org/package=igraph) includes many network cluster detection algorithms. The example below uses Blondel and co-authors’ fast community detection algorithm implemented by igraph’s cluster_louvain() function to segment the thresholded precision matrix of stocks into groups. The code produces an igraph graph object g, with vertices colored by group membership.

suppressMessages(library(igraph))

threshold = 0.90
Q = P * (P > quantile(P, probs=threshold))                           # thresholded precision matrix
g = graph.adjacency(Q, mode="undirected", weighted=TRUE, diag=FALSE) # ...expressed as a graph

# The rest of the code lumps any singletons lacking edges into a single 'unassociated' group shown in gray
# (also assigning distinct colors to the other groups).
x = groups(cluster_louvain(g))
i = unlist(lapply(x, length))
d = order(i, decreasing=TRUE)
x = x[d]
i = i[d]
j = i > 1
s = sum(j)
names(x)[j] = seq(1, s)
names(x)[! j] = s + 1
grp = as.integer(rep(names(x), i))
clrs = c(rainbow(s), "gray")[grp[order(unlist(x))]]
g = set_vertex_attr(g, "color", value=clrs)

Use the latest threejs package to make a nice interactive visualization of the network (you can use your mouse/trackpad to rotate, zoom and pan the visualization).

library(threejs)
graphjs(g, vertex.size=0.2, vertex.shape=colnames(X), edge.alpha=0.5)

The stock groups identified by this method are uncanny, but hardly all that surprising really. Look closely and you will see clusters made up of bank-like companies (AIG, BAC, BK, C, COF, GS, JPM, MET, MS, USB, WFC), pharmaceutical companies (ABT, AMGN, BIIB, BMY, CELG, GILD, JNJ, LLY, MRK, PFE), computer/technology-driven companies (AAPL, ACN, CSCO, IBM, INTC, MSFT, ORCL, QCOM, T, TXN, VZ, and so on. With the threshold value of 0.9 above, a few stocks aren’t connected to any others – they appear in gray.

The groups more or less correspond to what we already know!

The FB, GOOG, AMZN, PCLN (Facebook, Alphabet/Google, Amazon, Priceline) group is interesting–it includes credit card companies V (Visa), MA (Mastercard). Perhaps the returns of FB, GOOG and AMZN are more closely connected to consumer spending than technology!

This way of looking at things also nicely highlights connections between groups. For instance, the pharma group is connected to consumer products group through JNJ and PG (Johnson and Johnson and Proctor and Gamble). See the appendix below for a visualization that explores different precision matrix threshold values, including lower values with far greater network connectivity.

Review

We downloaded daily closing stock prices for 100 stocks from the S&P 500, and, using basic tools of statistics and analysis like correlation and regularization, we grouped the stocks together in a network that highlights associations within and between the groups. The structure teased out of the stock price data is reasonably intuitive.

Appendix: threejs tricks

The following self-contained example shows how the network changes with threshold value. It performs the same steps as we did above, but uses some tricks in threejs and an experimental extension to the crosstalk package and a few additional R packages to present an interactive animation. Enjoy!

suppressMessages({
library(quantmod)
library(igraph)
library(threejs)
library(crosstalk)
library(htmltools)
# using an experimental extension to crosstalk:
library(crosstool) # devtools::install_github('bwlewis/crosstool')
})

# Download the processed log(returns) data:
suppressMessages(load(url("http://illposed.net/logreturns.rdata")))

X = cor(log_returns)
L = eigen(X, symmetric=TRUE)
N = 10  # (use 1st 10 eigenvectors, set N larger to reduce regularization)
P = L$vectors[, 1:N] %*% ((1 / L$values[1:N]) * t(L$vectors[, 1:N]))
P = P / tcrossprod(sqrt(diag(P)))
colnames(P) = colnames(X)

# A function that creates a network for a given threshold and precision matrix
f = function(threshold, P)
{
  Q = P * (P > quantile(P, probs=threshold))                           # thresholded precision matrix
  g = graph.adjacency(Q, mode="undirected", weighted=TRUE, diag=FALSE) # ...expressed as a graph

  x = groups(cluster_louvain(g))
  i = unlist(lapply(x, length))
  d = order(i, decreasing=TRUE)
  x = x[d]
  i = i[d]
  j = i > 1
  s = sum(j)
  names(x)[j] = seq(1, s)
  names(x)[! j] = s + 1
  grp = as.integer(rep(names(x), i))
  clrs = c(rainbow(s), "gray")[grp[order(unlist(x))]]
  g = set_vertex_attr(g, "color", value=clrs)
  set_vertex_attr(g, "shape", value=colnames(P))
}

threshold = c(0.97, 0.95, 0.90, 0.85, 0.8)
g = Map(f, threshold, MoreArgs=list(P=P)) # list of graphs, one for each threshold

# Compute force-directed network layouts for each threshold value
# A bit expensive to compute, so run in parallel!
library(parallel)
l = mcMap(function(x) layout_with_fr(x, dim=3, niter=150), g, mc.cores=detectCores())

sdf = SharedData$new(data.frame(key=paste(seq(0, length(threshold) - 1))), key=~key)
slider = crosstool(sdf, "transmitter",
                sprintf("<input type='range' min='0' max='%d' value='0'/>",
                length(threshold) - 1), width="100%", height=20, channel="filter")
vis = graphjs(g, l, vertex.size=0.2, main=as.list(threshold), defer=TRUE, edge.alpha=0.5, deferfps=30,
        crosstalk=sdf, width="100%", height=900)

browsable(div(list(HTML("<center>"), tags$h3("Precision matrix quantile threshold (adjust slider to change)"), slider, vis)))

Precision matrix quantile threshold (adjust slider to change)

Editors Note: This post was revised on 8/28/17 to include a correction by the author that rescales the precision matrix.

Visualizations with R and Databases

Wed, 16 Aug 2017 00:00:00 +0000

The Challenge

Visualizations are one of R’s strengths. There are many functions and packages that create complex plots, often with one simple command. These plotting functions do two things: first, they take the raw data and run the calculations needed for a given visualization, and second, they draw the plot. If the source of the data resides within a database, the usual approach is to import all of the data and then create the plot. This is a problem, especially if the data is large.

A strategy to address this problem is found in the new Database with RStudio website. The Creating Visualizations page outlines a solution that introduces the “Transform in Database, plot in R” concept, and demonstrates its practical implementation. The article focused on knowledge sharing, rather than on providing a tool.

Introducing `dbplot`

The new dbplot package is meant to collect multiple functions for in-database visualization code. It implements the principles laid out in the Creating Visualizations page, and it provides three types of functions:

Helper functions that return a ggplot2 visualization
Helper functions that return the results of the plot’s calculations
The db_bin() function introduced in the Creating Visualizations page

The package provides calculations or “base” ggplot2 visualizations for the following:

Bar plot
Line plot
Histogram
Raster

Installation

Install dbplot from GitHub using the devtools package

devtools::install_github("edgararuiz/dbplot")

Example

This example will use a Microsoft SQL Server database connection to provide a quick glance of how the package works. For more examples, please visit the package’s GitHub repository.

dbplot functions

The dbplot_histogram() function creates a 30-bin histogram by default. Because it uses dplyr commands to perform the bin calculations, the function will work with any database that has dplyr support, including sparklyr. The only caveat is that the database must support basic functions like max() and min(), which some database types do not support.

library(dbplyr)

tbl(con, "airports") %>% 
  dbplot_histogram(alt)

This example shows how the resulting plot object can be further refined after the dbplot_histogram() function returns a plot:

tbl(con, "airports") %>% 
  dbplot_histogram(alt, binwidth = 700) + 
  labs(title = "Airports Altitude") +
  theme_minimal()

db_compute functions

If more control over the plot is needed, then the db_compute_bins() function returns a data frame with the lowest value of each bin and the record count per bin:

tbl(con, "airports") %>% 
  db_compute_bins(alt)

## # A tibble: 28 x 2
##       alt count
##     <dbl> <int>
##  1  -54.0   559
##  2  250.4   176
##  3  554.8   203
##  4  859.2   131
##  5 1163.6    82
##  6 1468.0    40
##  7 1772.4    20
##  8 2076.8    18
##  9 2381.2    16
## 10 2685.6    12
## # ... with 18 more rows

The results of the compute command can then be piped into a plot:

tbl(con, "airports") %>% 
  db_compute_bins(alt) %>%
  ggplot() +
  geom_col(aes(alt, count, fill = count))

db_bin()

The dbplot package includes the db_bin() function, first introduced in the Creating Visualizations page. For more information, please read the Histogram section.

db_bin(any_field)

## (((max(any_field) - min(any_field))/(30)) * ifelse((as.integer(floor(((any_field) - 
##     min(any_field))/((max(any_field) - min(any_field))/(30))))) == 
##     (30), (as.integer(floor(((any_field) - min(any_field))/((max(any_field) - 
##     min(any_field))/(30))))) - 1, (as.integer(floor(((any_field) - 
##     min(any_field))/((max(any_field) - min(any_field))/(30))))))) + 
##     min(any_field)

Next steps

More plots will be possible as dplyr-to-SQL translations are fine-tuned and enhanced. The dbplot package will be the place where new calculations and plots will be implemented.

End-to-end visualization using ggplot2

Mon, 14 Aug 2017 00:00:00 +0000

ggplot2 is kind of a household word for R users. I’ve ended up using it for complex data munging and wrangling work, where I needed to get clarity on different aspects of the data, especially being able to get different views, slices and dices of it, but in a nice visualization. At some point along the line, I slowly stopped using more traditional plotting functions like plot(), matplot(), barplot(), etc.

This article is an end-to-end data visualization exercise, using only ggplot2(). It has been helpful for me to see such pieces online on the endless possibilities of ggplot2(), so I wanted to give back to the community by doing one of my own.

1. Pima Indian Diabetes data

Consider the Pima Indian Diabetes dataset available in R. It looks at the population of women who were at least 21 years of age, of Pima Indian heritage and living near Phoenix, Arizona, and were tested for diabetes according to WHO criteria. In this exercise, I will use the 332 test data subjects. There are no missing values in this data. It is a very simple dataset, but my goal is to use it to demonstrate the tools available in ggplot2 to visually investigate a dataset we know very little about. This is part of the important data exploration phase of a data science project, to help prepare for the modeling phase.

library(MASS)
d <- Pima.te
summary(d)

##      npreg             glu              bp              skin      
##  Min.   : 0.000   Min.   : 65.0   Min.   : 24.00   Min.   : 7.00  
##  1st Qu.: 1.000   1st Qu.: 96.0   1st Qu.: 64.00   1st Qu.:22.00  
##  Median : 2.000   Median :112.0   Median : 72.00   Median :29.00  
##  Mean   : 3.485   Mean   :119.3   Mean   : 71.65   Mean   :29.16  
##  3rd Qu.: 5.000   3rd Qu.:136.2   3rd Qu.: 80.00   3rd Qu.:36.00  
##  Max.   :17.000   Max.   :197.0   Max.   :110.00   Max.   :63.00  
##       bmi             ped              age         type    
##  Min.   :19.40   Min.   :0.0850   Min.   :21.00   No :223  
##  1st Qu.:28.18   1st Qu.:0.2660   1st Qu.:23.00   Yes:109  
##  Median :32.90   Median :0.4400   Median :27.00            
##  Mean   :33.24   Mean   :0.5284   Mean   :31.32            
##  3rd Qu.:37.20   3rd Qu.:0.6793   3rd Qu.:37.00            
##  Max.   :67.10   Max.   :2.4200   Max.   :81.00

head(d)

##   npreg glu bp skin  bmi   ped age type
## 1     6 148 72   35 33.6 0.627  50  Yes
## 2     1  85 66   29 26.6 0.351  31   No
## 3     1  89 66   23 28.1 0.167  21   No
## 4     3  78 50   32 31.0 0.248  26  Yes
## 5     2 197 70   45 30.5 0.158  53  Yes
## 6     5 166 72   19 25.8 0.587  51  Yes

The target variable, type, tells us whether a patient is diabetic or not.

2. Distributions across categories

When the target is categorical, as in this case, type, I like to start by examining distributions for the continuous input columns. This gives us an overall sense of which input is likely to be useful. To do this, I like to do both boxplots and a density plot, since each has a different goal.

2.1. Boxplots

First, I’ll use boxplots, but ggplot2-style. I really like the look of a ggplot2() boxplot. It also allows me to seamlessly have multiple plots in a grid, as well as tinker around with the plotting parameters more flexibly than in a classical boxplot() approach, and end up with a nice-looking plot. We can see below how some inputs clearly vary across the 2 target categories, and others don’t.

df <- subset(d, select=c(glu, bp, skin, bmi, ped, age, type))  

library(gridExtra)
library(ggplot2)
p <- list()

for (j in colnames(df)[1:6]) {
  p[[j]] <- ggplot(data=df, aes_string(x="type", y=j)) + # Specify dataset, input or grouping col name and Y
            geom_boxplot(aes(fill=factor(type))) + guides(fill=FALSE) + # Boxplot by which factor + color guide
            theme(axis.title.y = element_text(face="bold", size=14))  # Make the Y-axis labels bigger/bolder
}

do.call(grid.arrange, c(p, ncol=3))

2.2. Density plots

I have used various overlay-density packages in the past, sm.density.compare() for example. I find the overlay-density rendering in ggplot2() to be more visually pleasing, with little plotting parameter tuning. E.g., it’s clear in the plot below that diabetic patients are associated with more number of pregnancies. I really like the alpha parameter.

df$npreg <- d$npreg
g <- ggplot(df, aes(npreg))
g + geom_density(aes(fill=factor(type)), alpha=0.8) + 
    labs(title="Density plot", 
         subtitle="# Pregnancies Grouped by Diabetes Type",
         x="# Pregnancies",
         fill="Diabetes Type")

3. Grid views

Next, I want to mix things up a little, so that I can have multi-dimensional views. By this, I mean that I want to know how the target is distributed across a few important inputs, but I want to link those inputs up as well. Sort of like a 3-way table, but visualized nicely instead of numbers. I came across this problem recently in one of the projects, and while it seems like a basic must-have output to dig deeper, I really needed something like ggplot2 to implement it. Using facet_grid() was amazing, even more so on account of the smooth control one has on the plotting parameters within a ggplot2 setup.

3.1. Data preparation

Facet-wrapping and gridding is a must-have tool for deeper data views, but the process is a multi-step one. Not too complicated though - very intuitive under ggplot2. We start with creating some new categorical columns using the continuous ones. Note that this can be done in different ways: appending new columns directly to the data frame, or using the more sleeker dplyr() in combination with magrittr(), which I absolutely love. This integrates a number of operations into a single chunk, making it quite seamless. I am also loading up plyr(), since I will be using it later.

library(magrittr)
library(plyr)
library(dplyr)
df_grid <- d %>% 
          mutate(Skin = ifelse(d$skin <= 29, "low skin fold", "high skin fold"),
                  BMI = ifelse(d$bmi <= 33, "low BMI", "high BMI"),
                  Ped = ifelse(d$ped <= 0.31, "low pedigree",
                ifelse(d$ped > 0.3134 & d$ped <= 0.5844, "medium pedigree", "high pedigree"))) %>% 
  
            mutate(Ped = factor(Ped, levels = c("low pedigree", "medium pedigree", "high pedigree")))

3.2. Reshaping the data

Next, we need to prepare the data a little more before throwing it into the facet_grid() mix. Most importantly, we need to “reshape” it, i.e., while our data is a “wide”-form data frame, we need to convert this to a “long”-form to enable facet_grid() to easily pick up what it needs to “facet” the plot by. We will also add a “size” column - this will allow us to make more granular adjustments in our plot. I will also rename columns in order to enable easier axis labeling when plotting. Again, notice that instead of using reshape2(), which I have used for many years, we’re using gather() from tidyr(), all sewn together with the pipe in magrittr().

library(tidyr)
DF <- df_grid %>% 
    subset(select=c(type, Skin, BMI, Ped)) %>% 
    gather(variable, value, -c(Skin, Ped, BMI))

colnames(DF)[5] <- "Diabetes_Value"
DF$size <- rep(1.5, nrow(DF))
s <- 1.5

3.3. Facet Grid

We’ll try the basic facet_grid() plot, after which we’ll go in and make some adjustments. For now, our goal is the following: to see a “matrix” or “grid” of the BMI distribution across diabetes type, as a 2x2 table of pedigree/skin fold combinations. In other words, for low pedigree/low skin fold, how does BMI distribute across diabetes type? You can see the amount of information you can pack into just one plot. I have found this to be useful when presenting to an end-user or customer. It becomes all the more useful since its a very clear representation of this slice/dice, with little room for ambiguity.

# Simple
library(ggplot2)
ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  facet_grid(Skin ~ Ped)   # wrap up everything to showcase by multiple cols

This looks nice, but I would like to add more of a “pop”. I am going to outline each box, and bolden the fonts. Note that you can also color the “grid strips”, but I won’t do that right now.

# More color
p <- ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  geom_rect(aes(fill=NA, size=size),xmin =-Inf,xmax=Inf,ymin=-Inf,ymax=Inf,alpha = 0.0002, colour="black",show.legend = F) +   # use box drawn around each location to cleanly separate facets + suppress guide
  scale_size(range=c(s,s), guide=FALSE) + # use line width/size feature for cleaner plotting
  facet_grid(Skin ~ Ped) +   # wrap up everything to showcase by multiple cols
  theme(strip.text.x = element_text(face="bold", size=12)) +
  theme(strip.text.y = element_text(face="bold", size=12)) 
  # optional changes in strip
#+ theme(strip.text.x = element_text(face="bold", size=12, colour="white")) +
#  theme(strip.text.y = element_text(face="bold", size=12, color="white")) +
#  theme(strip.background = element_rect(fill="black"))
plot(p)

Much better. Look how nicely this granular plot adjustment in ggplot2 allows each “block” in the matrix to pop out. Its very clear how BMI is distributed across diabetes type, and how that in turn is distributed across both pedigree function and skin fold. We see that (as expected): 1. A higher triceps skin fold thickness is associated with a higher BMI, as well as a higher count of diabetic people. 2. The above is more true for a higher diabetes pedigree function.

This kind of a grid plot presents a very powerful tool for such multi-dimensional data views.

4. Heatmaps

I like heatmaps - there’s a sense of drama in the way you can see where “something is happening”. I’ve used heatmap.2() to implement hierarchical clustering and translating that to a heatmap. But I wanted to use ggplot2() to simply look at a dataset as a heatmap, without any underlying analysis, to detect patterns before any analysis begins.

In this case, I want ggplot2() to show me patterns across different input columns, for the two diabetes types, i.e., what inputs seem to differ across diabetic/non-diabetic patients. This will be clear once we render our dataset into a nice ggplot2() heatmap.

4.1. Data preparation

As usual, we need to prep our data before pushing it into the ggplot2() function. We’ll reshape and scale the data first, all within the plyr(), dplyr(), and magrittr() framework. I’ll also specify some plotting parameters that I will call into my ggplot2() function. I’m going to rely on RColorBrewer() for these.

df_heat <- d[order(d$type),1:8]
DF_Heat <- df_heat %>%
          mutate(id = 1:nrow(df_heat)) %>%
          select(c(npreg:age, id))  %>%
          gather(variable, value, -id)  %>%
          ddply(.(variable), transform,
                    rescale = scale(value))  # Notice that this reorders by "variables"
          
# Color scale for heatmap
library(RColorBrewer)
colors <- brewer.pal(9, 'Reds')

# Lines to split patients into diabetic/non-diabetic
my.lines <- data.frame(x1 = 0.5, x2 = 7.5, y1 = 223.5, y2 = 223.5)

4.2. Rendering the heatmap

# Basic plot
p <- ggplot(DF_Heat, aes(as.factor(variable), as.factor(id), group=id)) + 
  geom_tile(aes(fill = rescale),colour = "white") +
  scale_fill_gradient(low="green", high="red")

# Make adjustments
base_size <- 9
p_adj <- p + theme_grey(base_size = base_size) + labs(x = "",y = "") + scale_x_discrete(expand = c(0, 0)) +
  scale_y_discrete(expand = c(0, 0)) +
  geom_segment(data=my.lines, aes(x = x1, y = y1, xend=x2, yend=y2), size=1, inherit.aes=F) +
  theme(axis.text.y = element_blank(), axis.ticks.y = element_blank()) 
plot(p_adj)

Note that I have suppressed the ticks on the Y-axis. We can clearly see regions of interest on the heatmap. It would be better for these to easily pop out at the viewer, to enable which, I am going to invoke geom_rect().

# Borders of rectangles to indicate areas of interest on heatmap
my.lines.rect.1 <- data.frame(xmin = 1.5, xmax = 2.5, ymin = 223.5, ymax = 255.5)
my.lines.rect.2 <- data.frame(xmin = 3.5, xmax = 4.5, ymin = 223.5, ymax = 332)
my.lines.rect.3 <- data.frame(xmin = 5.5, xmax = 6.5, ymin = 223.5, ymax = 280.5)

p_adj + geom_rect(data=my.lines.rect.1, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=2, inherit.aes = F) +
  geom_rect(data=my.lines.rect.2, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=5, inherit.aes = F) +
  geom_rect(data=my.lines.rect.3, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=4, inherit.aes = F)

Much better.

5. Segmentation in a scatterplot

Finally, I want to try to implement some “basic-level clustering”. This is not model-based clustering; rather, it is simply using a scatterplot and a few nice plotting parameters in ggplot2() to make some things pop right out at the viewer - again, with little room for ambiguity. What I like most here is the boxes that we can draw nicely to showcase the “clusters” a little better, along-with the multi-layered information, e.g., age, BMI, glucose, etc.

The conclusions are logical and obvious from the following plot, but quite nicely illustrate the use of ggplot2() for such a specific purpose.

d$Age <- ifelse(d$age < 30, "<30 yrs", ">= 30 yrs")

ggplot(d, aes(x = glu, y = bmi)) +
  geom_rect(aes(linetype = "High BMI - Diabetic"), xmin = 160, ymax = 40, fill = NA, xmax = 200, 
            ymin = 25, col = "black") + 
  geom_rect(aes(linetype = "Low BMI - Not Diabetic"), xmin = 0, ymax = 25, fill = NA, xmax = 120, 
            ymin = 10, col = "black") + 
  geom_point(aes(col = factor(type), shape = factor(Age)), size = 3) +
  scale_color_brewer(name = "Type", palette = "Set1") +
  scale_shape(name = "Age") +
  scale_linetype_manual(values = c("High BMI - Diabetic" = "dotted", "Low BMI - Not Diabetic" = "dashed"),
                        name = "Segment")

Hopefully, this little exercise will be helpful for someone wanting to use ggplot2() for an innovative slice/dice of a complex dataset, and to visualize it nicely.

Portfolio Volatility Shiny App

Wed, 09 Aug 2017 00:00:00 +0000

In our 3 previous posts, we walked through how to calculate portfolio volatility, then how to calculate rolling volatility, and then how to visualize rolling volatility. Today, we will wrap all of that work into a Shiny app that allows a user to construct his or her own five-asset portfolio, choose a benchmark and a time period, and visualize the rolling volatilities over time.

Here is the final app:

There will be a slight departure in form today because we will use a helpers.r file to hold our functions - those same functions that we worked so hard to create in the previous three posts. There are a few reasons to put them into a helper file.

The end user won’t be able to see them, which leads to a tangent on the idea of reproducibility. We need to ask, “Reproducible by whom?”. In this case, the Shiny app is 100% reproducible by anyone who has access to that helper file, which would be my colleagues with access to my files or Github repository. But, if the end user is an external client, for example, that end user wouldn’t have access and the analytic functions would remain a black box. In the world of finance, that’s necessary most of the time.
From a workflow perspective, that helper file allows us to test those functions in different formats. I can create an R Markdown report that uses the file, test different Shiny apps, or tweak the functions themselves without having to alter the actual application code.
From a stylistic perspective, the helper file keeps the Shiny app code a bit cleaner and much shorter. It’s not right or wrong to use a helpers file, but it’s worth thinking about as our Shiny apps get more involved.

Here’s how we load that file and have access to the functions and objects in it:

source("function-folder/simple-vol-helpers.r")

Since this is a blog post and we want to be complete, the code chunk below contains all the code from that simple-vol-helpers.r file.

# Calculate component returns

componentReturns_df <- function(stock1, stock2, stock3, stock4, stock5, start_date){
  
  symbols <- c(stock1, stock2, stock3, stock4, stock5)
  
  prices <- 
    getSymbols(symbols, src = 'yahoo', from = start_date, 
               auto.assign = TRUE, warnings = FALSE) %>% 
    map(~Cl(get(.))) %>% 
    reduce(merge) %>%
    `colnames<-`(symbols)
  
  # generate daily return series for funds
  prices_monthly <- to.monthly(prices, indexAt = "first", OHLC = FALSE)
  returns <- na.omit(ROC(prices_monthly, 1, type = "continuous"))
  
  
  returns_df <- returns %>% 
    as_tibble(preserve_row_names = TRUE) %>% 
    mutate(date = ymd(row.names)) %>% 
    select(-row.names) %>% 
    select(date, everything())
}


# Calculate rolling Portfolio Standard Deviation

rolling_portfolio_sd <- function(returns_df, start = 1, window = 6, weights){
  
  start_date <- returns_df$date[start]
  
  end_date <-  returns_df$date[c(start + window)]
  
  interval_to_use <- returns_df %>% filter(date >= start_date & date < end_date)
  
  returns_xts <- interval_to_use %>% as_xts(date_col = date) 
  
  w <- weights
  
  results_as_xts <- StdDev(returns_xts, weights = w, portfolio_method = "single")
  results_as_xts <- round(results_as_xts, 4) * 100
  
  results_to_tibble <- as_tibble(t(results_as_xts[,1])) %>% 
    mutate(date = ymd(end_date)) %>% 
    select(date, everything()) 
  
}

# Look how long this code chunk is. Easier to stash this in a helpers.r file!

All of those functions were explained and constructed in our previous Notebooks, so we won’t dwell on them today. Let’s move on to the appearance of the app itself!

First, we need to create an input sidebar where the user can choose assets, weights, a date, and a benchmark for comparison.

# This creates the sidebar input for the first stock and its weight.
# We'll need to copy and paste this fluidRow for each of the assets in our portfolio. 
fluidRow(
  column(6,
  textInput("stock1", "Stock 1", "SPY")),
  column(4,
  numericInput("w1", "Portf. %", 40, min = 1, max = 100))
)  

# Let the user choose a benchmark to compare to the portfolio volatility.
# We'll default to the Russell 2000 small cap index.
textInput("benchmark", "Benchmark for Comparison", "^RUT")


fluidRow(
  column(6,
  dateInput("start_date", "Start Date", value = "2013-01-01")),
  column(3,
  numericInput("window", "Window", 6, min = 3, max = 20, step = 1))
)

# This action button is important for user experience and server resources.
actionButton("go", "Submit")

That last line creates an actionButton, which is important for the end user. We have more than 10 user inputs in that sidebar, and without that actionButton, the app will start firing and reloading every time a user changes any of the inputs. This would be annoying for the user and taxing on the server! We will make sure the reactives wait for the user to click that button by using eventReactive.

For example, in the lines below, the app will wait to calculate the rolling portfolio volatility because the value of portfolio_rolling_vol is an eventReactive that won’t fire until input$go is true.

portfolio_rolling_vol <- eventReactive(input$go, {
  
  returns_df <- 
    componentReturns_df(input$stock1, input$stock2, input$stock3, input$stock4, 
                        input$stock5, input$start_date) %>% 
    mutate(date = ymd(date))
  
  weights <- c(input$w1/100, input$w2/100, input$w3/100, input$w4/100, input$w5/100)
  
  window <- input$window
  
  roll_portfolio_result <-
    map_df(1:(nrow(returns_df) - window), rolling_portfolio_sd, 
         returns_df = returns_df, window = window, weights = weights) %>%
    mutate(date = ymd(date)) %>% 
    select(date, everything()) %>%
    as_xts(date_col = date) %>% 
    `colnames<-`("Rolling Port SD")
   # an xts comes out of this
})

The user is going to choose a benchmark for comparison and we need another eventReactive to take that input and calculate rolling volatility for the benchmark. The asset is passed via input$benchmark from the sidebar.

benchmark_rolling_vol <- eventReactive(input$go, {
  
  benchmark_prices <- 
    getSymbols(input$benchmark, src = 'yahoo', from = input$start_date, 
               auto.assign = TRUE, warnings = FALSE) 
  benchmark_close <- Cl(get(benchmark_prices))
    
  benchmark_prices_monthly <- to.monthly(benchmark_close, indexAt = "first", OHLC = FALSE)
  benchmark_returns <- na.omit(ROC(benchmark_prices_monthly, 1, type = "continuous"))
  
  benchmark_rolling_sd <- rollapply(benchmark_returns,
                             input$window,
                             function(x) StdDev(x))
  benchmark_rolling_sd <- round(benchmark_rolling_sd, 4) * 100
  
  
})

Finally, when we visualize, it’s nice to include the chosen benchmark in the title. Thankfully, that is a simple eventReactive.

benchmark <- eventReactive(input$go, {input$benchmark})

We have now calculated three reactive objects: portfolio_rolling_vol(), benchmark_rolling_vol(), and benchmark(). We pass them to highcharter and tweak aesthetics on the y-axis.

renderHighchart({
  highchart(type = "stock") %>% 
    hc_title(text = paste("Portfolio Volatility vs", benchmark(), "Volatility", sep = " ")) %>%
    hc_yAxis(title = list(text = "Vol Percent"),
           labels = list(format = "{value}%"),
           opposite = FALSE) %>% 
    hc_add_series(portfolio_rolling_vol(), name = "Portfolio Vol", color = "blue") %>%
    hc_add_series(benchmark_rolling_vol(), 
                  name = paste(benchmark(), "Vol", sep = " "),
                  color = "green") %>%
    hc_add_theme(hc_theme_flat()) %>%
    hc_navigator(enabled = FALSE) %>% 
    hc_scrollbar(enabled = FALSE)
})

We’ve presented nothing new of substance today, as those analytical functions were all built in previous posts. However, this app does allow the user to build a custom portfolio and compare to a benchmark of his or her choosing. Have fun with it, and try to find some assets whose volatility has been increasing since the election last November.

Next time, we’ll take a closer look at the VIX and how it compares to realized volatility. Until then!

R and Interactive Graphics

Mon, 07 Aug 2017 00:00:00 +0000

Judging from the number of JSM talks that incorporated interactive visualizations of some sort or another, it appears that interactive graphics have captured the attention of a good many statisticians. I found this a little surprising. Statisticians, on the whole, are not easily impressed by “eye candy”, and I believe that there are many, like me, who think that base R graphics remain a powerful tool for data exploration. The ability of R’s plot() function to quickly produce plots for all sorts of objects helps an R user attain that state of flow that makes R such a productive environment for data analysis.

But, it is exactly the desire to preserve the flow of exploratory analysis when working with large, complex data sets that is driving the creation of new visualization tools.

Below, I highlight two talks that address the need for new tools, and a third talk by Chambers Awared winner Carson Sievert who discussed design principles for creating interactive graphics software. These talks illustrated the state of the art of the integration of interactive graphics with R, and also showed how interactive visualization can be a real aid to statistical inference.

In his talk, Applications of Interactive Exploration of Large Multi-Panel Faceted Displays Using Trelliscope, Ryan Hafen showed how the Small Multiples capability of Trelliscope can help you simultaneously compare complex behavior of several entities. Ryan’s slides convey the salient points of his presentation.

Note that the interactive features work to keep you in the analytical flow by letting you easily change views and comparisons. Look here for documentation on the underlying tressiscopejs package.

Visualizing genomic data provides several challenges: one being the need to integrate multiple assays and annotations across many regions of a genome. For this, scientists have favored heatmaps for some time. In her presentation, Extracting Insights form Genomic Data via Complex Interactive Heatmaps Alicia Schep showed how interactivity takes heatmaps to the next level. Alicia’s iheatmaper package aids exploration, and allows scientists to add information and build multiple subplots around the main heatmap as the following diagram illustrates.

You can find Alicia’s slides here.

In his talk, delivered with the aid of the following slides, Carson Sievert discussed the difference between exploratory and expository visualizations along with design principles for creating interactive graphics software.

Perhaps the biggest take away from Carson’s presentation was his exhortation: “Interactive graphics software must be opinionated - Not enough statisticians influence the design / implementation”. The idea is that technical mastery notwithstanding, you shouldn’t expect even the best JavaScript developers to invent tools that really facilitate statistical inference without guidance. To give support to his argument that statisticians used to be better at, and more engaged with the development interactive graphics, Carson points to the ASA’s video library.

This is an exciting time to be involved with interactive graphics.

A Postcard from JSM

Wed, 02 Aug 2017 00:00:00 +0000

Baltimore has the reputation of being a tough town: hot in the summer and gritty, but the convention center hosting the Joint Statistical Meetings is a pretty cool place to be. There are thousands of people here and so many sessions (over 600) that it’s just impossible to get an overview of all that’s going on. So, here are couple of snapshots from an R-focused, statistical tourist.

First Snapshot: What’s in a Vector?

Yesterday, Gabe Becker showed some eye opening work he is doing with Luke Tierney and Thomas Kalibera on features that are making their way into the R core. Gabe started his presentation by running the following code:

x = 1 : 1e15
head(x)
tail(x)
x < 4
sort(x, decreasing = TRUE)

Don’t try this at home. It will almost certainly be unpleasant if you do, but it all ran instantaneously on Gabe’s laptop. This is a simple example of the new “ALTREP Framework” that promises considerable performance improvements to basic statistical calculations by exploiting two insights:

Sometimes things are easy if you have some previous knowledge about your data
Maybe it’s not always necessary to work with contiguous blocks of memory

For example, Gabe’s code ran so quickly because his new experimental version of R “knew” that x was sorted and that there were no NAs. As Gabe put it: “Knowing is a lot more than half the battle”. Another thing that R will eventually “know” is not to convert strings to characters until absolutely necessary. Preliminary work suggests that this feature will provide significant speedup to ordinary things like working with the model design matrix.

A really exciting aspect of the new work is that the R will export the ALTREP Framework making it possible for package developers to take advantage of knowledge about their particular data structures. If things go well, the new framework should be part of a 2018 release.

Second Snapshot: Statistical Learning with Big Data

In his talk on Monday, Professor Rob Tibshirani covered considerable ground that included definitions of Machine Learning and Statistical Learning, “A Consumer Report” for five popular machine learning algorithms and three case studies, one of which involved the use of a relatively new algorithm: Generalized Random Forests. The definitions:

Machine learning constructs algorithms that can learn from data
Statistical learning is a branch of Statistics that was born in response to Machine Learning, emphasizing statistical models and assessment of uncertainity

I think, capture the difference between that computer science and statistics mindsets.

The Consumer Report, captured in the following table, summarizes Professor Tibshirani’s current thinking on the strengths and weaknesses of the machine learning algorithms.

Although the table does offer definite advice on model selection, Professor Tibshirani also advised that practitioners experiment with several methods when dealing with difficult problems.

That’s it for now: wish you were here. I will write again soon.

Looking for R at JSM

Fri, 28 Jul 2017 00:00:00 +0000

I am very much looking forward to attending JSM which begins this Sunday. And once again, I will be spending a good bit of my time hunting for new and interesting applications of R. In years gone by, this was a difficult game at JSM because R, R Package, Shiny, tidyverse and the like did not often turn up in a keyword search. This year, however, there is quite a bit of low hanging fruit. Below twenty-five sessions are listed by their JSM Activity Number.

If you are going to the conference, try to work some of these into your program, and please feel free to write to me (joseph.rickert@rstudio.com) if you see an R related talk that was particularly interesting.

Sunday, 07/30/2017

24 - Acc: An R Package to Process, Visualize, and Analyze Accelerometer Data Jae Joon Song, University of Texas Health Science Center; Michael Swartz, The University of Texas Health Science Center At Houston, School of Public Health; Matthew G. Cox, University of Texas MD Anderson Cancer Center; Karen Basen-Engquist, University of Texas MD Anderson Cancer Center

83 - R Package TDA for Statistical Inference on Topological Data Analysis Jisu Kim, Carnegie Mellon University

Monday, 07/31/2017

118 - Identification of Potentially Problematic Prescribing to a Vulnerable Population Ryan Jarrett; Richard Epstein, Chapin Hall at the University of Chicago; Michael Cull, Vanderbilt University; David Schlueter, Vanderbilt University; Kathy Gracey, Vanderbilt University; Molly Butler, Vanderbilt University; Rameela Chandrasekhar, Vanderbilt University

159 - No Punt Intended Nicholas Eisemann, California State Univ Chico; Camille Pensabene, St. John Fisher College; Dylan Gouthro, Chico State; Demond Handley, Kansas State University ; MyDoris Soto, California State University-Fullerton

159 - A Shiny App for Investigation of NHL Draft Pick Career Trajectories James White, St. Lawrence University; Michael Schuckers, St. Lawrence University 162 - Forced Migration: Visualizing Patterns of Refugee Movements Jacquelyn Neal, Vanderbilt University; Hannah E Weeks, Vanderbilt University

171 - Developing a Web-Based Tool to Evaluate Survey Performance Using R Shiny Scott Fricker, U.S. Bureau of Labor Statistics; Jeffrey Gonzalez, Bureau of Labor Statistics; Randall Powers, US Bureau of Labor Statistics

182 - Identification of Potentially Problematic Prescribing to a Vulnerable Population Ryan Jarrett; Richard Epstein, Chapin Hall at the University of Chicago; Michael Cull, Vanderbilt University; David Schlueter, Vanderbilt University; Kathy Gracey, Vanderbilt University; Molly Butler, Vanderbilt University; Rameela Chandrasekhar, Vanderbilt University

243 - An R Shiny Foundation for Standardized Clinical Review Tools Jimmy Wong, FDA

250 - A Shiny App for Investigation of NHL Draft Pick Career Trajectories James White, St. Lawrence University; Michael Schuckers, St. Lawrence University 253 - Forced Migration: Visualizing Patterns of Refugee Movements Jacquelyn Neal, Vanderbilt University; Hannah E Weeks, Vanderbilt University

Tuesday, 08/01/2017

279 - Wilcoxon Rank-Based Tests for Clustered Data with R Package Clusrank Jun Yan, University of Connecticut; Yujing Jiang, University of Connecticut; Mei-Ling Ting Lee, University of Maryland; Xin He, University of Maryland, College Park; Bernard Rosner, Harvard Medical School

358 - Contextualizing Sensitivity Analysis in Observational Studies: Calculating Bias Factors for Known Covariates (Note this is also listed as Activity Number 252) Lucy D’Agostino McGowan; Robert A Greevy, Jr, Vanderbilt University

389 - Applications of Interactive Exploration of Large Multi-Panel Faceted Displays Using Trelliscope — Ryan Hafen, Purdue University

389 - Extracting Insights from Genomic Data via Complex Interactive Heatmaps — Alicia Schep, Stanford School of Medicine

389 - Visualizing the Changing Landscape of Software Developers — David Robinson, StackOverflowcom

Wednesday, 08/02/2017

454 - An R Package for Spatio-Temporal Change of Support Andrew Raim, U.S. Census Bureau; Scott H. Holan, University of Missouri; Jonathan R Bradley, Florida State University; Christopher Wikle, University of Missouri

504 - Discriminative Power and Variable Selection in Logistic Regression Models under Outcome Misspecification Jia Bao, NYU Langone Medical Center; Yongzhao Shao, New York University-School of Medicine

518 - EyeTrVis: An R Package for the Extraction and Visualization of Eye Tracking Data from People Looking at Posters — Chunyang Li ; Jurgen Symanzik, Utah State University

518 - In with the New: a Shiny App for Exploring Summary Data Tables from the Medical Expenditure Panel Survey — Emily Mitchell, Agency for Healthcare Research and Quality

518 - RCAP Designer: Creating Analytical Dashboards on the RCloud Platform — Robert Archibald, Macrosoft Inc ; Ganesh Subramaniam, AT&T ; Simon Urbanek, AT&T Labs Research

518 - Standardizing Assessment of Time-Series Forecasting Models — Nicholas G. Reich, University of Massachusetts Amherst ; Joshua Kaminsky, Johns Hopkins Bloomberg School of Public Health ; Lindsay Keegan, Johns Hopkins Bloomberg School of Public Health ; Stephen Lauer, University of Massachusetts - Amherst ; Krzysztof Sakrejda, University of Massachusetts - Amherst ; Justin Lessler, Johns Hopkins Bloomberg School of Public Health

543 - Discussing the Uses and Creation of R Shiny Applications Justin Post, North Carolina State University

548 - A girl geek’s guide to new research on interactive data visualization for statistics with lots of data — Dianne Cook, Monash University

587 - Ggmapr - Tools for Working with and Visualizing Maps Heike Hofmann, Iowa State University

Thursday, 08/03/2017

599 - Guiding Principles for Interactive Graphics Based on LIBD Data Science Projects Leonardo Collado-Torres, Lieber Institute for Brain Development; Bill Ulrich, Lieber Institute for Brain Development; Stephen Semick, Lieber Institute for Brain Development; Andrew E Jaffe, Lieber Institute for Brain Development

599 - Interactive Data Visualization on the Web Using R — Carson Sievert, Carson Sievert

June 2017 New Package Picks

Wed, 26 Jul 2017 00:00:00 +0000

Two hundred and thirty-eight new packages were added to CRAN in June. Below are my picks for the “Top 40”, organized into six categories: Biostatistics, Data, Machine Learning, Miscellaneous, Statistics and Utilities. Some packages, including geofacet and secret, already seem to be gaining traction.

Biostatistics

BIGL v1.0.1: Implements response surface methods for drug synergy analysis, including generalized and classical Loewe formulations and the Highest Single Agent methodology. There are vignettes on Methodology and Synergy Analysis.

colorpatch v0.1.2: Provides functions to show color patches for encoding fold changes (e.g., log ratios) and confidence values within a diagram; especially useful for rendering gene expression data and other types of differential experiments. See the vignette.

eesim v0.1.0: Provides functions to create simulated time series of environmental exposures (e.g., temperature, air pollution) and health outcomes for use in power analysis and simulation studies in environmental epidemiology. The vignette gives an overview of the package.

personalized v0.0.2: Provides functions for fitting and validating subgroup identification and personalized medicine models under the general subgroup identification framework of Chen et al. The vignette provides a brief tutorial.

tidygenomics v0.1.0: Provides method to deal with genomic intervals the “tidy way”. The vignette explains how they work.

Data

alfred v0.1.1: Provides direct access to the ALFRED and FRED databases. The vignette gives a brief example.

CityWaterBalance v0.1.0: Provides functions to retrieve data and estimate unmeasured flows of water through an urban network. Data for US cities can be gathered via web services using this package and dependencies. See the vignette for an introduction to the package.

censusapi v0.2.0: Provides a wrapper for the U.S. Census Bureau APIs that returns data frames of census data and metadata. Available data sets include the Decennial Census, American Community Survey, Small Area Health Insurance Estimates, Small Area Income and Poverty Estimates, and Population Estimates and Projections. There is a brief vignette.

dataverse v0.2.0: Provides access to Dataverse version 4 APIs, enabling data search, retrieval, and deposit. There are four vignettes: Introduction, Search and Discovery, Retrieval and Data Archiving.

data.world v1.1.1: Provides high-level tools for working with data.world data sets. There is a Quickstart Guide and a vignette for writing Queries.

SimMultiCorrData v0.1.0: Provides functions to generate continuous, binary, ordinal, and count variables with a specified correlation matrix that can be used to simulate data sets that mimic real-world situations (e.g., clinical data sets, plasmodes). There are several vignettes including an Overall Workflow for Data Simulation and a Comparison to Other Packages.

tidycensus v0.1.2: Provides an integrated R interface to the decennial US Census and American Community Survey APIs, and the US Census Bureau’s geographic boundary files.

ukbtools v0.9.0: Provides tools to work with UK Biobank datasets. The vignette shows how to get started.

wpp2017 v1.0-1: Provides and interface to data sets from the United Nation’s World Population Prospects 2017.

Machine Learning

cld3 v1.0: Provides an interface to Google’s experimental Compact Language Detector 3 algorithm, a neural network model for language identification that is the successor of cld2.

datafsm v0.2.0: Implements a method that automatically generates models of dynamic decision-making that both have strong predictive power and are interpretable in human terms. The vignette provides an example.

diceR v0.1.0: Provides functions for cluster analysis using an ensemble clustering framework. The vignette shows some examples.

glmertree v0.1-1: Implements recursive partitioning based on (generalized) linear mixed models (GLMMs) combining lmer() and glmer() from lme4 and lmtree() and glmtree() from partykit. The vignette shows an example.

greta v0.2.0: Lets users write statistical models in R and fit them by MCMC on CPUs and GPUs, using Google TensorFlow. There is a website, a Getting Started Guide, and vignettes providing Examples andTechnical Details.

penaltyLearning v2017.07.11: Implements algorithms from Learning Sparse Penalties for Change-point Detection using Max Margin Interval Regression. There is a vignette.

SentimentAnalysis v1.2-0: Implements functions to perform sentiment analysis of textual data using various existing dictionaries, such as Harvard IV, or finance-specific dictionaries, and create customized dictionaries. The vignette provides an introduction.

Miscellaneous

convexjlr v0.5.1: Provides a high-level wrapper for Julia package Convex.jl, which makes it easy to describe and solve convex optimization problems. There is a very nice vignette that shows how to optimize the parameters for several machine learning models.

interp v1.0-29: Implements bivariate data interpolation on both regular and irregular grids using either linear methods or splines.

pkggraph v0.2.0: Allows users to interactively explore and plot package dependencies for CRAN.

parallelDist v0.1.1: Provides a parallelized alternative to R’s native dist function to calculate distance matrices for continuous, binary, and multi-dimensional input matrices with support for a broad variety of distance functions from the stats, prox and dtw R packages. The vignette offers some results on performance.

Stats

anchoredDistR v1.0.3: Supplements the MAD# software that implements the Method of Anchored Distributions for inferring geostatistical parameters. There is a vignette.

bssm v01.1-1: Efficient methods for Bayesian inference of state space models via particle Markov chain Monte Carlo and importance sampling type corrected Markov chain Monte Carlo. There is a vignette on Bayesian Inference of State Space Models and an example of a Logistic Growth Model.

factorMerger v0.3.1 Provides a set of tools to support results of post-hoc testing and enable to extract hierarchical structure of factors. There is an Introduction and vignettes on Cox Regression Factor Merging and Multidimensional Gaussian Merging.

MittagLeffleR v0.1.0: Provides density, distribution, and quantile functions as well as random variate generation for the Mittag-Leffler distribution based on the algorithm by Garrappa. There are short vignettes for the math, distribution functions and random variate generation.

walker v0.2.0: Provides functions for building dynamic Bayesian regression models where the regression coefficients can vary over time as random walks. The vignette shows some examples.

Utilities

charlatan v0.1.0: Provides functions to make fake data, including addresses, person names, dates, times, colors, coordinates, currencies, DOIs, jobs, phone numbers, ‘DNA’ sequences, doubles and integers from distributions and within a range. The Introduction will get you started.

colordistances v0.8.0: Provides functions to load and display images, selectively mask specified background colors, bin pixels by color, quantitatively measure color similarity among images,and cluster images by object color similarity. There is an Introduction and vignettes on Pixel Binning Methods and Color Distance Metrics.

dbplyr v1.1.0: Implements a dplyr back end for databases that allows working with remote database tables as if they are in-memory data frames. There is an Introduction, a vignette for Adding a new DBI backend and one for SQL translation.

geofacet v0.1.5: Provides geofaciting functionality (the ability to arrange a sequence of plots for different geographical entities into a grid that preserves some geographical orientation) for ggplot2. There is a Package Reference vignette and an Introduction. The package is already getting some traction. This is a user submission:

ggformula v0.4.0: Provides a formula interface to ggplot2. There is a vignette explaining how it works.

gqlr v0.0.1: Provides an implementation of the GraphQL query language created by Facebook for describing data requirements on complex application data models. gqlr should be useful for integrating R computations into production applications that use GraphQL.

later v0.3: Allows users to execute arbitrary R or C functions some time after the current time, after the R execution stack has emptied. The vignette shows how to use later from C++.

secret v1.0.0: Allows sharing sensitive information like passwords, API keys, etc., in R packages, using public key cryptography. There is a vignette.

sessioninfo v1.0.0: Provides functions to query and print information about the current R session. It is similar to utils::sessionInfo(), but includes more information.

webglobe v1.0.2: Provides functions to display geospatial data on an interactive 3D globe. There is a vignette

Visualizing Portfolio Volatility

Fri, 21 Jul 2017 00:00:00 +0000

This is the third post in our series on portfolio volatility, variance and standard deviation. If you want to start at the beginning with calculating portfolio volatility, have a look at the first post here - Intro to Volatility. The second post on calculating rolling standard deviations is here: Intro to Rolling Volatility.

Today we will visualize rolling standard deviations with highcharter using two objects from that second post.

The charts, which are the fun payoff after all of the equations and functions that we have ground out in the previous posts, should highlight any unusual occurrences or volatility spikes/dips that we might want to investigate.

First, load the .RDat file saved from our previous Notebook (or you can run the scripts from the previous posts).

load('rolling-sd.RDat')

We now have 2 objects in our Global Environment - spy_rolling_sd - an xts object of rolling SPY standard deviations - roll_portfolio_result - an xts object of rolling portfolio standard deviations. Because both of those are xts objects, we can pass them straight to highcharter with the hc_add_series() function and set a name and a color with the name and color arguments. Nothing too complicated here - we did the hard work in our previous Notebooks.

highchart(type = "stock") %>%
  hc_title(text = "SPY v. Portfolio Rolling Volatility") %>%
  hc_add_series(spy_rolling_sd, name = "SPY Volatility", color = "blue") %>%
  hc_add_series(roll_portfolio_result, name = "Port Volatility", color = "green") %>%
  hc_navigator(enabled = FALSE) %>% 
  hc_scrollbar(enabled = FALSE)

It is interesting to note that from late April 2016 to late October 2016, SPY’s rolling standard deviation dipped below that of the diversified portfolio. The portfolio volatility was plunging at the same time, but SPY’s was falling faster. What happened over the 6 preceding months to explain this?

Maybe we should add a flag to highlight this event. We can also add flags for the maximum SPY volatility, maximum and minimum portfolio rolling volatility and might as well include a line for the mean rolling volatility of SPY to practice adding horizontal lines.

We will use two methods for adding flags. First, we’ll hard code the date for the flag as “2016-04-29” using the date when rolling SPY volatility dipped below the portfolio. Second, we’ll set a flag with the date
as.Date(index(roll_portfolio_result[which.max(roll_portfolio_result)]),format = "%Y-%m-%d") which looks like a convoluted mess but is adding a date for whenever the rolling portfolio standard deviation hit its maximum.

This is a bit more ‘dynamic’ because we can change our assets but keep this code the same and it will find the date with the maximum rolling standard deviation. Our first flag is not dynamic in the sense that it is specific to the comparison between SPY and this exact portfolio.

spy_important_date <- as.Date(c("2016-04-29"), format = "%Y-%m-%d")

port_max_date <- as.Date(index(roll_portfolio_result[which.max(roll_portfolio_result)]),
                         format = "%Y-%m-%d")
port_min_date <- as.Date(index(roll_portfolio_result[which.min(roll_portfolio_result)]),
                         format = "%Y-%m-%d")
spy_max_date <- as.Date(index(spy_rolling_sd[which.max(spy_rolling_sd)]),
                         format = "%Y-%m-%d")


highchart(type = "stock") %>%
  hc_title(text = "SPY v. Portfolio Rolling Volatility") %>%
  hc_add_series(spy_rolling_sd, name = "SPY Volatility", color = "blue", id = "SPY") %>%
  hc_add_series(roll_portfolio_result, name = "Portf Volatility", color = "green", id = "Port") %>%
  hc_add_series_flags(spy_important_date,
                      title = c("SPY Vol Dips"), 
                      text = c("SPY rolling sd dips below portfolio."),
                      id = "SPY") %>%
  hc_add_series_flags(spy_max_date,
                      title = c("SPY Max "), 
                      text = c("SPY max rolling volatility."),
                      id = "SPY") %>%
   hc_add_series_flags(port_max_date,
                      title = c("Portf Max"), 
                      text = c("Portfolio maximum rolling volatility."),
                      id = "Port") %>%
  hc_add_series_flags(port_min_date,
                      title = c("Portf Min"), 
                      text = c("Portfolio min rolling volatility."),
                      id = "Port") %>%
  hc_yAxis(title = list(text = "Mean SPY rolling Vol"),
           showFirstLabel = FALSE,
           showLastLabel = FALSE,
           plotLines = list(
             list(value = mean(spy_rolling_sd), color = "#2b908f", width = 2)))  %>% 
  hc_navigator(enabled = FALSE) %>% 
  hc_scrollbar(enabled = FALSE)

Hover on the flags and you can see the text we added for explanation.

It’s remarkable how rolling volatility has absolutely plunged since early-to-mid 2016. Since August of 2016, both the portfolio and SPY rolling standard deviations have been well below the SPY mean.

Thanks for sticking with this three-part introduction to volatility. Next time, we’ll port our work to Shiny and play with different assets and allocations.

Introduction to Rolling Volatility

Tue, 18 Jul 2017 00:00:00 +0000

This is the second post in our series on portfolio volatility, variance and standard deviation. If you missed the first post and want to start at the beginning with calculating portfolio volatility, have a look here - Introduction to Volatility. We will use three objects created in that previous post, so a quick peek is recommended.

Today we focus on two tasks:

Calculate the rolling standard deviation of SPY monthly returns.
Calculate rolling standard deviation of monthly returns of a 5-asset portfolio consisting of the following.
- AGG (a bond fund) weighted 10%
- DBC (a commodities fund) weighted 10%
- EFA (a non-US equities fund) weighted 20%
- SPY (S&P500 fund) weighted 40%
- VGT (a technology fund) weighted 20%

First though, why do we care about rolling standard deviations when in our previous Notebook we calculated ‘the’ standard deviation of monthly returns for both SPY and the portfolio? In that Notebook, what we calculated was the standard deviation of monthly returns for our entire sample, which was monthly returns for four-year period 2013-2017. What we might miss, for example, is a 3-month or 6-month period where the volatility spiked or plummeted or did both. And the longer our sample size, the more likely we are to miss something important. If we had 10 or 20 years of data and we calculated the standard deviation for the entire sample, we could fail to notice an entire year in which volatility was very high. Hence, we would fail to ponder the probability that it could occur again.

Imagine a portfolio which had a standard deviation of returns for each 6-month period of 3% and it never changed. Now, imagine a portfolio whose vol fluctuated every few 6-month periods from 0% to 6% . We might find a 3% standard deviation of monthly returns over a 10-year sample for both of these, but those two portfolios are not exhibiting the same volatility. The rolling volatility of each would show us the differences and then we could hypothesize about the past causes and future probabilities for those differences. We might also want to think about dynamically re-balancing our portfolio to better manage volatility if we are seeing large spikes in the rolling windows. We’ll look more into re balancing as this series progresses.

For now, let’s load the .RDat file saved from our previous Notebook.

load('portfolio-returns.RDat')

We now have 3 objects in our Global Environment:

+ `spy_returns` - an `xts` object of SPY monthly returns
+ `portfolio_component_monthly_returns_xts` - an `xts` object of returns of the 5 funds in our portfolio
+ `weights` - a vector of portfolio weights

Our least difficult task is calculating the rolling standard deviation of SPY returns. We use zoo::rollapply for this and just need to choose a number of months for the rolling window.

window <- 6

spy_rolling_sd <- na.omit(rollapply(spy_returns$SPY, window, 
                           function(x) StdDev(x)))

We now have an xts object called spy_rolling_sd that contains the 6-month rolling standard deviation of returns of SPY. Keep in mind that the chosen window is important and can affect the results quite a bit. Soon we’ll wrap this work to a Shiny app where changing the window and visualizing the results will be easier.

Next, we calculate the rolling volatility of our weighted portfolio. The rollapply function doesn’t play nicely with the weights argument that we need to supply to StdDev(). We will craft our own version of roll apply to make this portfolio calculation, which we will use in conjunction with the map_df() function from purrr.

Before we do that, a slight detour from our substance. Below are two piped workflows to quickly convert from xts to dataframe and back to xts. These rely heavily on the as_tibble() and as_xts() functions from the tidyquant.

# toggle from an xts object to a tibble
portfolio_component_monthly_returns_df <- 
  portfolio_component_monthly_returns_xts %>% 
  as_tibble(preserve_row_names = TRUE) %>% 
  mutate(date = ymd(row.names)) %>% 
  select(-row.names) %>% 
  select(date, everything())

# toggle from a tibble back to xts.
returns_xts <- portfolio_component_monthly_returns_df %>% 
  as_xts(date_col = date)

Why did we take that detour? Because we will use map_df(), mutate() and select() when we apply our custom function with the %>% operator and that will require a tibble/data.frame.

Before we step through the code of the custom function, let’s write out the goal and logic.

Our goal is to create a function that takes a data.frame of asset returns and calculates the rolling standard deviation based on a starting date index and a window, for a portfolio with specified weights for each asset. We will need to supply four arguments to the function, accordingly.

Here’s the logic I used to construct that function (feel free to eviscerate this logic and replace it with something better).

Assign a start date and end date based on the window argument. If we set window = 6, we’ll be calculating 6-month rolling standard deviations.
Use filter() to subset the original data.frame down to one window. I label the subsetted data frame as interval_to_use. In our example, that interval is a 6-month window of our original data frame.
Now we want to pass that interval_to_use object to StdDev(), but it’s not an xts object. We need to convert it and label it returns_xts.
Before we call StdDev(), we need weights. Create a weights object called w and give the value from the argument we supplied to the function.
Pass the returns_xts and w to StdDev().
We now have an object called results_as_xts. What is this? It’s the standard deviation of returns of the first 6-month window of our weighted portfolio.
Convert it back to a tibble and return.
We now have the standard deviation of returns for the 6-month period that started on the first date, because we default to start = 1. If we wanted to get the standard deviation for a 6-month period that started on the second date, we could set start = 2, etc.

rolling_portfolio_sd <- function(returns_df, start = 1, window = 6, weights){
 
  start_date <- returns_df$date[start]
  
  end_date <-  returns_df$date[c(start + window)]
  
  interval_to_use <- returns_df %>% filter(date >= start_date & date < end_date)
  
  returns_xts <- interval_to_use %>% as_xts(date_col = date) 
  
  w <- weights
  
  results_as_xts <- StdDev(returns_xts, weights = w, portfolio_method = "single")
  
  results_to_tibble <- as_tibble(t(results_as_xts[,1])) %>% 
    mutate(date = ymd(end_date)) %>% 
    select(date, everything()) 
  
}

We’re halfway there. We need to apply that function starting at the first date in our portfolio_component_monthly_returns_df object, and keep applying it to successive date indexes until the date that is 6 months before the final date. Why end there? Because there is no rolling 6-month standard deviation that starts only 1, 2, 3, 4 or 5 months ago!

We will invoke map_df() to apply our function to date 1, then save the result to a data.frame, then apply our function to date 2, and save to that same data.frame, and so on until we tell it stop at the index that is 6 before the last date index.

window <- 6

roll_portfolio_result <-
  map_df(1:(nrow(portfolio_component_monthly_returns_df) - window), rolling_portfolio_sd, 
         returns_df = portfolio_component_monthly_returns_df, window = window, weights = weights) %>%
  mutate(date = ymd(date)) %>% 
  select(date, everything()) %>%
  as_xts(date_col = date) %>% 
  `colnames<-`("Rolling Port SD")

head(roll_portfolio_result)

##            Rolling Port SD
## 2013-08-01      0.02411152
## 2013-09-01      0.02604424
## 2013-10-01      0.02726744
## 2013-11-01      0.02887195
## 2013-12-01      0.02888576
## 2014-01-01      0.02051912

Have a look at the rolling standard deviations. Why is the first date August of 2013? Do any of the results stand our as unusual, when compared to the SPY results? It’s hard to make the comparison until we chart them, and that’s what we do in the next post. See you then!

Control Systems Toolbox in R - a GSoC 2017 Project

Thu, 06 Jul 2017 00:00:00 +0000

Introduction

Control theory is an interdisciplinary branch of mathematics and engineering that has the objective of controlling physical systems. A control system is a device or a collection of devices that manage, command, direct or regulate the behavior of other devices or systems. Control systems engineering is a major application of control theory, and involves the study, design/modeling of automatic control systems. Control theory has also been applied to several other fields such as finance, sociology, psychology, physiology and other sciences that analyze dynamic systems.

In recent times, control systems have played a major role in the development and advancement of modern technology and civilization. Control systems are essential in any field of engineering and science, as they are an important and integral part of systems such as space-vehicle systems, robotic systems, modern manufacturing systems, nuclear reactors, navigation systems and industrial operations involving control of temperature, pressure, humidity, flow, speed/velocity, etc.

Example of a Control System

Imagine the operation of an Automatic Electric Iron where its heating elements are controlled by the temperature output of the iron. As long as there is power supply to the electric iron, and a desired heat-level is selected, the electric iron automatically controls at what time to turn-on and switch-off heating so as to maintain that temperature. This type of control system is called a feedback control system: a system in which the output is fed back to the input as a control signal.

Figure 1 shows a simple block diagram depicting a sample feedback control system for a heating system.

Figure 1: Block Diagram Model for a Heating System

Another example is the control system for an aircraft or a space shuttle. Truly, such control system would be complex as there are several sub-systems that are interconnected in such transport systems. Therefore, it is very necessary to simulate how these systems could behave using a computer before implementing them physically. This is the reason why computer simulation of control systems have been proliferated.

Control Systems R-GSoC Project

For this Google Summer of Code (GSoC) 2017 project, we are working to develop a control system toolbox for the R language. The rationale is that there is a control toolbox for most commercial and open-source technical computing platforms out there, and while the R language is experiencing tremendous growth in its capabilities and user base, it seems necessary to extend the language to serve other scientific and engineering communities. For example, in Python, the development of NumPy and the general SciPy stack, added some dimensions and attracted several scientific domains to the use of Python. This GSoC project hopes to make basic design, analysis and simulation of control systems possible within the R ecosystem.

The major objective of this year’s GSoC project is to try to:

Design and develop a framework for the package that defines data structures and creates flexibility of usage and extension for future users and interested contributors to easily adopt the package. The framework is also expected to mimic the I/O style of existing control systems toolkits.
To port a set of functions from the RLabPlus control-toolbox and adapt them to the defined data structures and general framework.
To test the functions, create documentation, usage examples and build them for distribution.

Benefits of a Control Systems Toolbox in R

Some benefits of having a control-systems-toolbox in R are:

A control package in R could be useful to engineers and students who are looking to migrate to an open-source language like R that has better visualization options and interfaces to several technologies. Models for several dynamic systems could then be analyzed using R.
The package could assist in further developments in areas like optimal control theory which has several applications to economics and management sciences.

Current State of the Project

At the time of writing this article, the first phase of coding for GSoC has elapsed, and we are delighted to meet our target of creating 18 functions that define the general framework and structures. Consequently, a user can, at this stage, perform the following:

Create LTI system models of state-space, transfer function and zero-pole classes
Conversions from one system model to other system models
Convert continuous-time system models to discrete-time models
Obtain time response for LTI systems
Install the development version of the package using: devtools::install_github(“benubah/control”)

Practical Example

Analyzing the Time Response of a Bandpass RLC Filter Circuit

Wikipedia: “A bandpass filter is a device that passes frequencies within a certain range and rejects (attenuates) frequencies outside that range.” Bandpass filters are commonly used in wireless transmitters and receivers to prevent transmitters from interfering with other stations and to allow signals within selected frequency ranges to be decoded, while preventing signals at unwanted frequencies from passing through to the receiver. In astronomy, bandpass filters are used to allow only certain portion of light spectrum pass through to an instrument. They have several other applications in neuroscience, digital signal processing, atmospheric sciences, etc.

The following example is taken from a Mathworks web-page and it shows how to analyze the time response of a common RLC bandpass filter circuit using the current available functions of the Control Systems Toolbox in R. The attenuation properties of the filter circuit, G2 can be ascertained by simulating how this filter transforms sine waves with frequency 0.9, 1, and 1.1 rad/s.

Figure 2: Bandpass RLC Filter Circuit.

For this example, we will try to show how the following could be achieved using the developing control library:

Creating a transfer function model
Converting a transfer function to state-space representation
Converting a continuous-time model to discrete-time model
Performing Linear simulation using lsim
Obtaining step response using lsim

The following values are given R = 1, L = 1, C = 1, R2 = 20

The transfer function for this circuit is:

\[ G(s) = \frac{V_output}{V_input} = \frac{s/(RC)}{s^2 + s/(RC) + 1/(LC)} \]

On the R Console (or in an R script), after installing the control package, the following code obtains the time response of the circuit:

library(control)

## 
## Attaching package: 'control'

## The following object is masked from 'package:base':
## 
##     append

R <- C <- L <- 1
R2  <-  20
t  <-  seq(0,250,0.05)
G1 <-  tf(c(1/(R*C), 0), c(1, 1/(R*C), 1/(L*C) ) )
G2  <-  tf(c(1/(R2*C), 0), c(1, 1/(R2*C), 1/(L*C) ) )
print(G1)

## 
## y1:
##              s 
##   - - - - - - - - -
##      s^2 +  s + 1 
## 
## 
## Transfer Function: Continuous time model

tf2ss(G1) # example: convert the transfer function to state-space representation

## 
##  sys.A = 
## 
##      x1   x2
## x1   -1   -1
## x2    1    0
##    
## sys.B = 
## 
##      u1
## x1    1
## x2    0
##    
## sys.C = 
## 
##      x1   x2
## y1    1    0
##    
## sys.D = 
## 
##      u1
## y1    0
##    
## 
##  State-Space system: Continuous time model

c2d(G1, 0.1) # example: convert the system G1 from continuous-time to discrete-time model

## 
## y1:
##   0.09500408 z^1 - 0.09500408 
##   - - - - - - - - - - - - - - - - -
##      z^2 - 1.895329 z + 0.9048374 
## 
## 
## Sample Time = 0.1 
## Transfer function: Discrete time model

response <- lsim(G2,sin(t),t)
par(mfrow = c(3,1))
plot(t, sin(t), type = "l", xlab = "Time, secs", ylab = "Amplitude", main = "Linear Simulation Response - w = 1", col = "lightgray")
lines(t, response$y, type = "l", col = "blue", lwd = 2)
grid(5,5, col = "lightgray")

response2 <- lsim(G2, sin(0.9*t), t)
plot(t, sin(t), type = "l", xlab = "Time, secs", ylab = "Amplitude", main = "Linear Simulation Response - w = 0.9", col = "lightgray")
lines(t, response2$y, type = "l", col = "blue", lwd = 2)
grid(5,5, col = "lightgray")

response3 <- lsim(G2, sin(1.1*t), t)
plot(t, sin(t), type = "l", xlab = "Time, secs", ylab = "Amplitude", main = "Linear Simulation Response - w = 1.1", col = "lightgray")
lines(t, response3$y, type = "l", col = "blue", lwd = 2)
grid(5,5, col = "lightgray")

Figure 1: Figure 3: R Plot for Linear Simulation Response of RLC bandpass circuit

The output of the simulation displayed in Figure 3 shows a pure sine wave (in light-gray color) behind the wave response from our linear simulation (in blue). At w = 1 rad/s, after the transients have died off, the wave remains a pure sine wave – no attenuation observed. However, the waves at 0.9 and 1.1 rad/s are significantly attenuated.

Please get the code here: https://github.com/benubah/controldev/blob/master/R/lsimExample.R

To obtain a step response for the RLC filter circuit using lsim as shown in Figure 4, the following code is used:

dims <- dim(as.matrix(t))
   u <- matrix(rep(1,dims[1]), dims[1], dims[2])
   stepresponse <- lsim(G2, u, t)
plot(t, stepresponse$y, type = "l", xlab = "Time, secs", ylab = "Amplitude", main = "Step Response - G2", col = "red", lwd = 2)
grid(7,7, col = "lightgray", lwd = 2)

Figure 2: Figure 4: R Plot for Step Response of the RLC bandpass circuit

Next Coding Phase

In the next coding stage, we plan to attempt creating functions that would perform the following:

Evaluate transfer function expressions in the s-domain within R as part of the framework.
Control system block diagram algebra functions – system interconnections

Challenges

The expected challenge for this project is to accurately set up a framework that defines data structures and how functions operate within the framework in such a manner that would be attractive (due to simplicity, reusability and extensibility) to prospective users and contributors.

To overcome this challenge, through the assistance of my mentors, we have examined the data architectures and plotting layout of other open-source scientific languages, and from these, designed a plan and framework that would integrate reasonably with the architecture of the R language while remaining open to improvements.

Dependencies

The Control Systems Toolbox for the R language is expected to depend on R packages like, pracma, signal and expm.

Feedback

We would be glad to hear from the R community – feel free to get in touch with any suggestions, opinions or comments.

Conclusion

In this article, we have introduced the Control Systems Toolbox GSoC 2017 project, its current state, benefits and practical examples showing some functions that are currently available. We have also listed the future coding plans for the next GSoC coding phase. We hope to be able to update the R community as things develop.

Printing From Flex Dashboard

Wed, 28 Jun 2017 00:00:00 +0000

Shiny applications of all stripes (including flexdashboard with runtime Shiny) are revolutionary in that they put the power of R directly in the end user’s hands without needing to interact directly with the language. A common way end-users wish to interact with their data is via a dashboard that they can manipulate on the fly. Flexdashboard streamlines the process of turning an R-based analysis into a dashboard, so R users can create good-looking output for their analyses - and deploy this output to the web - with very little additional effort.

There are various ways to extract information from flexdashboard, but no good way to programatically return a printable version of the dashboard itself. This is partially by design - responsive HTML dashboards aren’t meant to be printed. There are well-documented workflows about how to share information between a web-centric output and a complementary report that renders to a printable format.

That said, there are some reasons to want a printable output that looks similar to a related HTML report. Snapshots are valuable for documentation/training materials, for embedding in emails or other reports, or for the all-too-common case of meetings with executives who like dashboards but want to consume paper-based reports. Plus, flexdashboards are an incredibly quick way to create visually appealing (read: pretty) summary output of a combination of outputs on a single page.

Both Harrison and Aaron¹ have struggled with trying to print flexdashboards in their respective practices. In a recent instance, Harrison was working with a client whose end users would be carrying snapshots of the dashboard into an industrial setting where mobile devices were not desired. It was important to the client to maintain a similar look between the on-screen and paper representations of the dashboard. It was also important to make generating these snapshots an integrated part of the application; it was not sufficient to suggest that the client manually take a screenshot of their browser window. This turned out to be a blessing, because (as you will see below) it allows the creation of a web-friendly ‘tabbed’ report with a companion ‘flat’ printed rendering.

Specific Task

Our goal was to create a “printable” flexdashboard with runtime Shiny. Flexdashboard produces a beautiful HTML dashboard with very little effort. Like all R Markdown documents, it also allows Shiny objects to be embedded inside of it, so that when rendered locally or on a server, it produces a reactive web application with minimal developer effort. Like many HTML files, however, it does not print well.

The flexdashboard in question also had tabsets, which make perfect sense on a computer or mobile device, but don’t translate well to printed media. When printed, tabsets communicate that there is pertinent information hidden from the viewer; better to remove the tabset in the printed version. If the information is pertinent, it should be displayed in the printed document; if it is not, it should be removed.

PhantomJS and Headless Chrome

The approach we settled on as good enough and easy enough was to use a headless browser to take a screenshot of a non-reactive HTML document. It’s not a perfect solution: a PDF would be better than a png file, headless browsers add a code dependency, and there is some administrative overhead to create the document to print. It is, however, quick to implement, easy to understand, and robust.

Example²

The mechanics of building a screen-capture-style report from flexdashboard are neither difficult nor obvious. In general, it requires the following steps:

Add a downloadButton to a flexdashboard with runtime Shiny
When the download button is invoked, temporarily save non-reactive copies of any objects of interest
Knit a static HTML version of the flexdashboard, using the objects saved in the prior step
Use webshot::webshot or decapitated::chrome_shot to capture a .png image of the static dashboard³
Deliver this version to the client using the downloadHandler function

The solution requires two separate R documents. The process is as follows:

Dynamic Flexdashboard document


---
title: "Snapshot Example"
runtime: shiny
output:
  flexdashboard::flex_dashboard:
    orientation: rows
    vertical_layout: scroll
---

This is standard YAML header for a flexdashboard, including the command runtime: shiny to create a Shiny application from an R Markdown document. Let’s examine the next chunks:

gears = mtcars$gear %>% as.factor() %>% levels()
selectizeInput("grs", "GEARS",
               gears, selected = gears[1])
renderUI({
downloadButton("downloadFile", "Download")
})

In this chunk, we have built a selectizeInput object to choose the number of gears to select, as well as a downloadButton named ‘downloadFile’. Notice that we have wrapped the button in a renderUI() context. This is standard shiny / flexdashboard code.

mpgp = reactive({
  mtcars %>% filter(gear %in% input$grs) %>% ggplot(aes(x = hp, y = mpg)) + geom_point() + geom_smooth()
  
})
renderPlot({
  mpgp()
})

For veteran shiny programmers, this block may seem redundant. We’re creating a plot called mpgp in a reactive context and then rendering it. In a ‘standard’ shiny implementation, we would simply have wrapped the plot code in the renderPlot() context. However, as you will see below, we’re going to need to reuse the object mpgp.

output$downloadFile <- downloadHandler(filename = function() { 
return(paste('Cars', '.png', sep=''))
},
    content = function(file){
    to_save <- list(
    mpgg = mpgg(),
    mpgp = mpgp()
    )
      saveRDS(to_save, "config_data.RDS")
      rmarkdown::render("ShadowCars.Rmd")
      webshot::webshot("ShadowCars.html", file = file)
                        })

This chunk contains the secret sauce. When the downloadButton is invoked, the program saves all elements to appear in the downloaded report in a file called ‘config_data.RDS’. These elements are then used as arguments to render a shadow document, ‘ShadowCars.Rmd’. Finally, webshot::webshot converts the hidden document to a .png and delivers it to the client side.

The full dashboard looks like this:

Shadow Document

The key elements of the shadow document code are presented below:

---
title: "Snapshot Example"
output:
  flexdashboard::flex_dashboard:
    orientation: rows
---

Note that our YAML for the shadow document is similar to the dashboard, except it does not include runtime: shiny

The code blocks that render the objects created in the dynamic document are straightforward to the point of triviality; they simply reference the saved objects (which are retrieved using readRDS) and render them using whatever we choose, in this case kable() for the table and the native print() for the plot.

data <- readRDS("config_data.RDS")
data$mpgg %>% kable()

data$mpgp %>% print()

And of course, while mpgp was reactive in the original dashboard, it is static in the shadow doc.

The report delivered to the client is as follows:

Conclusion

In this short post, we have shown how to create a downloadable custom report by using existing tools to create a static flexdashboard. From the .png environment, the rendered image can be converted to other formats as required.

Harrison Schramm is an Operations Research Analyst at CANA Advisors. Aaron Berg is a Customer Success Representative at RStudio.↩
To recreate this minimal example for yourself, please download the files from this gist.↩
The webshot package requires phantomJS to be installed as an external dependency, but will work on all systems. decapitated will only work on systems with versions of Chrome that can operate in headless mode.↩

May New Package Picks

Fri, 23 Jun 2017 00:00:00 +0000

Two hundred and twenty-nine new packages were submitted to CRAN in May. Here are my picks for the “Top 40”, organized into five categories: Data, Data Science and Machine Learning, Education, Miscellaneous, Statistics and Utilities.

Data

angstroms v0.0.1: Provides helper functions for working with Regional Ocean Modeling System (ROMS) output.

bikedata v0.0.1: Download and aggregate data from public bicycle systems from around the world. There is a vignette.

datasauRus https://CRAN.R-project.org/package=datasauRus v0.1.2: The Datasaurus Dozen is a set of datasets that have the same summary statistics, despite having radically different distributions. As well as being an engaging variant on the Anscombe’s Quartet, the data is generated in a novel way through a simulated annealing process. Look here for details, and in the vignette for examples.

dwapi v0.1.1: Provides a set of wrapper functions for data.world’s REST API. There is a quickstart guide.

HURDAT v0.1.0: Provides datasets from the Hurricane Research Division’s Hurricane Re-Analysis Project, giving details for most known hurricanes and tropical storms for the Atlantic and northeastern Pacific ocean (northwestern hemisphere). The vignette describes the datasets.

neurohcp v0.6: Implements an interface to the Human Connectome Project. The vignette shows how it works.

osmdata v0.0.3: Provides functions to download and import of OpenStreetMap data as ‘sf’ or ‘sp’ objects. There is an Introduction and a vignette describing Translation to Simple Features.

parlitools v0.0.4: Provides various tools for analyzing UK political data, including creating political cartograms and retrieving data. There is an Introduction, and vignettes on the British Election Study, Mapping Local Suthorities, and Using Cartograms.

rerddap v0.4.2: Implements an R client to NOAA’s ERDDDAP data servers. There is an Introduction.

soilcarbon v1.0.0: Provides tools for analyzing the Soil Carbon Database created by Powell Center Working Group. The vignette launches a local Shiny App.

suncalc v0.1: Implements an R interface to the ‘suncalc.js’ library, part of the SunCalc.net’s project for calculating sun position, sunlight phases, moon position and lunar phase for the given location and time.

Data Science and Machine Learning

EventStudy v0.3.1: Provides an interface to the EventStudy API. There is an Introduction, and vignettes on Preparing EventStudy, parameters, and the RStudio Addin.

kmcudaR v1.0.0: Provides a fast, drop-in replacement for the classic K-means algorithm based on Yingyang K-Means. Look here for details.

openEBGM v0.1.0: Provides an implementation of DuMouchel’s Bayesian data mining method for the market basket problem. There is an Introduction, and vignettes for Processing Raw Data, Hyperparameter Estimation, Empirical Bayes Metrics, and Objects and Class Functions.

spacyr v0.9.0: Provides a wrapper for the Python spaCy Natural Language Processing library. Look here for help with installation and use.

Education

learnr v0.9: Provides functions to create interactive tutorials for learning about R and R packages using R Markdown, using a combination of narrative, figures, videos, exercises, and quizzes. Look here to get started.

olsrr v0.2.0: Provides tools for teaching and learning ordinary least squares regression. There is an Introduction and vignettes on Heteroscedascitity, Measures of Influence, Collinearity Diagnostics, Residual Diagnostics and Variable Selection Methods.

rODE v0.99.4: Contains functions to show students how an ODE solver is made and how classes can be effective for constructing equations that describe natural phenomena. Have a look at the free book Computer Simulations in Physics. There are several vignettes providing brief examples, including one on the Pendulum and another on Planets.

Miscelaneous

atlantistools v0.4.2: Provides access to the Atlantis framework for end-to-end marine ecosystem modelling. There is a package demo and vignettes for model preprocessing, model calibration, species calibration, and model comparison.

phylodyn v0.9.0: Provides statistical tools for reconstructing population size from genetic sequence data. There are several vignettes including a Coalescent simulation of genealogies and a case study using New York Influenza data.

Statistics

adaptiveGPCA v0.1: Implements the adaptive gPCA algorithm described in Fukuyama. The vignette shows an example using data stored in a phyloseq object.

BayesNetBP v1.2.1: Implements belief propagation methods for Bayesian Networks based on the paper by Cowell. There is a function to invoke a Shiny App.

RPEXE.RPEXT v0.0.1: Implements the likelihood ration test and backward elimination procedure for the reduced piecewise exponential survival analysis technique described in described in Han et al. 2012 and 2016. The vignette provides examples.

sfdct v0.0.3: Provides functions to construct a constrained ‘Delaunay’ triangulation from simple features objects. There is a vignette.

simglm v0.5.0: Provides functions to simulate linear and generalized linear models with up to three levels of nesting. There is an Introduction and vignettes for simulating GLMs and Missing Data performing Power Analysis and dealing with Unbalanced Data.

Utilities

checkarg v0.1.0: Provides utility functions that allow checking the basic validity of a function argument or any other value, including generating an error and assigning a default in a single line of code.

CodeDepends v0.5-3: Provides tools for analyzing R expressions or blocks of code and determining the dependencies between them. The vignette shows how to use them.

desctable v0.1.0: Provides functions to create descriptive and comparative tables that are ready to be saved as csv, or piped to DT::datatable() or pander::pander() to integrate into reports. There is a vignette to get you started.

lifelogr v0.1.0: Provides a framework for combining self-data from multiple sources, including fitbit and Apple Health. There is a general introduction as well as an introduction for visualization functions.

processx v2.0.0: Portable tools to run system processes in the background.

printr v0.1: Extends knitr generic function knit_print() to automatically print objects using an appropriate format such as Markdown or LaTeX. The vignette provides an introduction.

RHPCBenchmark v0.1.0: Provides microbenchmarks for determining the run-time performance of aspects of the R programming environment, and packages that are relevant to high-performance computation. There is an Introduction.

rlang v0.1.1: Provides a toolbox of functions for working with base types, core R features like the condition system, and core ‘Tidyverse’ features like tidy evaluation. The vignette explains R’s capabilities for creating Domain Specific Languages.

readtext v0.50: Provides functions for importing and handling text files and formatted text files with additional meta-data, including ‘.csv’, ‘.tab’, ‘.json’, ‘.xml’, ‘.pdf’, ‘.doc’, ‘.docx’, ‘.xls’, ‘.xlsx’ and other file types. There is a vignette

tangram v0.2.6: Provides an extensible formula system to implements a grammar of tables for creating production-quality tables using a three-step process that involves a formula parser, statistical content generation from data, and rendering. There is a vignette introducing the Grammar, a Global Style for Rmd, and duplicating SAS PROC Tabulate.

tatoo v1.0.6: Provides functions to combine data.frames and to add metadata that can be used for printing and xlsx export. The vignette shows some examples.

Visualizations

ContourFunctions v0.1.0: Provides functions for making contour plots. A vignette introduces the package.

mbgraphic v1.0.0: Implements a two-step process for describing univariate and bivariate behavior similar to the cognostics measures proposed by Paul and John Tuke. First, measures describing variables are computed and then plots are selected. The vignette describes the details.

polypoly v0.0.2: Provides tools for reshaping, plotting, and manipulating matrices of orthogonal polynomials. The vignette provides an overview.

RJSplot v2.1: Provides functions to create interactive graphs with ‘R’. It joins the data analysis power of R and the visualization libraries of JavaScript in one package There is a tutorial.

Analytics Administration for R

Wed, 21 Jun 2017 00:00:00 +0000

Analytic administrator is a role that data scientists assume when they onboard new tools, deploy solutions, support existing standards, or train other data scientists. It is a role that works closely with IT to maintain, upgrade, and scale analytic environments. Analytic admins have a multiplier effect - as they go about their work, they influence others in the organization to be more effective. If you are a data scientist using R, you might consider filling the role of analytic admin for your organization.

Consider the data scientist who wants to make R a legitimate part of their organization. This person has to introduce a new technology and help IT build the architecture around it. In this role, the data scientist – acting as an analytic admin – influences their entire organization.

The need for analytic admins

What organizations need analytic admins? Analytic admins are important for any organization that wants to:

Modernize their analytic tools
Take advantage of all their data
Build analytic products and applications
Develop a best-in-class data science team

Despite the fact that the need for analytic admins is pervasive in industry, companies rarely list it as a dedicated role. Instead, they require teamwork between data science and IT operations, or they may require data scientists to function as their own admins. But the need is real. Most organizations need help bridging the gap between data science and IT. If you see an opportunity to function in the capacity as an analytics admin, I suggest you take it.

Analytic admins typically have to train themselves and carve out their own career. It is common for data scientists who operate as analytic admins to feel as though they are in no-man’s land. It is natural to feel lost between the worlds of data science and information technology. As someone who had been there, I can say the feeling is disorienting. However, I can also say the value of that position is tremendous. If you feel like you are operating in no-man’s land as you function in this role, just know you are exactly where you need to be.

R tooling and integration

At RStudio, we think about doing data science as a development process that begins with accessing and understanding your data, and then communicating your results. This process is thoroughly explained in the book R for Data Science, by Wickham and Grolemond.

RStudio builds open-source and enterprise-ready products to help you do data science in R. These products include the RStudio IDE, RStudio Connect, and Shiny Server. These are designed to work with open-source R packages like Shiny, R Markdown, and the Tidyverse.

Most of the software that RStudio makes is open source, but enterprises often require additional professional features. Common Professional features are security, authentication, high availability, administration, and load balancing.

R is also used with production environments for hosting web applications, exposing APIs, and automating workflows. R is sometimes integrated into other systems such as data warehouses, Hadoop, and Spark. The role of the analytic admin is to provide tooling for data scientists, as well as to integrate R into production systems.

Linux and R

RStudio products run on Linux, so understanding Linux will help you become self-sufficient, use R with other systems, and build better solutions. We will talk more about what you can do with Linux commands in an upcoming blog post.

There are many resources for learning Linux online. Here is just one offered by the Linux Foundation. Analytics admins need to know how to navigate (e.g., cd, pwd, ls), install Linux packages (e.g., apt-get install), and execute commands as root (e.g., sudo). Also important are tab completion, keyboard shortcuts, and text editors (e.g., vim, nano).

Did you know you can execute basic Linux commands from inside RStudio Server using the Tools > Shell option? You can also execute Linux commands inside the R console with the system function.

Another major benefit of learning Linux is the ability to administer production systems that run with Shiny Server, and the ability to deploy Shiny web applications into production.

Running Shiny in production

There is a growing trend in using Shiny web apps in production analytic workflows. The vibrant Shiny community now spans all verticals including pharmaceuticals, high technology, and finance. For many organizations, adopting Shiny is their first experience in running R in production.

Production environments that depend on Shiny also need analytic admins who can deploy and support these applications. For example, some organizations now have complex Shiny applications that serve hundreds of end users over a cluster of load-balanced Shiny Servers. These applications often go through a standard development > test > production deployment process. New tools are being built for correctness testing and load testing in Shiny. RStudio and other platform vendors are making significant investments in building architectures - like Shiny Server and RStudio Connect - that will help Shiny grow over the long term.

The growth of Shiny opens an opportunity to analytic admins who want to make analytic content available to a wide audience. Shiny apps allow end users who know nothing about R to take advantage of the power of the R programming language. They have the potential to influence decision-makers who can take actions and see results based on the work data scientists share with them. There is an immediate need for analytic admins who understand Shiny and can help support environments that depend on Shiny.

Getting started: Installing RStudio Server

A great way to get started learning analytics administration is to build your own open source RStudio Server on Linux. Building an RStudio Server by hand is the analytic admin equivalent of the Jedi building their own light sabers. It’s a core skill, so you should be able to do it yourself no matter what.

An easy way to get started with RStudio Server is to set it up on Ubuntu with Amazon Web Services. AWS even has an instruction guide for running R on AWS. The core commands of the install are the following four lines of code (note: this installs RStudio Server version 1.0.143).

$ sudo apt-get install r-base
$ sudo apt-get install gdebi-core
$ wget https://download2.rstudio.org/rstudio-server-1.0.143-amd64.deb
$ sudo gdebi rstudio-server-1.0.143-amd64.deb

Of course, your installation is going to require more than just installing RStudio Server. You will probably want use the CRAN repository, install Linux dependencies, add users, and manage R packages. Here is a complete script I used to set up RStudio Server on a simple AWS AMI (ami-efd0428f) using a T2-medium instance. I included instructions from this document on how to install R from CRAN. I also opened port 8787 in my AWS security group so I could log into RStudio Server via my web browser.

### Simple RStudio Server Install
### Based on AWS image: ami-efd0428f
 
## Install R from CRAN repository
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
$ sudo add-apt-repository 'deb [arch=amd64,i386] https://cran.rstudio.com/bin/linux/ubuntu xenial/'
$ sudo apt-get update
$ sudo apt-get -y install r-base
 
## Install RStudio Server version 1.0.143
$ sudo apt-get install gdebi-core
$ wget https://download2.rstudio.org/rstudio-server-1.0.143-amd64.deb
$ sudo gdebi rstudio-server-1.0.143-amd64.deb
 
## Add a new user
$ sudo useradd -m myuser
$ sudo passwd myuser
 
## (Optional - may take time) Install common Linux dependencies
$ sudo apt-get -y install libcurl4-openssl-dev openssl libssl-dev
$ sudo apt-get -y install texlive texlive-latex-extra libxml2-dev
 
## (Optional - may take time) Install common R packages
$ sudo Rscript -e 'install.packages("shiny", repos = "http://cran.rstudio.com/")'
$ sudo Rscript -e 'install.packages("tidyverse", repos = "http://cran.rstudio.com/")'
 
## Point your browser to <AWS-instance-IP>:8787

If you don’t want to install RStudio Server from scratch, there are other ways to get started. One is to use a community AMI like this one. Another is to use the AWS Marketplace to install RStudio Server Pro with 1-Click Launch.

Conclusion

Installation is just the first step to administering R. You should also consider the topics of authentication, security, scale, integration, hardware sizing, and configuration. Systems administrators have to do a lot of their own training, and analytic admins are no different. Fortunately, there are plenty of references to help you get started. Here are a few useful references for learning analytic administration for R, RStudio, and Shiny.

RStudio Products

Authentication and security

Managing R Packages

Shiny Server

Mapping Quandl Data with Shiny

Wed, 14 Jun 2017 00:00:00 +0000

Today, we are going to wrap our previously built Quandl/world map Notebook into an interactive Shiny app that lets users choose both a country and a data set for display. As usual, we did a lot of the heavy lifting in the Notebook to make our work more reproducible and our app more performant. The final app is available here.

Devotees of this Reproducible Finance blog series will note similarities to this Shiny app, but today’s app will have different and richer functionality.

First, we are going to be pulling in data from Quandl, so we won’t be using ticker symbols, but rather will be using country codes plus data set codes. That would allow us to open this app to the vast number of data sets available via Quandl.

Second, the user will be able to use the sidebar to select different data sets, rather than being restricted to a country ETF. For example, the user might want to select GDP-per-capita, or the exchange rate, or any data set that we wish to make available. Our app today will include 6 economic indicators.

Third, we will display a chart using highcharter instead of dygraphs. There’s not a substantive reason to be doing this, beyond that it’s a good chance to start exploring highcharter, which is a popular tool in the financial world.

Lastly, we’ll port raw data to a data.table and include a few buttons for easy download, in case our end users want to reproduce our charts or import data for their own work. We are going to use the as_tibble function from the fantastically useful new tidyquant package to facilitate our data.table. We won’t have space to cover the data download section in this post, but all the code is available in the live app.

Without further ado, let’s get to it.

The first line of substance in this app loads in the leaflet map we constructed in our Notebook. We named the object leaf_world and saved it in a file called wdiMapData.RDat. If you want to refresh your memory on how we did that, have a look back at the previous post. Apologies if that sounds tedious, but hopefully it emphasizes the workflow of doing the heavy map-building in the Notebook when possible. If you or your team ever want to use that map as the basis for another Shiny app, that Notebook will be convenient to reuse.

To load our map, we run the following:

# Load the .RDat file where the leaflet map is saved.
load('wdiMapData.RDat')

Now we can access whatever R objects were saved in that file and, in this case, that means one R object called leaf_world. That might not be a great name, but it’s the leaflet map of the world we built in the Notebook. When we are ready to build the map in our Shiny app, we’ll simply render that object.

Before we get to the map, though, let’s construct a sidebar where the user can choose which data set to display. We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators WDI. The Quandl code for WDI is WWDI and thus we’ll append WWDI/ to each data set call - but note that this will not appear in the sidebar because we won’t actually pass any data to Quandl there. We have to wait for the user to click on a county, and thus will handle the data passing in a future code chunk. This sidebar has one purpose: for the user to select the code for the economic indicator.

We want to give the user a drop-down of choices, but we don’t want the choices to be the Quandl codes. We want the choices to be the familiar names of the data sets. In other words, the user will see as a choice ‘GDP Per Capita’, instead of the Quandl code _NY_GDP_PCAP_KN. For that reason, we’ll first create an object called dataChoices that holds our name-value pairs. The name is the familiar title of the time series - for example, ’GDP Per Capita` - and the value is the Quandl code.

dataChoices <- c("GDP Per Capita" = "_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "_FR_INR_RINR",
                  "Exchange Rate" = "_PX_REX_REER",
                  "CPI" = "_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "_SL_TLF_ACTI_ZS")

Next, in our selectInput statement, we will set choices equal to dataChoices, which will allow the user to see intuitive names but choose the Quandl codes.

selectInput("indicatorSelect", 
            "Choose an economic indicator",
            # Give the user a choice from the object we created above.
            choices = dataChoices,
            selected = "GDP Per Capita")

Our simple sidebar is finished, and the end result is that when a user makes a choice, we have an input reactive with the value of the data set code. Let’s put that code aside for a minute and turn our attention to the map.

Remember, we did the map-building work on this in our Notebook, then loaded the object in the setup code chunk. That leaves us with a simple call to renderLeaflet() to pass it leaf_world.

leafletOutput("map1")

output$map1 <- renderLeaflet({
  leaf_world
  })

Alright, that wasn’t too cumbersome, and perhaps the simplicity of that step makes the hard work in the previous Notebook a bit more tolerable.

And now the fun part, wherein we build the machinery to let a user click the map and grab data from Quandl. We will proceed in four steps:

capture the country code of the country that gets clicked
call the economic time series that the user selected in the sidebar
paste the country code and the time series together to create one data set code
pass that one data set code to Quandl and import the time series

On to step 1, capturing the clicked country.

clickedCountry <- eventReactive(input$map1_shape_click, {
  return(input$map1_shape_click$id)
})

It’s same process we used in this post, but there is one difference. Let’s review what happened here.

Recall that we set layerID = ~iso_a3 when we built the map in our Notebook. That’s because Quandl appends the iso_a3 country codes to the data set code. In other words, if we want GDP-per-capita for Brazil, we need the Brazil country code ‘BRA’ so we can later append it to the GDP-per-capita code ’_NY_GDP_PCAP_KN’.

In the code chunk above, we used an observeEvent function to capture the layerID of whatever shape a user clicks, and in the Notebook, we had set that ID to be the ~isa_a3 code. When a user clicks Brazil, our reactive captures the country code ‘BRA’.

Next, we need to append that country code to the data set code. Recall that the user chose this code in our sidebar. We just need to grab his selection via another reactive.

# Nothing fancy here, a reactive value from our sidebar input.
indicatorValue <- reactive({input$indicatorSelect})

Let’s pause and take inventory of what we have captured thus far: the country code from the click on the map is stored as clickedCountry(), and the data set code from the sidebar is stored as indicatorValue(). Now we want to paste those together and pass them to Quandl. We’ll do that via another reactive, but we are also going to pass that pasted chain to Quandl in the reactive. Thus, the chunk below has one reactive that does two things.

First, it pastes together our inputs to form one data set code. Second, it passes that code to Quandl and returns the desired time series.

Notice that when we paste the codes together, we start with WWDI/. That’s because all of our data comes from the WWDI data source on Quandl. We are not letting the user choose a different data source. We could have done so in the sidebar, but consider how that would have complicated our data set code reactive input in the sidebar.

Back to our data import chunk wherein we create our data set code by pasting inputs together and then pass them to Quandl:

countryData <- reactive({
  
  # WWDI is the World Bank data set. 
  # We aren't giving the user a choice of data sources, but we could.
  dataSet <- paste("WWDI/",
                   # The country that was clicked.
                   as.character(clickedCountry()),
                   # The time series that was chosen in the sidebar.
                   as.character(indicatorValue()), 
                   sep = "")
  # Now pass that pasted data set code object to Quandl.
  Quandl(dataSet, type = "xts")
})

We now have an object called countryData that is an xts object imported from Quandl. It holds the time series (1) from the WWDI data source, (2) for the country that was clicked, (3) for the economic indicator that was chosen in the sidebar

From a data import perspective, we’re done. We could fit a model to this data, run forecasts, etc., but I am going to head straight to visualization using highcharter.

As mentioned at the outset, dygraph would have done a fine job here, but highcharter seems to be a more popular visualization tool in the financial industry, and that’s a good reason to start exploring its capabilities.

The code chunk below might look heavy, but most of the lines are for aesthetics, creating a nice title and removing scroll bars that seem clunky to me. The key substantive statements below are highchart(type = "stock") and hc_add_series(countryData()).

The first statement tells highcharter to use its built-in stock format (this is good for graphing all sorts of economic time series, not just stocks), and the second passes our data object to the graphing function hc_add_series. It’s a well-named function because it adds a series to the chart. If we wanted to include another series, we would use that same function and pass in another xts object.

output$highchart <- renderHighchart({
  
  # Make a nice title for the chart.
  # The next three linese are purely aesthetic to that our graph has an intuitive
  # title. 
  indicatorName1 <- names(dataChoices[dataChoices == input$indicatorSelect])
  countryName1 <- countrycode(as.character(clickedCountry()), "iso3c", "country.name")
  title <- paste(countryName1, indicatorName1, sep = " ")
  
  # Call highchart and ask nicely for it to use the built-in 'stock' format.
  highchart(type = "stock") %>% 
    hc_title(text = title) %>%
    # Pass our time series object countryData to highcharter function hc_add_series.
    hc_add_series(countryData(), name = title) %>%
    # I don't like the look of the navigator/scrollbar in a small space, but others might. 
    # Change these to enabled = TRUE and check out the results.
    hc_navigator(enabled = FALSE) %>% 
    hc_scrollbar(enabled = FALSE)
})

That’s all for today. This post completes what we can retrospectively consider our 3-part series of Quandl Shiny apps. The other two posts are here and here. There are definitely interesting ways to combine the functionalities in these apps, but I’ll leave that to the awesome community of Shiny developers.

See you next time, when we visit the wonderful world of portfolio theory and volatility!

What is the tidyverse?

Thu, 08 Jun 2017 00:00:00 +0000

Last week, I had the opportunity to talk to a group of Master’s level Statistics and Business Analytics students at Cal State East Bay about R and Data Science. Many in my audience were adult students coming back to school with job experience writing code in Java, Python and SAS. It was a pretty sophisticated crowd, but not surprisingly, their R skills were stitched together in a way that left some big gaps. Many for example, didn’t fully understand the importance of CRAN Task Views as curated source for the best packages to support their work in machine learning, time series and the other areas of Statistics they were studying. So, it made sense that even though ggplot2 and dplyr were mentioned in some of the student’s questions, a faculty member present asked: “What is the tidyverse?” in an attempt to cover an area that he knew was one of those gaps.

There is an incredible amount of good material available online about the tidyverse, and I will point to some of that below. But here, I’ll elaborate on the answer I gave during the Q&A.

The Basics

The tidyverse is a coherent system of packages for data manipulation, exploration and visualization that share a common design philosophy. These were mostly developed by Hadley Wickham himself, but they are now being expanded by several contributors. Tidyverse packages are intended to make statisticians and data scientists more productive by guiding them through workflows that facilitate communication, and result in reproducible work products. Fundamentally, the tidyverse is about the connections between the tools that make the workflow possible.

It is also the case that the tidyverse is work in progress. You can find the current state of development at tidyverse.org. Clicking on the icon for each package on this website will bring you to detailed documentation for each package. The following figure illustrates a canonical data science workflow, and shows how the individual packages fit in.

If you have some experience with R, you ought to be able to jump right into the online documentation and find your way around. If you are new to R, and maybe new to data science as well, you can’t do any better than work through the book R for Data Science by Hadley Wickham and Garrett Grolemund.

Advantages

The advantages of the tidyverse include consistent functions, workflow coverage, a path to data science education, a parsimonious approach to the development of data science tools, and the possibility of greater productivity.

Consistency

The tidyverse aspires to consistency on multiple levels. Examples of “micro”-level consistency include the convention of having variable names glide along in snake_case, and the signatures of tidyverse functions follow a regular pattern. (The first formal argument is always a data frame that provides the function’s input.) Higher-level consistency includes the idea of tidy data - a data frame where each row is an observation and each column contains the value of a single variable - and the way in which the pipe operator, %>%, channels the flow of tidy operations. Under the covers, there are even more levels of structure that aid the pursuit of consistency, including uniform standards for package organization, testing procedures, coding style, etc.

Coverage

The workflow shown above, with tidyverse packages associated with the various steps, or more usually rendered with the following iconic tidyverse diagram, preceded and motivated the development of the tidyverse.

It is an abstraction of the canonical data analysis workflow that has always guided statisticians, but now informs data science as a map to organize, streamline, automate and optimize the various processes involved. The fact that tidyverse packages are associate with all of the processes indicates that it comprises enough fundamental building blocks to support the entire end-to-end workflow for a variety of data sources and analysis goals. Moreover, the relatively recent addition of the purrr package extends the reach of the tidyverse to support the creation of new data science tools.

Critical Mass

A great strength of the R language is that with over ten thousand user contributed packages on CRAN alone, it has a lot to offer. This kind of organic growth makes it inevitable that packages will offer overlapping features. Users have to make decisions about which package, or suite of packages, they will make the effort to learn. For many users, the decision hinges on whether a collection of packages visibly supports important work. Does it have a large community of users and is it backed by committed developers and maintainers? All of the signals indicate that (at least, among R-using data scientists) the tidyverse has reached critical mass. For example, the tidyverse package has been downloaded 50,000 times in the last month. Moreover, it appears that tidyverse principles are propagating into other application areas. The tidyquant package, for example, is a serious attempt to bring tidy principles to Finance.

Education

A typical R user gets involved with R in the first place through a desire to compute in some quantitative field. The path to R competency frequently begins with mastering a small number of relevant functions. Statisticians, for example, may learn to read in data from a .csv file and build a linear regression model with lm(). Financial analysts may be introduced to R through a package like quantmod, which enables a new user to do quite a bit of real work. The tidyverse provides the path of least resistance, or “pit of success”, for data scientists interested in R. For example, the small number of compatible building blocks provided by dplyr enable even a relatively inexperienced user to tidy up a messy data set quickly and easily.

Parsimony

The packages and functions of the tidyverse are the result of trial-and-error experimentation carried out over several years, to find a minimum set of functions that are sufficient to enable the canonical data science workflow. Those of you who have been following Hadley’s work will remember cast() and melt() from the reshape and reshape2 packages, and ddply() from the plyr package, which were early attempts to find a vocabulary for wrangling data frames. After several attempts to identify and construct the most advantages set of primitive building blocks, the tidyverse has matured into its present form.

Productivity

Hadley has always been clear that a major goal for the tidyverse - and indeed much of his work over the years - has been to help anyone who needs to analyze data work productively, and he is fond of quoting Hal Abelson: “Programs must be written for people to read and only incidentally for machines to execute”. My take is that a major reason for the popularity of tidyverse packages is that they help people achieve and maintain flow in their daily data analysis work.

Some Limitations

The tidyverse, of course, is not without limitations. Some of these are due to factors that are beyond the designer’s control, and others may be by design. Limitations of the first kind may arise from a lack of agreement as to whether some data can be, or should be, forced into a “rectangular” data structure. For example, although there are scientists and data scientists working in genomics that are fans of dplyr and ggplot2, much of the work done in the Bioconductor Project remains outside of the tidyverse workflow.

The need for the close coordination of tidyverse packages produces some limitations of the second sort. There are many high-quality R packages that are of great use to data scientists, but based on design goals that differ from those of the tidyverse. There will always be more than the tidyverse.

A Bigger Picture

A powerful, but perhaps under-appreciated, capability of the R language is its ability to support the design and programming of Domain Specific Languages. Joe Cheng highlighted this feature in an interview he gave to R Views last year. He described R as being “shockingly close to LISP”, of which Joe says: “it’s almost like you change the language itself to be a DSL for whatever problem you’re trying to solve … the elegant, terse syntax of dplyr and the pipe operator are possible because of how malleable a language R is, and how great it is for writing DSLs in it.”

So, from a wider perspective, the tidyverse can be seen as sub-dialect of the R language that is evolving to express ideas and tasks inherent in Data Science workflows and software development. This dialect may not be for everyone, but it does seem to be helping many R fluent data scientists frame their conversations.

Some Resources

The following are some resources that you may find helpful in learning and mastering the tidyverse.

The video of Hadley Wickham’s Keynote address at rstudio::conf 2017
The slides corresponding to the above video
R for Data Science by Hadley Wickham and Garrett Grolemund
Text Mining with R
Getting Started with the Tidyverse in R

April New Package Picks

Tue, 30 May 2017 00:00:00 +0000

Here are my picks for the “Top 40” new packages submitted to CRAN in April 2017. These selections, which were culled from 208 submissions, are organized into four categories: Data, Finance, Statistics and Utilities. The number of entries in the Data and Utilities categories reflect the initiatives of R developers to connect to external resources.

Data

comtradr v0.0.1: Provides functions to extract country-level shipping data for a variety of commodities data from the United Nations Comtrade API.

countyfloods v0.0.2: Provides access to county-level United States flood data using United States Geological Service (USGS) API. The vignette provides examples.

EdSurvey v1.0.6: Provides functions to access and analyze survey and assessment data from the National Center for Education Statistics (NCES), including the National Assessment of Educational Progress (NAEP. There are is an Introduction, and vignettes for Advanced Data Manipilation, NAEP Accommodations, Statistical Methods, and Achievement Levels.

RNRCS v0.1.1: Provides functions to download data from the Natural Resources Conservation Service (NRCS) for sites in the Soil Climate Analysis Network (SCAN), and Snow Telemetry (SNOTEL and SNOLITE) networks. Metadata can be returned for all sites in the NRCS’ Air and Water Data Base (AWDB).

roadoi v0.2: Implements a web client interface to oaDOI, a service providing free full-text access to academic papers through various sources including Crossref, Bielefeld Academic Search Engine (BASE), and the Directory of Open Access Journals (DOAJ). The vignette shows how to use it.

rsoi v0.2.1: Provides functions to download Southern Oscillation Index data and Oceanic Nino Index data from NOAA.

rsolr v0.0.4: Implements an API for querying Apache Solr databases. There is a vignette.

tubern v0.1.0: Implements an R client for the YouTube Analytics and Reporting API. The vignette shows how to use the API.

wikitaxa v0.1.4: Provides access to taxonomic information from Wikipedia, Wikicommons, Wikispecies, and Wikidata. There is a vignette.

Finance

partialCI v1.1.0: Provides functions for estimating, testing, and simulating partially cointegrated time series. Clegg et al. provide an in-depth discussion of the package functionality, as well as illustrating examples in the fields of finance and macroeconomics. The vignette provides a guide to the package.

PortfolioOptim v1.0.3: Provides two functions for financial portfolio optimization by linear programming.

Statistics

bigKRLS v1.5.2: Provides functions for Kernel-Regularized Least Squares that are optimized for speed and memory usage. The author’s website has sample code and more. A vignette shows the basics.

brglm2 v0.1.4: Provides functions for achieving bias reduction in Generalized Linear Models via solving the mean bias-reducing adjusted score equations in Firth and Kosmidis and Firth, or the median bias-reduction adjusted score equations in Kenne et al., or through the direct subtraction of an estimate of the bias of the maximum likelihood estimator from the maximum likelihood estimates as in Cordeiro and McCullagh. There are vignettes on bias reduction and on Detecting Separation.

casebase v0.1.0: Implements the case-base sampling approach of Hanley and Miettinen, Saarela and Arjas, and Saarela, for fitting flexible hazard regression models to survival data with single event type or multiple competing causes via logistic and multinomial regression. There are vignettes on Competing Risk Analysis, Population Time Plots, and Casebase Sampling.

CausalImpact v1.2.0: Implements a Bayesian approach to causal impact estimation in time series, as described in Brodersen et al. Look here to get started, and also at the vignette.

To estimate a causal effect, specify which period causal should be used for training the model (pre-intervention period), and which period for computing a counterfactual prediction (post-intervention period).

ggeffects v0.1.0: Provides functions to compute marginal effects at the mean or average marginal effects from statistical models, and returns the result as tidy data frames. The vignette shows how to use it.

glmnetUtils v1.0.2: Provides a formula interface for the glmnet package. The vignette shows how to use it.

lspline v1.0-0: Implements linear splines with convenient parameterizations such that (1) coefficients are slopes of consecutive segments, or (2) coefficients are slope changes at consecutive knots. The vignette gives examples.

marcher v0.0-2: Provides tools for likelihood-based estimation, model selection, and testing of two- and three-range shift and migration models for animal movement data as described in Gurarie et al.. The vignette shows how to use the model and create visualizations.

Simulation of Range Shift Processes

Meiosis v1.0.2: Provides tools for simulation of meiosis in plant breeding research. There is a vignette.

MultiVarSel v1.0: Implements a novel variable selection approach in the multivariate framework of the general linear model, taking into account the dependence that may exist between the columns of the observations matrix. For details, see the paper Perrot-Dockes et al.. The vignette shows an example.

mrbsizeR v1.0.1: Implements a method for the multiresolution analysis of spatial fields and images to capture scale-dependent features based on scale space smoothing. The vignette provides an example.

networktools v1.0.0: Includes assorted tools for network analysis, including functions for calculating impact statistics (global strength impact, network structure impact, edge impact), and for calculating and visualizing expected influence. The vignette describes the impact statistics.

philentropy v0.3.3: Provides functions to compute forty-six optimized distance and similarity measures for comparing probability functions that are useful in a broad range of scientific disciplines. There is an Introduction and vignettes for Distances Measures and Information Theory.

plink v1.5-1: Provides functions to facilitate the linking of mixed-format tests for multiple groups under a common item design using unidimensional and multidimensional IRT-based methods. The very thorough vignette presents the theory.

RLT v3.1.0: Random forest with a variety of additional features for regression, classification, and survival analysis.

spup v0.1-0: Implements for analyzing uncertainty propagation in spatial environmental modeling using the methodologies of Heuvelink et al. and Brown and Heuvelink. There are vignettes on spatially cross-correlated variables, spatially variable sd, uncertainty propagation, and categorical variables.

Utilities

ajv v1.0.0: Provides a thin wrapper around the ajv JSON validation package for JavaScript. There is a vignette to get you started.

aws.s3 v.3.1: Implements a simple client package for the Amazon Web Services (AWS) Simple Storage Service (S3) REST API.

cranlike v1.0.0: Provides a set of functions to manage ‘CRAN’-like repositories efficiently.

d3Tree v0.1.0: Provides functions to create and customize interactive collapsible ‘D3’ trees using the D3 JavaScript library and the htmlwidgets package, which can be used directly from the R console, from RStudio, in Shiny apps, and in R Markdown documents. Look here for examples.

library(d3Tree)

  d3tree(list(root = df2tree(rootname = 'book',
                             struct = stan.models),
                             layout = 'collapse'))

dataMeta v0.1.0: Provides functions to create a basic data dictionary and append it to the original dataset’s attributes list. The vignette shows how it works.

dparser v0.1.3: Implements a scannerless GLR parser based on the Tomita algorithm.

glue v1.0.0: Provides an implementation of interpreted string literals, inspired by Python’s Literal String Interpolation, Docstrings, and Julia’s Triple-Quoted String Literals. The README shows how they work.

icosa v0.98.1: Employs triangular tessellation to refine icosahedra defined in 3D space. The vignette provides the details.

leaflet.minicharts v0.2.0: Allows users to add and modify small charts on interactive maps created with the leaflet package

RcppQuantuccia v0.0.1: Provides QuantLib bindings for R using ‘Rcpp’ and the header-only Quantuccia variant.

shinymaterial v0.2.1: Allows shiny developers to incorporate UI elements based on Google’s Material Design. See the package website more information.

unitizer v1.4-2: Provides functions to simplify regression tests by comparing objects produced by test code with earlier versions of those same objects. There is an Introduction, and vignettes for the interactive environment, Reproductble Tests, unitizeR details, and Miscellanea.

unix v1.3: Provides bindings to various Unix system utilities.

Civic Data Wrangling: in R and on data.world

Fri, 26 May 2017 00:00:00 +0000

One of the most valuable things I have learned working on Data for Democracy’s Medicare drug spending project has been the value of collaborative tools. It has been my first in-depth experience using Github collaboratively, for one, but it has also introduced me to data.world.

data.world is an intuitive way to store, organize, explore, and visualize individual data files, making them more visible to and usable for anyone who’s interested. Using the data.world R package, I was able to collect the data I needed for one dashboard, create and publish a derived dataset, and build a Shiny dashboard that pulls live data from the site — all with complete visibility to my D4D teammates, who can now directly and easily access both the original and derived datasets.

The Beginning

Back in December, I started seeing posts about Data for Democracy floating around, and the idea immediately piqued my interest. I joined up not knowing what I would be doing exactly.

One of the ideas among pilot projects suggested by D4D leader Jonathon Morgan was looking at Medicare spending. Specifically, starting with the prescription spending data for 2011-2015 released by the Centers for Medicare and Medicaid Services. Out of all the original project ideas (check out the current and growing list of projects and be amazed), this one was most in my wheelhouse, and I jumped into the discussions to see what might happen.

D4D started developing some leadership structure, and Matt Gawarecki and I were asked to lead the drug spending project. We began by tidying the Medicare Part D spending data available from CMS.gov, which was straightforward enough (thanks, CMS!), and initially stored all that on our Github repo.

Soon, our A+ team of volunteers found data on therapeutic classes, lobbying expenditures, manufacturers, and more. As we looked for a better place to store and collaborate on our datasets, D4D projects started to leverage data.world. We decided that data.world was a good place for our data, too.

At that point, one of the first things we wanted to do was look at spending and prescribing patterns for specific medications, which would help us see what we were dealing with and generate some ideas for future analysis. James Porter drafted a Shiny app for visualizing users, claims, and costs for the most commonly prescribed medications over the five years of data available. I built on top of James’ work, and the full source code is available on Github.

The Data

Setup and Initial Analysis

Once we got the CMS spending data pulled out of its original Excel format, tidied, and uploaded, I used the data.world package to query the schema metadata and see exactly how tables were named, choose the right files, and pull them directly into R.

(To run this, you’ll need your data.world API token, which I saved as DW_API in an .Renviron file to make my code shareable.)

library(tidyverse)
library(data.world)

# API token is saved in .Renviron

data.world::set_config(cfg_env("DW_API"))

# Datasets are identified by their URL
drugs_ds <- "https://data.world/data4democracy/drug-spending"

# List tables
data_list <- data.world::query(
  qry_sql("SELECT * FROM Tables"),
  dataset = drugs_ds)

# data_list is a tbl_df with two columns: tableID and tableName.
data_list$tableName

##  [1] "FDA_NDC_Product"             "Data"                       
##  [3] "Methods"                     "Variables"                  
##  [5] "Pharma_Lobby"                "atc-codes"                  
##  [7] "companies_drugs_keyed"       "drug_list"                  
##  [9] "drug_uses"                   "drugdata_clean"             
## [11] "drugnames_withclasses"       "lobbying_keyed"             
## [13] "manufacturers_drugs_cleaned" "meps_full_2014"             
## [15] "spending-2011"               "spending-2012"              
## [17] "spending-2013"               "spending-2014"              
## [19] "spending-2015"               "spending_all_top100"        
## [21] "usp_drug_classification"

Yearly Data Consolidation

The CMS data is stored in the five spending-201x tables, one per year. I read those directly from data.world by writing a simple SQL query identifying those table names, then used dplyr and purrr to quickly add a year variable to each and combine them:

# Function to read in a single year's CSV from data.world and add year
get_year <- function(yr) {
  data.world::query(qry_sql(paste0("SELECT * FROM `spending-", yr, "`")),
    dataset = drugs_ds)[,-1] %>%
    ## First column is a row number; don"t need that
    mutate(year = yr)
}

# Read in and combine all years' data
spend <- map_df(2011:2015, get_year)

Most Popular Generic Drugs

To keep runtime and drop-down menus reasonable, we currently only include the 100 drugs with the most users over the five-year period.

The app allows users to select any one of those to look at people-related values (e.g., the number of users or the number of claims) and spending-related values (e.g., total costs, average out-of-pocket costs for users receiving and not receiving low-income subsidies).

The datasets include one record per drug brand name.

Each generic drug may appear with several brand names. Metformin HCL, for example, has over 35 brand names - Glucophage XR, Glumetza, etc. To include them in my analysis, I calculated the total number of users and units per year for each generic, added those values to the original dataset, then selected the 100 most common generics during that five-year period.

# Add a row for each generic with overall summaries of each variable ----------
spend_overall <- spend %>%
  group_by(drugname_generic, year) %>%
  summarise(
    claim_count = sum(claim_count, na.rm = TRUE),
    total_spending = sum(total_spending, na.rm = TRUE),
    user_count = sum(user_count, na.rm = TRUE),
    unit_count = sum(unit_count, na.rm = TRUE),
    user_count_non_lowincome = sum(user_count_non_lowincome, na.rm = TRUE),
    user_count_lowincome = sum(user_count_lowincome, na.rm = TRUE)
  ) %>%
  mutate(
    total_spending_per_user = total_spending / user_count,
    drugname_brand = "ALL BRAND NAMES",
    ## Add NA values for variables that are brand-specific
    unit_cost_wavg = NA,
    out_of_pocket_avg_lowincome = NA,
    out_of_pocket_avg_non_lowincome = NA
  ) %>%
  ungroup()

# Select top 100 generics by number of users across all five years ------------
by_user_top100 <- group_by(spend_overall, drugname_generic) %>%
  summarise(total_users = sum(user_count, na.rm = TRUE)) %>%
  arrange(desc(total_users)) %>%
  slice(1:100)

# For top 100 generics, add ALL BRAND NAMES rows to by-brand-name rows --------
spend_all_top100 <- bind_rows(spend, spend_overall) %>%
  filter(drugname_generic %in% by_user_top100$drugname_generic) %>%
  arrange(drugname_generic)

Saving The Work

Once I had the final dataset with just our top 100, I uploaded it directly to data.world as a CSV via the dwapi package, which is bundled, and loaded automatically, with data.world:

# Write final file to data.world using package ------------------------------------

# data.world's REST API methods are available via the dwapi package
dwapi::upload_data_frame(
  dataset = drugs_ds,
  file_name = "spending_all_top100.csv",
  data_frame = spend_all_top100
)

The App

Now that the initial data management is done, the app’s global.R reads the “top 100” CSV directly from data.world, and we’re all set.

# Read in datasets used for all plots ----------------------------------------
library(data.world)

# API token is saved in .Renviron (DW_API)
data.world::set_config(cfg_env("DW_API"))

# Read in dataset from data.world
drug_costs_everything <-
  data.world::query(
    qry_sql("SELECT * FROM `spending_all_top100`"),
    dataset = "data4democracy/drug-spending")

The app is available to the public and can be accessed at https://jenniferthompson.shinyapps.io/shinydashboard-medicared/.

What’s next?

The current app only uses a few of the datasets that are available on data.world, but we hope to be able to classify drugs by therapeutic class (e.g., all diabetes medications) soon. Once we have that information stored on data.world, we will be able to read data from both tables, join them, and give users the option to visualize some information by therapeutic class as well as by individual medication.

This is harder than one might think, and we welcome expertise from those who might try!

Finally, if you have tried data.world and have used it for your projects, make sure to share your stories, suggestions and complaints with the team.

About the author

Jennifer Thompson is a well-rounded conversationalist and a standup woman who spends her days as a biostatistician and avid R user. Jennifer is @jthompson on data.world and @jent103 on Twitter.

Growth of DataFest over the years

Wed, 24 May 2017 00:00:00 +0000

In a previous post, I introduced DataFest and how one can streamline the organization of this event using Google Forms and tools from the tidyverse. In this post, I’ll walk through building a Shiny app that demonstrates the growth of DataFest over the years, both in terms of host locations and participating institutions, as well as in terms of the number of students who participated in each event.

Here is a list of all packages used in this article:

library(tidyverse)
library(googlesheets)
library(devtools)
library(ggmap)
library(stringr)
library(leaflet)

The data were contributed by the event organizers, and were collected using a Google Form.

To begin, the data are read using the googlesheets package.

datafest_wide <- gs_title("DataFest over the years (Responses)") %>%
  gs_read()

Data prep

Then minimal manipulation is applied to column names, and a new column concatenating city, state, and country is added to be used in geocoding.

# rename columns
yrs <- sort(rep(2011:2017, 3))
cols <- c("df_", "num_part_", "other_inst_")

names(datafest_wide) <- c("timestamp", "host", "city", "state", "country", "url",
                     paste0(cols, yrs))

# geocode host location
datafest_wide <- datafest_wide %>%
  mutate(address = paste(city, state, country)) %>% 
  mutate_geocode(address)

Note that we need to use the development version of the ggmap package for mutate_geocode() to play nicely with a tbl_df. You can install this version with install_github("dkahle/ggmap").

Next, we convert the data from wide to long format using functionality from the tidyr package. First, we gather the columns that contain yearly information (for each year, we have an indicator for whether an event was hosted at the location, the number of students that participated, and other participating institutions, if any). Then, we strip the year information from variable names, and instead save it as a variable in the dataset. Finally, we spread the key-value pair across three columns.

datafest_long <- datafest_wide %>% 
  gather(key, value, df_2011:other_inst_2017) %>%
  mutate(year = as.numeric(str_match(key, "[0-9]+"))) %>%
  mutate(key = str_replace(key, "_[0-9]+", "")) %>%
  spread(key, value) %>%
  mutate(num_part = as.numeric(num_part))

Map of 2017 ASA DataFests

The eventual goal of this post is to make a Shiny app that maps DataFest spread and growth over the years; however, I’ll start by making a map for just one year, 2017, to develop the code for the map, and then use this code within a Shiny app.

Going forward, I’ll refer to the long dataset as datafest.

datafest <- datafest_long

First, I take a subset of the data for hosts that held an event in 2017:

datafest_2017 <- filter(datafest, year == 2017 & df == "Yes")

Then, I set a few colors to be used in the plot,

href_color <- "#A7C6C6"
marker_color <- "black"
part_color <- "#89548A"

as well as the bounds of the plot based on the min/max longitude/latitude.

left <- floor(min(datafest$lon))
right <- ceiling(max(datafest$lon))
bottom <- floor(min(datafest$lat))
top <- ceiling(max(datafest$lat))

I will be making the map using the leaflet package, as this package allows for easily overlaying markers and popups to maps. The popups are text bubbles that appear when a point is clicked, and that contain additional information about that data point. This is a good place to add some event-specific information, such as name of host, and link to their event homepage, other participating institutions (if any), and number of participants.

host_text <- paste0(
  "<b><a href='", datafest_2017$url, "' style='color:", 
  href_color, "'>", datafest_2017$host, "</a></b>"
)

other_inst_text <- paste0(
  ifelse(is.na(datafest_2017$other_inst), 
         "", 
         paste0("<br>", "with participation from ", datafest_2017$other_inst))
)

part_text <- paste0(
  "<font color=", part_color,">", datafest_2017$num_part, 
  " participants</font>"
)

popups <- paste0(
  host_text, other_inst_text, "<br>", part_text
)

We’re finally ready to make our map! Note that the radii of the points are proportional to the log of the number of participants (times an arbitrary factor for visual appeal).

leaflet() %>%
  addTiles() %>%
  fitBounds(lng1 = left, lat1 = bottom, lng2 = right, lat2 = top) %>%
  addCircleMarkers(lng = datafest_2017$lon, lat = datafest_2017$lat,
                   radius = log(datafest_2017$num_part) * 1.2, 
                   fillColor = marker_color,
                   color = marker_color,
                   weight = 1,
                   fillOpacity = 0.5,
                   popup = popups)

Shiny app

Next, we build upon our earlier plot to create a Shiny app that has the following three components:

A slider input with animation for values between 2011 and 2017 (DataFest years, so far)
A line plot that shows the increase in the number participants over the year
A map that shows the spread of DataFest geographically over the years

You can find and interact with the app at https://gallery.shinyapps.io/datafest-map-all-years/, and the code for the app, as well as all steps up to this point, can be found at this GitHub repo.

Review of Efficient R Programming

Fri, 19 May 2017 00:00:00 +0000

In the crowded market space of data science and R language books, Lovelace and Gillespie’s Efficient R Programming (2016) stands out from the crowd. Over the course of ten comprehensive chapters, the authors address the primary tenets of developing efficient R programs. Unless you happen to be a member of the R core development team, you will find this book useful whether you are a novice R programmer or an established data scientist and engineer. This book is chock full of useful tips and techniques that will help you improve the efficiency of your R programs, as well as the efficiency of your development processes. Although I have been using R daily (and nearly exclusively) for the past 4+ years, every chapter of this book provided me with new insights into how to improve my R code while helping solidify my understanding of previously learned techniques. Each chapter of Efficient R Programming is devoted to a single topic, each of which includes a “top five tips” list, covers numerous packages and techniques, and contains useful exercises and problem sets for consolidating key insights.

In Chapter 1. Introduction, the authors orient the audience to the key characteristics of R that affect its efficiency, compared to other programming languages. Importantly, the authors address R efficiency not just in the expected sense of algorithmic speed and complexity, but broaden its scope to include programmer productivity and how it relates to programming idioms, IDEs, coding conventions, and community support – all things that can improve the efficiency of writing and maintaining code. This is doubly important for a language like R, which is notoriously flexible in its ability to solve problems in multiple ways. The first chapter concludes by introducing the reader to two valuable packages: (1) microbenchmark, an accurate benchmarking tool with nanosecond precision; and (2) profvis, a handy tool for profiling larger chunks of code. These two packages are repeatedly used throughout the remainder of the book to illustrate key concepts and highlight efficient techniques.

In Chapter 2. Efficient Setup, the reader is introduced to techniques for setting up a development environment that facilitates efficient workflow. Here the authors cover choices in operating system, R version, R start-up, alternative R interpreters, and how to maintain up-to-date packages with tools like packrat and installr. I found their overview of the R startup process particularly useful, as the authors taught me how to modify my .Renviron and .Rprofile files to cache external API keys and customize my R environment, for example by adding alias shortcuts to commonly used functions. The chapter concludes by discussing how to setup and customize the RStudio environment (e.g., modifying code editing preference, editing keyboard shortcuts, and turning off restore .Rdata to help prevent bugs), which can greatly improve individual efficiency.

Chapter 3. Efficient Programming introduces the reader to efficient programming by discussing “big picture” programming techniques and how they relate to the R language. This chapter will most likely be beneficial to established programmers who are new to R, as well as to data scientists and analysts who have limited exposure to programming in a production environment. In this chapter the authors introduce the “golden rule of R programming” before delving into the usual suspects of inefficient R code. Usefully, the book illustrates multiple ways of performing the same task (e.g., data selection) with different code snippets, and highlights the performance differences through benchmarked results. Here we learn about the pitfalls of growing vectors, the benefits of vectorization, and the proper use of factors. The chapter wraps up with the requisite overview of the apply function family, before discussing the use of variable caching (package memoise) and byte compilation as important techniques in writing fast, responsive R code.

Chapter 4. Efficient Workflow will be of primary use to junior-level programmers, analysts, and project managers who haven’t had enough time or practice to develop their own efficient workflows. This chapter discusses the importance of project planning, audience, and scope before delving into common tools that facilitate project management. In my opinion, one of best aspects of R is the huge, maddeningly broad number of packages that are available on CRAN and GitHub. The authors provide useful advice and techniques for identifying the packages that will be of most use to your project. A brief discussion on the use of R Markdown and knitr concludes this chapter.

Chapter 5. Efficient Input/Output is devoted to efficient read/write operations. Anybody who has ever struggled with loading a big file into R for analysis will appreciate this discussion and the packages covered in this chapter. The rio package, which can handle a wide variety of common data file types, provides a useful starting point for exploratory work on a new project. Other packages that are discussed (including readr and data.table) provide more efficient I/O than those in base R. The chapter ends with a discussion of two new file formats and associated packages, (feather and RProtoBuf), that can be used for cross-language, fast, efficient serialized data I/O.

Chapter 6. Efficient Data Carpentry introduces what are, in my opinion, the most useful R tools for data munging – what Lovelace and Gillespie prefer to call by the more admirable term “data carpentry.” This chapter could more aptly be titled the “Tidyverse” or the “Hadleyverse”, for most of the tools discussed in this chapter were developed by prolific R package writer, Hadley Wickham. Sections of the chapter are devoted to each of the primary packages of the tidyverse: tibble, a more useful and user-friendly data.frame; tidyr, used for reshaping data between short and long forms; stringr, which provides a consistent API over obtuse regex functions; dplyr, used for efficient data processing including filtering, sorting, mutating, joining, and summarizing; and of course magrittr, for piping all these operations together with %>%. A brief section on package data.table rounds out the discussion on efficient data carpentry.

Chapter 7. Efficient Optimization begins with the requisite optimization quote by computer scientist Donald Knuth:

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

In this chapter, the authors introduce profvis, and they illustrate the utility of this package by showing how it can be used to identify bottlenecks in a Monte Carlo simulation of a Monopoly game. The authors next examine alternative methods in base R that can be used for greater efficiency. These include discussion of if() vs. ifelse(), sorting operations, AND (&) and OR (|) vs. && and ||, row/column operations, and sparse matrices. The authors then apply these tricks to the Monopoly code to show a 20-fold decrease in run time. The chapter concludes with a discussion and examples of parallelization, and the use of Rcpp as an R interface to underlying fast and efficient C++ code.

I found the chapter Efficient Hardware to be the least useful in the book (spoiler alert: add more RAM or migrate to cloud-based services), though the chapter on Efficient Collaboration will be particularly useful for novice data scientists lacking real-world experience developing data artifacts and production applications in a distributed, collaborative environment. In this chapter, the authors discuss the importance of coding style, code comments, version control, and code review. The final chapter Efficient Learning, will find appreciative readers among those just getting started with R (and if this describes you, I would suggest that you start with this chapter first!). Here the authors discuss using and navigating R’s excellent internal help utility, as well as the importance of vignettes and source code in learning/understanding. After briefly introducing swirl, the book concludes with a discussion of online resources, including Stack Overflow; the authors thankfully provide the newbie with important information on how to ask the right questions and the importance of providing a great R reproducible example.

In summary, Lovelace and Gillespie’s Efficient R Programming does an admirable job of illustrating the key techniques and packages for writing efficient R programs. The book will appeal to a wide audience from advanced R programmers to those just starting out. In my opinion, the book hits that pragmatic sweet spot between breadth and depth, and it usefully contains links to external resources for those wishing to delve deeper into a specific topic. After reading this book, I immediately went to work refactoring a Shiny dashboard application I am developing and several internal R packages I maintain for our data science team. In a matter of a few short hours, I witnessed a 5 to 10-fold performance increase in these applications just by implementing a couple of new techniques. I was particularly impressed with the greatly improved end-user performance and the ease with which I implemented intelligent caching with the memoise package for a consumer decision tree application I am developing. If you care deeply about writing beautiful, clean, efficient code and bringing your data science to the next level, I highly recommend adding Efficient R Programming to your arsenal.

The book is published by O’Reilly Media and is available online at the authors’ website, as well as through Safari.

Databases using R

Wed, 17 May 2017 00:00:00 +0000

Current State

Using databases is unavoidable for those who analyze data as part of their jobs. As R developers, our first instinct may be to approach databases the same way we do regular files. We may attempt to read the data either all at once or as few times as possible. The aim is to reduce the number of times we go back to the data ‘well’, so our queries extract as much data as possible. After that, we spend cycles analyzing the data in memory. Here is what this model looks like:

Because the volume of data is significant with this approach, we usually attempt to come up with strategies to minimize the resources and time it takes to analyze the data. We may try to retrieve all rows of a few columns, or a few rows of several of columns. Another tactic is to save the query results into individual files for later analysis.

An improvement to the current approach would be to use the database’s SQL Engine to perform as much of the data exploration as possible. An enterprise-grade SQL server will have more power, and will be better tuned, to execute transformations of large amounts of data. Our goal would then be to bring into R a more targeted data set that will be used for visualization and modeling.

This improvement comes at a cost: we will need to know how to write SQL queries, and will have to switch between both languages. We may also end up using an external querying tool that is able to provide a list of tables and inline SQL code helpers. Of course, this involves switching between tools. On a personal note, I used to switch from R to Microsoft SQL Management Studio. After I that, I would bring the finalized query back into my code in R.

A better way

The dplyr package simplifies data transformation. It provides a consistent set of functions, called verbs, that can be used in succession and interchangeably to gain understanding of the data iteratively. The first time I re-wrote R code using dplyr, the new script was at least half as long and much easier to understand.

Another nice thing about dplyr is that it can interact with databases directly. It accomplishes this by translating the dplyr verbs into SQL queries. This incredibly convenient feature allows us to ‘speak’ directly with the database from R, thus resolving the issues brought up in the previous section:

Run data exploration over all of the data - Instead of coming up with a plan to decide what data to import, we can focus on analyzing the data inside the database, which in turn should yield faster insights.
Use the SQL Engine to run the data transformations - We are, in effect, pushing the computation to the database because dplyr is sending SQL queries to the database.
Collect a targeted dataset - After become familiar with the data and choosing the data points that will either be shared or modeled, a final query can then be used to bring back only that data into memory in R.
All your code is in R! - Because we are using dplyr to communicate with the database, there is no need to change language, or tools, to perform the data exploration.

Example

There are three things that we will need to get started:

A database we can access
A database driver installed in either our workstation or RStudio Server
All of the required packages installed in R

In this section, we will demonstrate how to access a Microsoft SQL Server database from a workstation that is running on Microsoft Windows.

Database Driver

A database driver is a program that allows the workstation and the database to communicate. In Microsoft Windows, the drivers that connect to MS SQL databases are installed by default. We need the name of the driver that will be used inside our code in R. The easiest way to do this is to open the ODBC Data Source Administrator. To find it in your, system please refer to this article: Check the ODBC SQL Server Driver Version (Windows) . Once the administrator program is open, click on the Drivers tab. In my laptop, these are the drivers available. I will use SQL Server for the Driver argument in my connection in R.

R packages

Besides dplyr, the following packages are required:

odbc - This is the interface between the database driver and R
DBI - Standardizes the functions related to database operations
dbplyr - Enables dplyr to interact with databases. It also contains the vendor-specific SQL translations.

The database accessibility feature is still being developed, so we will use the development versions of dbplyr and dplyr.

devtools::install_github("tidyverse/dplyr")
devtools::install_github("tidyverse/dbplyr")
devtools::install_github("rstats-db/odbc")
install.packages("DBI")

Connect to the database

We will use the dbConnect() function from the DBI package to connect to the database. The value for the Driver argument is the name we determined in the Database Driver section above.

library(DBI)

con <- dbConnect(odbc::odbc(),
                   Driver    = "SQL Server", 
                   Server    = "localhost",
                   Database  = "airontime",
                   UID       = [My User ID],
                   PWD       = [My Password],
                   Port      = 1433)

A very useful function in DBI is dbListTables(), which retrieves the names of available tables.

dbListTables(con)

[1] "airlines" "airport" "airports" "faithful" "flights" "iris"

Another useful function is the dbListFields, which returns a vector with all of the column names in a table.

dbListFields(con, "flights")

[1] "year" "month" "day" "dep_time" "sched_dep_time" [6] "dep_delay" "arr_time" "sched_arr_time" "arr_delay" "carrier" [11] "flight" "tailnum" "origin" "dest" "air_time" [16] "distance" "hour" "minute" "time_hour"

Interacting with the data using dplyr

Using dplyr, we can easily preview a database. The tbl() command creates a reference to the table.

library(dplyr)
tbl(con, "flights")

Source:     table<flights> [?? x 19]
Database:   Microsoft SQL Server 12.00.4422[username@localhost/airontime]

    year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay carrier flight tailnum origin  dest
   <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>     <dbl>   <chr>  <int>   <chr>  <chr> <chr>
1   2013     1     1      517            515         2      830            819        11      UA   1545  N14228    EWR   IAH
2   2013     1     1      533            529         4      850            830        20      UA   1714  N24211    LGA   IAH
3   2013     1     1      542            540         2      923            850        33      AA   1141  N619AA    JFK   MIA
4   2013     1     1      544            545        -1     1004           1022       -18      B6    725  N804JB    JFK   BQN
5   2013     1     1      554            600        -6      812            837       -25      DL    461  N668DN    LGA   ATL
6   2013     1     1      554            558        -4      740            728        12      UA   1696  N39463    EWR   ORD
7   2013     1     1      555            600        -5      913            854        19      B6    507  N516JB    EWR   FLL
8   2013     1     1      557            600        -3      709            723       -14      EV   5708  N829AS    LGA   IAD
9   2013     1     1      557            600        -3      838            846        -8      B6     79  N593JB    JFK   MCO
10  2013     1     1      558            600        -2      753            745         8      AA    301  N3ALAA    LGA   ORD
# ... with more rows, and 5 more variables: air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>

The tally() verb in dplyr returns the row count.

tally(tbl(con, "flights"))

Source:     lazy query [?? x 1]
Database:   Microsoft SQL Server 12.00.4422[username@localhost/airontime]

       n
   <int>
1 336776

When used against a database, the previous function is converted to a SQL query that works with MS SQL Server. The show_query() function displays the translation.

show_query(tally(tbl(con, "flights")))

<SQL> SELECT COUNT(*) AS "n" FROM "flights"

Bringing it all together

The last code sample shows how easy it is to find out what the top airlines are by number of flights. Additionally, we wish to see the names of the airlines and not their codes. The steps taken are:

Start with the flights table and join it to the carrier table to obtain the airline name
Group the data by the airline name
Tally the total rows by airline name
Order the data by the resulting tallies in a descending order

All of these steps are translated into a SQL statement and processed inside the database. We do not need to import the tables into R memory at any time, we just use dplyr to get the results quickly.

tbl(con, "flights") %>%
  left_join(tbl(con, "airlines"), by = "carrier") %>%
  group_by(name) %>%
  tally %>%
  arrange(desc(n))

Source:     lazy query [?? x 2]
Database:   Microsoft SQL Server 12.00.4422[username@localhost/airontime]
Ordered by: desc(n)

# S3: tbl_dbi
                       name     n
                      <chr> <int>
 1    United Air Lines Inc. 58665
 2          JetBlue Airways 54635
 3 ExpressJet Airlines Inc. 54173
 4     Delta Air Lines Inc. 48110
 5   American Airlines Inc. 32729
 6                Envoy Air 26397
 7          US Airways Inc. 20536
 8        Endeavor Air Inc. 18460
 9   Southwest Airlines Co. 12275
10           Virgin America  5162
# ... with more rows

Additional Resources

Here are links that will provide a deeper look into their respective subjects:

dplyr’s Official Site
Vignette of the DBI package
R for Data Science - An online book that covers how to use dplyr and other like packages that together are called the tidyverse.

Conclusion

When we have only one method available to us, it is sometimes hard to see its inherent flaws. The method does what we need, so we do our best to overcome its shortfalls.

Our hope is that highlighting the issues related to importing large amounts of data into R, and the advantages of using dplyr to interact with databases, will be the encouragement needed to learn more about dplyr and to give it a try.

We plan to continue writing about the subject of databases using R in future posts. We will cover different aspects and techniques to get the most out of working with these two great technologies.

Mapping Quandl Macroeconomic Data

Wed, 10 May 2017 00:00:00 +0000

In previous posts, we built a map to access global ETFs and a simple Shiny app to import and forecast commodities data from Quandl. Today, we will begin a project that combines those previous apps. Our end goal is to build an interactive map to access macroeconomic data via Quandl, allowing the user to choose an economic indicator and click on a country to access that indicator’s time series. For example, if a user wants to import and graph GDP-per-capita data from China, that user will be able to select GDP per Capita from a drop-down menu, and then click on China to display the time series. As usual, we’ll start with a Notebook for data import, test visualizations, and map building.

Careful readers will note that we are not simply reusing code from those previous projects; that is because we are going to introduce some new packages to our toolkit. Specifically:

Instead of downloading a spatial data frame from Natural Earth, we will use the new R package rnaturalearth and it’s function ne_countries() to import our geographic data.
Relatedly, we are not going work with a “spatial” data frame, but rather with a simple features data frame of class sf. Learn more here.
Finally, when we test our data import of several macroeconomic data sets, we will use the map() function from the purrr package instead of relying on lapply(), as we did previously when passing ticker symbols to getSymbols(). If you’re like me, you won’t be sad to be done with lapply().

Let’s get to it!

We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators (WDI). The Quandl code for WDI is WWDI, and thus we’ll prepend WWDI/ to each data set call.

One difference from our earlier work with Quandl is that previously we imported and visualized global time series, such as the price of oil, gold or copper. By global, I mean these time series were not associated with a particular country. That simplified our job a bit because we did not need to supply a country code to Quandl. Today, we do need to include a country code.

Let’s start with a simple example and import GDP-per-capita data for China, whose country code is CHN. We will pass the string WWDI/CHN_NY_GDP_PCAP_KN to Quandl, which consists of the WDI code WWDI/, appended by the China country code CHN, appended by the GDP-per-capita code _NY_GDP_PCAP_KN.

library(Quandl)
# Pass the code string to Quandl. 
China_GDPPC <- Quandl("WWDI/CHN_NY_GDP_PCAP_KN", type = 'xts') %>% 
  # Add a nice column name
  `colnames<-`("GDP Per Capita")

# Take a look at the result.
tail(China_GDPPC, n = 6)

##            GDP Per Capita
## 2010-12-31       30876.04
## 2011-12-31       33658.85
## 2012-12-31       36126.73
## 2013-12-31       38737.58
## 2014-12-31       41354.61
## 2015-12-31       43991.55

That looks pretty good, but note that the data only goes back to December of 2015 - not ideal, but fine for our illustrative purposes.

Our eventual Shiny app, though, will give the user a choice among many economic time series, and we want to test the import and visualization of all those choices. In addition to GDP-per-capita, let’s test GDP-per-capita growth, real interest rate, exchange rate, CPI, and labor force participation rate. Those offer a nice snapshot of a country’s economy.

To run our test in the code chunk below, we will first build a vector of data set codes. Note that each of them start with WWDI/, then CHN, and then the relevant code. Next, we will pipe that vector to Quandl using the map() function. map() takes a function, in this case the Quandl() function, and applies it to the elements in a vector, in this case our vector of data set codes. We also want to specify type = "xts".

If we stopped our pipe there, the output would be a list of six time series and that would suit our testing purposes, but I don’t love having to look at a list of six. Thus, we’ll keep on piping and use the reduce() function from the purrr package and call merge() to combine the six lists into one xts object. That is much easier to deal with, and easier to pass to dygraphs for our test visualization. The last piped function will invoke colnames<- to clean up the column names.

library(purrr)
# Create a vector of economic indicators that can be passed to Quandl via map().
# Include names and values for easy naming of xts columns.
econIndicators <- c("GDP Per Capita" = "WWDI/CHN_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "WWDI/CHN_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "WWDI/CHN_FR_INR_RINR",
                  "Exchange Rate" = "WWDI/CHN_PX_REX_REER",
                  "CPI" = "WWDI/CHN_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "WWDI/CHN_SL_TLF_ACTI_ZS")

# You might want to supply your Quandl api key. It's free to get one.
Quandl.api_key("d9EidiiDWoFESfdk5nPy")

China_all_indicators <- 
  # Start with the vector of Quandl codes
  econIndicators %>% 
  # Pass them to Quandl via map(). 
  map(Quandl, type = "xts") %>% 
  # Use the reduce() function to combine them into one xts objects.
  reduce(merge) %>% 
  # Use the names from the original vector to set nicer column names.
  `colnames<-`(names(econIndicators))
# Have a look.
tail(China_all_indicators, n = 6)

##            GDP Per Capita GDP Per Capita Growth Real Interest Rate
## 2011-12-31       33658.85              9.012854          -1.472149
## 2012-12-31       36126.73              7.332031           3.523236
## 2013-12-31       38737.58              7.226936           3.692594
## 2014-12-31       41354.61              6.755778           4.732429
## 2015-12-31       43991.55              6.376423           4.270386
## 2016-12-31             NA                    NA                 NA
##            Exchange Rate      CPI Labor Force Part. Rate
## 2011-12-31      102.6904 5.410850                   76.7
## 2012-12-31      108.4435 2.624921                   77.0
## 2013-12-31      115.2980 2.627119                   77.3
## 2014-12-31      118.9863 1.996847                   77.6
## 2015-12-31      131.6298 1.442555                     NA
## 2016-12-31      124.1864 2.007556                     NA

That seems to have done the job, and note that both CPI and Exchange have data up to December of 2016. Much better, and a sign that there will be date variation among countries and indicators. That won’t be crucial today, but it could be if we needed consistent dates for all of our data sets.

As a final step, let’s use the dygraph() function to visualize these time series to make sure things work properly before moving to Shiny.

library(dygraphs)

## Warning: package 'dygraphs' was built under R version 3.3.2

dygraph(China_all_indicators$`GDP Per Capita`, main = "GDP Per Capita")

dygraph(China_all_indicators$`GDP Per Capita Growth`, main = "GDP Per Capita Growth")

dygraph(China_all_indicators$`Real Interest Rate`, main = "Real Interest Rate")

dygraph(China_all_indicators$`Exchange Rate`, main = "Exchange Rate")

dygraph(China_all_indicators$`CPI`, main = "CPI")

dygraph(China_all_indicators$`Labor Force Part. Rate`, main = "Labor Force Part. Rate")

GDP-per-capita has been on a steady march higher, labor force participation plummeted in the 1990’s and 2000’s, probably as the labor market became more market-oriented, and real interest rates look quite choppy. Plenty to learn, and it signals that our Shiny app might be a nice exploratory tool.

For now, it’s on to map-building!

In previous posts, we have started map-building by grabbing a spatial object from the internet. Today, we will use the new rnaturalearth package and its fantastic function ne_countries(). We will specify type = "countries" and returnclass = 'sf'. We could have set returnclass = 'sp' if we wanted to work with a spatial data frame, but I have been migrating over to the sf package and simple features data frames for mapping projects. Lots to learn here, but in general, it seems to use more tidy concepts than the sp package.

library(rnaturalearth)
library(sp)
world <- ne_countries(type = "countries",  returnclass = 'sf')

# Take a peek at the name, gdp_md_est column and economy columns. 
# The same way we would peek at any data frame.
head(world[c('name', 'gdp_md_est', 'economy')], n = 6)

## Simple feature collection with 6 features and 3 fields
## geometry type:  MULTIPOLYGON
## dimension:      XY
## bbox:           xmin: -73.41544 ymin: -55.25 xmax: 75.15803 ymax: 42.68825
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
##                   name gdp_md_est                   economy
## 0          Afghanistan      22270 7. Least developed region
## 1               Angola     110300 7. Least developed region
## 2              Albania      21810      6. Developing region
## 3 United Arab Emirates     184300      6. Developing region
## 4            Argentina     573900   5. Emerging region: G20
## 5              Armenia      18770      6. Developing region
##                         geometry
## 0 MULTIPOLYGON(((61.210817091...
## 1 MULTIPOLYGON(((16.326528354...
## 2 MULTIPOLYGON(((20.590247430...
## 3 MULTIPOLYGON(((51.579518670...
## 4 MULTIPOLYGON(((-65.5 -55.2,...
## 5 MULTIPOLYGON(((43.582745802...

This output looks similar to an sp object, but it’s not identical. In particular, note the geometry column, which contains the multipolygon latitude and longitude coordinates. It’s a matter of personal preference, but I find this sf object more intuitive in terms of how the geometry corresponds with the rest of the data.

Now we’ll create our shading and popups. Note that this code is identical to when we were working with spatial data frames.

library(leaflet)
# Create shading by GDP. Let's go with purples.
gdpPal <- colorQuantile("Purples", world$gdp_md_est, n = 20)

# Create popup country name and income group so that something happens
# when a user clicks the map.
popup <- paste0("<strong>Country: </strong>", 
                world$name, 
                "<br><strong>Market Stage: </strong>", 
                world$income_grp)

Now, it’s time to build our leaflet map. This code is nearly identical to our code from previous projects, but with one major change. As we saw above, we need to pass in a country code to Quandl, not a country name and not a ticker symbol. Thus, we don’t want to set layerId = name, but we also don’t need to add anything special to our world object. It already contains a column called iso_a3, and that column contains the country codes used by Quandl. Fantastic!

We just need to set layerId = ~iso_a3 and we will be able to access country codes in our Shiny app when a user clicks the map. This is quite fortunate, but it also highlights a good reason to spend time studying the world object. It might contain other data that will be useful in future projects.

leaf_world <- leaflet(world) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7, 
      
      # Note the layer ID. Not a country name! It's a country code! 
      
      color = ~gdpPal(gdp_md_est), layerId = ~iso_a3, popup = popup)

leaf_world

Because Quandl uses the ISO 3-letter code to identify a country, and because the rnaturalearth sf object already contains a column with iso_a3 country codes, building our map wasn’t very hard. It’s worth pointing out that in various projects, we have now built maps with three different layerIDs. In the Global ETF Map project, we built one map with layerID = name, using the built-in name column in the rnaturalearth object, and we built another map with layerID = tickers, using a column of tickers that we added ourselves with the merge() function. Now, today, we have a map with layerID = iso_a3, using the built-in column of country codes. We are building up a nice portfolio of maps with different layerIDs should we ever want or need them in the future.

That’s all for today. We’ll save that leaflet_world object for use in our Shiny app! See you next time.

Shiny in Medicine

Wed, 03 May 2017 00:00:00 +0000

Shiny Apps are becoming ubiquitous as a way for data scientists to present the results of an analysis, and also to engage with information consumers who may not be coders. The trend I see is that the greater the variety of skills and interests of the information consumers for any particular project, the more valued are interactive visualizations that can be integrated into enterprise-wide communication workflows. So, it is not surprising to see Shiny apps popping up in all manner of healthcare and medical applications. If data scientists are going to bring predictive analytics into clinical workflows where doctors, nurses, scientists, technicians and administrators are all part of near real-time decision processes, they are going to have to be even more inventive in providing these multi-skilled teams with low-friction tools to ingest and manipulate information. Below are few interesting Shiny apps that are broadly related to Health Care and Medicine. My guess is that interactive visualizations like these will improve research and clinical workflows, and eventually change how all of us look at Health Care and Medicine.

The Genome viewer for ICGC cancer, built by the folks at Aridhia, is geared towards researchers. You can learn how to interpret the plot and learn the story behind its creation here.

The Emergency Department Simulation, built by a group of mathematicians and physicians, models patient flow information under different assumptions about emergency department case loads, and illustrates how predictive analytics and statistical analysis can be integrated into operational clinical workflows.

The Colorectal Cancer risk calculator from the Cleveland Clinic targets physicians and the general public to personalize the risk of this disease. I found working through different “what if” scenarios of great help in thinking about the risk factors that are under my control.

Finally, these two Shiny Apps that provide information about US hospitals should be of interest to public health planners, as well as to the general public. The Hospital Ranking App compares hospital outcomes for heart attack, heart failure, and pneumonia against national statistics.

The Access to Hospital Care Dashboard plots the density of hospitals throughout the United States, and indicates under-served areas.

If you or your team are doing this kind of work, we here at R Views would love to hear about it.

A Shiny App for Importing and Forecasting Commodities Prices from Quandl

Fri, 21 Apr 2017 00:00:00 +0000

In a previous post, we imported oil data from Quandl and applied a simple model to it. Today, we’ll port that work over to a Shiny app (by way of flexdashboard, of course) that allows a user to choose a commodity (oil, copper or gold), choose a frequency for the time series, and choose how many periods ahead to forecast. The app will display the price history and the forecasted price. To see it in action, have a look here.

As with the previous Notebook, the main purpose of this post is to build a template where a more sophisticated or proprietary model could be used for forecasting. By setting it up so that models could be selected as inputs, several different models could be included. We are not as concerned with the modeling as much as we are with providing a format that is friendly to both end users and any future collaborator that might want to take this app and expand upon it.

Let’s get to it! In the code chunk below, we are immediately faced with a few important decisions. The first of those is the format of the user input. At the extreme, we could use textInput and simply allow the user to enter the code for the desired data set. The benefit is that we would not limit the user in any way; he or she could choose any dataset on Quandl. The cost would be that the user would need to know - or go to Quandl and look up - the code for any data set.

For example, to import WTI oil prices, the user would have to type in “FRED/DCOILWTICO”. That’s no problem if most of the end users know that code, but it’s a big problem otherwise. We want to emphasize convenience and broad usability, so we are going with selectInput instead of textInput, meaning our app will show a drop-down of a few choices. The user just selects “WTI oil” instead of typing “FRED/DCOILWTICO”“, or selects”copper" instead of typing “ODA/PCOPP_USD”. But, if a user wants to work with a data set that we haven’t included, said user is out of luck.

Another big decision is how many choices to give the user. I have included only 3: oil, gold and copper. In industry, you would probably include several more, perhaps all of the industrial metals, but there’s is a cutoff somewhere. Or maybe we prefer one choice, because this app is just for oil analysis. Either way, the number of options in the drop-down menu is another trade-off between usability and flexibility.

The final decision is a bit more nuanced, and requires looking ahead to how these inputs will be used further down in the app. Have a peek at the object called dataChoices, and you might notice that we don’t strictly need that object. We could have put the vector of choices as an argument to selectInput, so that our code would have read choices = c("WTI oil" = "FRED/DCOILWTICO", ...) instead of choices = dataChoices. In that choice assignment, “WTI oil” is called the name and “FRED/DCOILWTICO” is called the value (together we can think of them as a name-value pair). The entire reason for building a separate dataChoices object is that we want the ability to extract either the name or the value of the name-value pair. Usually we would care only about the value, because we want to pass the value to Quandl and import the data, but that name is going to be useful as well when we label our graph.

The ability to extract names and values will become even more useful when we get to the frequency of the time series and forecasting. For now, let’s look at dataChoices and selectInput.

# Notice a tradeoff here: we're making it easy on our users because they don't need to 
# remember the naming conventions. But, we're also forced to severely limit their choices. 
# The dataChoices object is going to allow us to add a nicer label to the graph. 
# Notice also how easily we can include datasets from different sources, and not worry about
# their formats. Thanks, Quandl!

dataChoices <- c("WTI oil" = "FRED/DCOILWTICO", # oil data from Fred
                 "Copper" = "ODA/PCOPP_USD", # copper data from ODA
                 "Gold" = "CHRIS/CME_GC1") # gold data from CME

selectInput("dataSet",
            "Commodity",
            choices = dataChoices,
            selected = "WTI oil")

Alright, we have given the user the ability to choose a commodity. Next we want to ask about the frequency, and this gets a bit more complicated. We need to tell Quandl the frequency of the time series to import using the “daily”, “weekly”, “monthly” convention, so we set those to the values. But, further down the app when we want to store the forecast results, we’ll need to use the “days”, “weeks”, “months” phrasing and we’ll need to pull out the names from the name-value pair in frequencyChoices. We knew that this would be necessary because when we built our Notebook for importing, wrangling, and testing, we noted the different conventions and started thinking about how to deal with them in the Shiny context.

# The frequencyChoices object is going to allow us to pass different period conventions
# to different places further down the app. 

frequencyChoices <- c("days" = "daily",
                      "weeks" = "weekly", 
                      "months" = "monthly")

selectInput("frequency",
            "freq",
            choices = frequencyChoices, 
            selected = "months")

The remainder of the inputs should look familiar as they are in standard format.

dateRangeInput("dateRange",
               "Date range",
               start = "1980-01-01",
               end   = "2016-12-31")

numericInput("periods", "Periods to Forecast", 6, min = 1, max = 100)

Now that we’ve built the sidebar inputs, let’s put them to use. First, we will import the commodity time series data from Quandl. This will be familiar from the Notebook, but note in particular that we will use the value from the input$frequency choice because Quandl uses the “daily/weekly/monthly” frequency format.

# Let's pull in the Quandl data. 
# Nothing fancy here except we are going to use the reactive inputs.
# We will pass in the value from the input$dataSet key-value pair.
# We will also pass in the value from the input$frequency key-value pair. 

commodity <- reactive({
  # It might be a good idea to include your quandl api key if this will be used more than 50 times in a day. 
  Quandl.api_key("your_apikey_here")
  
  commodity <- Quandl(input$dataSet,
                      start_date = format(input$dateRange[1]),
                      end_date = format(input$dateRange[2]),
                      order = "asc",
                      type = "xts",
                      collapse = as.character(input$frequency)
  )
})

We have imported a time series object for the dataset, date range and frequency chosen by the user. Now we want to do some forecasting and create a visualization. We’ll first use the forecast() function, then we’ll combine the forecasted prices and the historical prices into one xts object that can be passed to dygraph. Let’s handle this in one reactive.

First, we’ll call forecast and pass it the periods input from the user.

combined_xts <- reactive({
  
  # Just like the Notebook, except periods is a reactive input.
  
  forecasted <- forecast(commodity(), h = input$periods)

Now we need to combine that forecasted object with the commodity object that was created in a previous code chunk. Here we will finally thank ourselves for thinking about the name-value issue when assigning frequencyChoices. When we create a dataframe to hold the forecasted time series, we build a ‘date’ column using the seq() function. The column starts on the end date of the historical data (which the user selected with the dateRange input) and runs for as many periods as the user chose in the periods input. But, we need to supply a value to the by = ... argument of the seq() function so that it knows if we want to move 6 days, 6 weeks, 6 months, etc. To do that, we need to extract the name (“days/weeks/months”) from the frequencyChoices name-value pair and pass it to seq. The way we extract the name is with this selection statement: names(frequencyChoices[frequencyChoices == input$frequency]).

  forecast_dataframe <- data.frame(
    date = seq(input$dateRange[2], 
               # The next line is very important, and it's the reason we 
               # created the frequencyChoices object.
               by = names(frequencyChoices[frequencyChoices == input$frequency]),
               length.out = input$periods),
    Forecast = forecasted$mean,
    Hi_95 = forecasted$upper[,2],
    Lo_95 = forecasted$lower[,2])

Now we convert the forecast_dataframe object to an xts and combine that new object with the commodity xts object. Remember, we imported the commodity data from Quandl in the form of an xts object, which saved us from having to do a conversion. This chunk should look very similar to the Notebook.

  forecast_xts <- xts(forecast_dataframe[,-1], order.by = forecast_dataframe[,1])
  
  combined_xts <- cbind(commodity(), forecast_xts)
  
  # Add a nicer name for the first column.
  
  colnames(combined_xts)[1] <- "Actual"
  
  # This is the combined object that will be passed to dygraphs below. 
  combined_xts
})

Next, we will create a chart of the actual price history. Nothing fancy here, except again we are going extract the ‘name’ portion of a name-value pair, this time from the dataSet object (the first one we created in the first code chunk). We want to label the graph with ‘WTI oil’ and not the Quandl code, so we select the name with names(dataChoices[dataChoices==input$dataSet]).

dygraphOutput("commodity")

output$commodity <- renderDygraph({
  dygraph(commodity(),
          # We pull out the name of the selected name-value input$dataSet like so: 
          # names(dataChoices[dataChoices==input$dataSet])
          main = paste("Price history of", names(dataChoices[dataChoices==input$dataSet]), 
                       sep = " ")) %>%
    dyAxis("y", label = "$") %>%
    dyOptions(axisLineWidth = 1.5, fillGraph = TRUE, drawGrid = TRUE)
})

Last but not least, let’s graph the historical and forecasted time series together on one graph. We could have ported the code directly from the Notebook, but I couldn’t help tinkering just a bit. I wanted to focus the graph on the time period around where the actual data ends and the forecast begins, because that is probably what’s of most interest to the user. To do that, we’ll use the dyRangeSelector() function and pass two values to the dateWindow variable: a start date and end an date. This is pure aesthetics, but I think it’s worth the effort here.

We are going to use seq() and names(frequencyChoices[frequencyChoices == input$frequency]) once again.

dygraphOutput("forecasted")

output$forecasted <- renderDygraph({

# We want the user to be able to see the forecasted area, so let's focus in on that 
# by truncating the view of the dygraph.
# We need to give the graph a start date and an end date. 
start_date <- tail(seq(input$dateRange[2], by = "-1 months", length = 6), 1)
end_date <- tail(seq(input$dateRange[2], 
                     by = names(frequencyChoices[frequencyChoices == input$frequency]), 
                     length = input$periods), 1)

Now we will supply those date objects to a piped dygraphs chain, and that will be a wrap. Have a close look at the chunk below and spot where we again extract the name from the input$dataSet name-value pair.

dygraph(combined_xts(), 
        # Name about to be extracted!
        main = paste(names(dataChoices[dataChoices==input$dataSet]), 
                     ": Historical and Forecast", sep = "")) %>%
  # Add the actual series.
  dySeries("Actual", label = "Actual") %>%
  # Add the three forecasted series.
  dySeries(c("Lo_95", "Forecast", "Hi_95")) %>% 
  # A range selector to focus on the where historical data ends and 
  # foracated data begins.
  # Note the user can still use the range selector to zoom out if so desired.
  dyRangeSelector(dateWindow = c(start_date, end_date))
})

That’s all for today! We’ve built an app that can be extended in many directions - more data set choices, an input selector so the user can choose different models, more visualizations - and in that sense it can serve as a template for ourselves and anyone that wants to build upon this work.

This app also relies on and highlights one of the tremendous benefits of Quandl: the ability to import datasets from different sources and have them arrive in the IDE in a consistent format. This makes using Quandl with Shiny quite enjoyable after the initial skeleton is built. For example, if we or a colleague did want to change this app and substitute in different commodities, there would be no worry beyond the finding the right code on Quandl and adding another name-value pair to dataChoices.

R for Enterprise: Understanding R’s Startup

Wed, 19 Apr 2017 00:00:00 +0000

R’s startup behavior is incredibly powerful. R sets environment variables, loads base packages, and understands whether you’re running a script, an interactive session, or even a build command.

Most R users will never have to worry about changing R’s startup process. In fact, for portability and reproducibility of code, we recommend that users do not modify R’s startup profile. But, for system administrators, package developers, and R enthusiasts, customizing the launch process can provide a powerful tool and help avoid common gotchas. R’s behavior is thoroughly documented in R’s base documentation: “Initialization at Start of an R Session”. This post will elaborate on the official documentation and provide some examples. Read on if you’ve ever wondered how to:

Tell R about a local CRAN-like repository to host and share R packages internally
Use a different version of Python, e.g., to support a Tensorflow project
Define a proxy so R can reach the internet in locked-down environments
Understand why Packrat creates a .Rprofile
Automatically run code at the end of a session to capture and log sesssionInfo()

We’ll also discuss how RStudio starts R. Spoiler: it’s a bit different than you might expect!

.Rprofile, .Renviron, and R*.site oh my!

R’s startup process follows three steps: starting R, setting environment variables, and sourcing profile scripts. In the last two steps, R looks for site-wide files and user- or project-specific files. The R documentation explains this process in detail.

Common Gotchas and Tricks:

The Renviron file located at R_HOME/etc is unique and different from Renviron.site and the user-specific .Renviron files. Do not edit the Renviron file!
A site-wide file, and either a project file or a user file, can be loaded at the same time. It is not possible to use both a user file and a project file. If the project file exists, it will be used instead of the user file.
The environment files are plain-text files in the form name=value. The profile files contain R code.
To double check what environment variables are defined in the R environment, run Sys.getenv().
Do not place things in a profile that limit the reproducibility or portability of your code. For example, setting options(stringsAsFactors = FALSE) is discouraged because it will cause your code to break in mysterious ways in other environments. Other bad ideas include: reading in data, loading packages, and defining functions.

Where to put what?

The R Startup process is very flexible, which means there are different ways to achieve the same results. For example, you may be wondering which environment variables to set in .Renviron versus Renviron.site. (Don’t even think about calling Sys.setenv() in a Rprofile…)

A simple rule of thumb is to answer the question: “When else do I want this variable to be set?”

For example, if you’re on a shared server and you want the settings every time you run R, place .Renviron or .Rprofile in your home directory. If you’re a system admin and you want the settings to take affect for every user, modify Renviron.site or Rprofile.site.

The best practice is to scope these settings as narrowly as possible. That means if you can place code in .Rprofile instead of Rprofile.site you should! This practice complements the previous warnings about modifying R’s startup. The narrowest scope is to setup the environment within the code, not the profile.

Quiz

What is the best way to modify the path? The answer depends on the desired scope for the change.

For example, in an R project using the Tensorflow package, I might want R to use the version of Python installed in /usr/local/bin instead of /usr/bin. This change is best implemented by reordering the PATH using PATH=/usr/local/bin:${PATH}. This is a change I only want for this project, so I’d place the line in a .Renviron file in the project directory.

On the other hand, I may want to add the JAVA SDK to the path so that any R session can use the rJava package. To do so, I’d add a line like PATH=${PATH}:/opt/jdk1.7.0_75/bin:/opt/jdk1.7.0_75/jre/bin to Renviron.site.

R Startup in RStudio

A common misconception is that R and RStudio are one in the same. RStudio runs on top of R and requires R to be installed separately. If you look at the process list while running RStudio, you’ll see at least two different processes: usually one called RStudio and one called rsession.

RStudio starts R a bit differently than running R from the terminal. Technically, RStudio doesn’t “start” R, it uses R as a library, either as a DLL on Windows or as a shared object on Mac and Linux.

The main difference is that the script wrapped around R’s binary is not run, and any customization to the script will not take affect. To see the script try:

cat $(which R)

For most people, this difference won’t be noticeable. Any settings in the startup files will still take affect. For user’s that build R from source, it is important to include the --enable-R-shlib flag to ensure R also builds the shared libraries used by RStudio.

R Startup in RStudio Server Pro

RStudio Server Pro acts differently from R and the open-source version of RStudio. Prior to starting R, RStudio Server Pro uses PAM to create a session, and sources the rsession-profile. In addition, RStudio Server Pro launches R from bash, which means settings defined in the user’s bash profile are available.

In short, RStudio Server Pro provides more ways to customize the environment used by R. You might ask why you’d ever want more options. Recall our rule of thumb: “When else do I want this variable to be set?”

In server environments, there are often environment variables set every time a user interacts with the server. These environment variables are placed in a user’s bash profile by a system admin. Normally R wouldn’t pick up these settings. RStudio Server Pro allows R to make use of the work the system admin has already done by picking up these profiles.

Likewise, there may be some actions that take place on the server when a user logs in that have to happen before R starts. For example, a Kerberos ticket used by the R session to access a data source must exist before R is started. RStudio Server Pro uses PAM sessions to enable these actions.

There may also be actions or variables that should only be defined for RStudio, and not any other time R is run. To facilitate this use case, RStudio Server Pro provides the rsession-profile. For example, if your environment makes use of RStudio Server Pro’s support for multiple versions of R, you’d place any environment variables that should defined for all versions of R inside of rsession-profile.

Examples:

Define proxy settings in Renviron.site

Renviron.site is commonly used to tell R how to access the internet in environments with restricted network access. Renviron.site is used so the settings take affect for all R sessions and users. For example,

http_proxy=http://proxy.mycompany.com

This article contains more details on how to configure RStudio to use a proxy.

Add a local CRAN repository for all users

Organizations with offline environments often use local CRAN repositories instead of installing packages directly from a CRAN mirror. Local CRAN repositories are also useful for sharing internally developed R packages among colleagues.

To use a local CRAN repository, it is necessary to add the repository to R’s list of repos. This setting is important for all sessions and users, so Rprofile.site is used.

old_repos <- getOption("repos")
local_CRAN_URI <- paste0("file://", normalizePath("path_to_local_CRAN_repo"))
options(repos = c(old_repos, my_repo = lcoal_CRAN_URI))

More information on setting up a local CRAN repository is available here.

Record sessionInfo automatically

Reproducibility is a critical part of any analysis done in R. One challenge for reproducible scripts and documents is tracking the version of R packages used during an analysis.

The following code can be added to a .Rprofile file within an RStudio project to automatically log the sessionInfo() after every RStudio session.

This log could be referenced if an analysis needs to be run at a later date and fails due to a package discrepancy.

.Last <- function(){
  if (interactive()) {
    
    ## check to see if we're in an RStudio project (requires the rstudioapi package)
    if (!requireNamespace("rstudioapi"))
      return(NULL)
    pth <- rstudioapi::getActiveProject()
    if (is.null(pth))
      return(NULL)
    
    ## append date + sessionInfo to a file called sessionInfoLog
    cat("Recording session info into the project's sesionInfoLog file...")
    info <-  capture.output(sessionInfo())
    info <- paste("\n----------------------------------------------",
                  paste0('Session Info for ', Sys.time()),
                  paste(info, collapse = "\n"),
                  sep  = "\n")
    f <- file.path(pth, "sessionInfoLog")
    cat(info, file = f, append = TRUE)
  }
}

Automatically turn on packrat

Packrat is an automated tool for package management and reproducible research. Packrat acts as a super-set of the previous example. When a user opts in to using packrat with an RStudio project, one of the things packrat automatically does is create (or modify) a project-specific .Rprofile. Packrat uses the .Rprofile to ensure that each time the project opens, Packrat mode is turned on.

To Wrap Up

R’s startup behavior can be complex, sometimes quirky, but always powerful. At RStudio, we’ve worked hard to ensure that R starts and stops correctly whether you’re running RStudio Desktop, serving a Shiny app on shinyapps.io, rendering a report in RStudio Connect, or supporting hundreds of users and thousands of sessions in a load balanced configuration of RStudio Server Pro.

March '17 New Package Picks

Fri, 14 Apr 2017 00:00:00 +0000

Two hundred and sixteen new packages were added to CRAN in March. The following are my picks for the Top Forty, organized into five categories: Bioscience, Data, Data Science, Statistics and Utilities.

Bioscience

BioInstaller v0.0.3: Provides tools to install and download massive bioinformatics analysis software and database, such as NGS analysis tools with its required database or/and reference genome.
DSAIDE v0.4.0: Provides a collection of Shiny Apps that allow users to simulate and explore infectious disease transmission. The vignette will get you started.
treespace v1.0.0: Provides tools for the exploration of distributions of phylogenetic trees. This package includes a Shiny interface which can be started from R. There are vignettes for Dengue trees, Transmission trees, and for exploring landscapes of phylogenetic trees. The following plot from the vignette shows clusters of similar trees.

library(treespace)
data(woodmiceTrees)
wm.res <- treespace(woodmiceTrees,nf=3)
wm.groves <- findGroves(wm.res, nclust=6)
plotGrovesD3(wm.groves)

Data

cropdatape v1.0.0: Provides access to data from the Agricultural Ministry of Peru.
rdataretriever v1.0.0: Provides R access to the Data Retriever command line interface that automates the tasks of finding, downloading, and cleaning public datasets.
rnaturalearth v0.1.0: Provides functions to fetch Natural Earth map data. There is an introduction and a vignette on working with countries.
ropercenter v0.1.0: Provides registered users with programmatic, reproducible access to data sets from The Roper Center for Public Opinion Research, which maintains the largest archive of public opinion data in existence. The vignette shows how to get started.
statsgrokse v 0.1.4: Provides an API for the server containing Wikipedia Page View Statistics for the years 2008 to 2015.
ukds v0.1.0: Enables reproducible, programmatic retrieval of datasets from the UK Data Service. The vignette shows how to setup and use it.

Data Science

anomalyDetection v0.1.1: Implements procedures to aid in detecting network log anomalies. The vignette provides examples.
kerasR v0.4.1: Provides an interface to the Keras Deep Learning Library, which provides specifications for describing dense neural networks, convolution neural networks (CNN), and recurrent neural networks (RNN) running on top of either TensorFlow or Theano. The vignette contains examples.
modeval v0.1.2: Allows users to easily compare multiple classifications models built with caret functions for small data sets. The vignette provides examples.

supc v0.1: Implements the self-updating process clustering algorithms proposed in Shiu and Chen. The vignette contains examples.
tensorflow v0.7: Provides an interface to TensorFlow, an open-source software library for numerical computation using data flow graphs.

Statistics

frailtyEM v0.5.4: Contains functions for fitting shared frailty models with a semi-parametric baseline hazard using the Expectation-Maximization algorithm. The vignette explains the math.
FRK v0.1.1: Provides functions to build, fit and predict spatial random effects, fixed rank kriging models with large datasets. The vignette introduces the theory and shows some examples.
hmi v0.6-3: Allows users to build single-level and multilevel imputation models using functions provided, or functions from the mice and MCMCglmm packages. There is a vignette.
margins v0.3.0: Ports Stata’s margins command for calculating marginal (or partial) effects. There is an introduction, a vignette on the Technical Implementation Details, and a comparison with the Stata Command.
mlrMBO v1.0.0: Provides a toolbox for Bayesian, model-based optimization. There is an introduction and vignettes for Mixed Space Optimization, Noisy Optimization, and Parallelization.

MonteCarlo v1.0.0: Simplifies Monte Carlo simulation studies by providing functions that automatically set up loops to run over parameter grids, parallelize the computations, and generate output in LaTeX tables. The vignette shows how to use it.
RankingProject v0.1.1: Provides functions to generate plots and tables for comparing independently sampled populations. There is an introduction and a vignette that reproduces the figures from “A Primer on Visualizations for Comparing Populations, Including the Issue of Overlapping Confidence Intervals” by Wright, Klein, and Wieczorek (2017, The American Statistician, in press).
rjmcmc v0.2.2: Provides functions to perform reversible-jump MCMC with the restriction introduced by Barker & Link. The vignette shows how to calculate posterior probabilities from the MCMC output.

Utilities

canvasXpress v1.5.2: Enables creation of visualizations using CanvasXpress, a stand-alone JavaScript library for reproducible research. The vignette shows how to get started.
collapsibleTree v0.1.4: Provides functions to build interactive Reingold-Tilford tree diagrams created using D3.js, where every node can be expanded and collapsed by clicking on it. There are some examples on the GitHub site.

cronR v0.3.0: Allows users to schedule R scripts and processes on Unix/Linux systems, while taskscheduleR allows the same from Windows. See the cronR vignette and the taskscheduleR vignette.
doctr v0.2.0: Provides tools to help check the consistency and quality of a dataset. There are vignettes for Comparing Tables, Dataset Diagnostics, and EDA Automation.
geojsonR v1.0.1: Provides functions to process GeoJson objects. The vignette gives the details.
ggedit v0.2.1: Provides an interactive layer and theme aesthetic editor for ggplot2. The ggedit gitbook explains how to use it.
implyr v0.1.0: Provides a SQL backend to dplyr for Apache Impala.
officer v0.1.0: Provides functions to manipulate Microsoft Word and Microsoft Powerpoint documents. This is a companion package to flextable.
flextable v.1.0: Includes functions to create tables in Word, PowerPoint, and HTML documents. There is an overview, and vignettes for explaining objects and table layout.
pivottabler v0.1.0: Allows users to create complex pivot tables and pivot tables with irregular layouts in R. There are multiple vignettes, including an introduction, a styling guide, a Shiny example and explanations of calculations and data groups.
Polychrome v0.8.2: Provides tools for creating, viewing, and assessing qualitative palettes with many (20-30 or more) colors. There is a colorful vignette

library(Polychrome)
pal2 <- alphabet.colors(26)
rancurves(pal2)

RApiDatetime v0.0.3: Provides a C-level API to allow packages to access C-level R date and datetime code.
Rcssplot v0.2.0.0: Provides tools to style plots with cascading style sheets. The vignette shows how.
reticulate v0.7: Implements an R interface to Python modules, classes, and functions. When calling into Python, R data types are automatically converted to their equivalent Python types. When values are returned from Python to R, they are converted back to R types. There is an overview and a vignette describing arrays in R and Python
shinyWidgets v0.2.0: Provides custom input widgets for Shiny apps. See the README for examples.

shinyjqui v0.1.0: An extension to shiny that brings interactions and animation effects from the jQuery UI library. There is an introduction and a vignette with examples.
valaddin v0.1.0: Provides tools to transform functions into functions with input validation checks, in a manner suitable for both programmatic and interactive use. The vignette shows how.

Copper, Gold and Ten-Year Treasury Notes

Wed, 12 Apr 2017 00:00:00 +0000

Today, we will continue getting familiar with data from Quandl, but will also devote more time to expanding our dygraphs toolkit. We will be building up a data visualization in discrete pieces, which isn’t very efficient, but will make things easier when we move this project into production as a Shiny app. From a substantive perspective, we will examine the relationship between the price ratio of copper-gold and 10-year Treasure yields.

Why do we care about the copper-gold price ratio and Treasury yields? First, both Jeff Gundlach (in this webcast) and Adam Robinson say so, and that’s probably good enough. The theory goes like this:

Copper is a useful industrial metal whose price tends to rise when the global economy expands. As firms produce more goods that require copper as an input, the increased demand for copper drives the price higher. Gold, on the other hand, is a somewhat less useful metal whose prices tends to rise when investors are fearful about a contracting global economy. Gold is a safe-haven investment, and a rising gold price signals either a contracting economy, investor fears of a contracting economy, or both. Gold prices tend to fall when the economy is humming along nicely. Thus, the copper-gold price ratio tends to be increasing when the economy is expanding.

The yield on 10-year Treasury Notes also tends to rise during economic expansion because investors’ inflation expectations are on the rise. When investors expect inflation to increase, they anticipate an uptick in interest rates (for those of you who are too young to remember what an interest rate is, take a look at rates in the mid-1980s) and start to seek higher yields today. That can drive down Treasury prices and increase yields.

Thus, we should observe a positive relationship between the copper-gold price ratio and 10-year yields. Both should be rising and falling based on the state of the world economy. There’s nothing too crazy here, but it’s an interesting relationship to think about and investigate. That’s what we’ll do today!

First, let’s import the relevant time series data from Quandl. We will specify type = "xts" in order to create xts objects and collapse = "daily" because we want daily prices. Note in particular our data sources: CME for copper and gold, and FRED for the 10-year yield. But, we just need the Quandl codes and to be careful about consistent start/end dates for each data set. It’s a liberating feeling to know before starting a project that we’ll be able to find whatever data we need in one source.

Let’s get to it.

library(Quandl)
library(dplyr)
library(xts)
library(lubridate)
library(dygraphs)

copper <- Quandl("CHRIS/CME_HG1", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

gold <- Quandl("CHRIS/CME_GC1", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

ten_year <- Quandl("FRED/DGS10", type = "xts", collapse = "daily",  
                    start_date = "2012-01-01", end_date = "2017-02-28")

Now we want to combine these into one xts object, which normally would be an easy invocation of merge.xts, but there’s a slight wrinkle. After creating one xts object, we know we need to calculate a ratio of copper/gold. That means that the presence of NAs will be a problem (I found this out the old-fashioned way, by running code that threw an error). Let’s take care of that by prepending the na.locf() function to our merge operation. That function will replace all NAs with the previous day’s value. Why might one of our time series have an NA when another doesn’t? Maybe one of the instruments has a trading holiday, or FRED publishes data for days that are general market holidays.

The downside, of course, is that we are adding observed values that are fictional. I can live with that here since we’re creating a visualization to help understand and gain an intuition about the hypothesized copper-gold yield relationship. If we were using this data to create a trading strategy or algorithm, our NA replacement would be unacceptably fuzzy and a more rigorous decision-making process would be needed to synchronize the data sets.

No matter which approach is taken, the most crucial thing is to explain and make easily reproducible whatever process is used for handling NAs or any data cleaning such as this. Your colleagues, future self, clients and research audience can scrutinize, discount, ignore or applaud it accordingly.

# We're going to merge our 3 xts objects into on xts objects. This would normally be
# very simple with merge.xts but we want to eliminat NAs with na.locf().

copper_gold_tenYear_merged <- na.locf(merge.xts(copper$Settle, gold$Settle, ten_year),
                                      formLast = TRUE)

colnames(copper_gold_tenYear_merged ) <- c("Copper", "Gold", "TenYear")

Now we create a new column to store the ratio of copper gold prices.

# Create the ratio of prices. I multiply copper by 100 to synch with the scale used by 
# Gundlach in his presentation.
copper_gold_tenYear_merged$ratio <- (copper_gold_tenYear_merged$Copper*100)/copper_gold_tenYear_merged$Gold

We have an xts object that holds four time series, but I want to chart only the copper-gold price ratio and 10-year yields. This next step is not necessary but to be extra clear, I am going to create a new xts object to hold only those two time series.

tenYear_ratio  <- merge(copper_gold_tenYear_merged$ratio, copper_gold_tenYear_merged$TenYear)

We use dygraphs to chart that one xts object, and call dySeries() for each of the columns to be included.

dygraph(tenYear_ratio) %>%
  dySeries("ratio") %>% 
  dySeries("TenYear")

That chart is completely unhelpful, of course, because our time series have different scales. I also don’t love the choppiness of the blue 10-year chart. Let’s address these two issues by adding a right-hand side scale and a dyRoller().

The dyRoller() will help smooth out our chart because each plotted point will be an average of the number of periods specified with rollPeriod = X. This won’t affect our xts object, where we store the data, it just makes the chart more readable.

Adding the right-hand-side y-axis requires a few more lines of code. First we need to invoke dyAxis() for the left-hand axis, called “y”. Then we invoke dyAxis() for the right-hand axis, called “y2”. We also need to set independentTicks = TRUE so that we can use a unique, independent value scale for the right-hand side. Next, in our dySeries() call for each time series, we assign each one to an axis. Here we assign “ratio” with axis = 'y', so that the copper-gold price ratio will be on the left-hand scale, and we assign “TenYear” with axis = 'y2', so the 10-year yield will be on the right-hand scale. I also like to include a label with LHS and RHS for each time series.

dygraph(tenYear_ratio) %>% 
  # Add the rollPeriod for smoothing.
  dyRoller(rollPeriod = 3) %>% 
  # Create two independent axes.
  dyAxis("y", label = "USD") %>%
  dyAxis("y2", label = "Percent (%)", independentTicks = TRUE) %>%
  # Assign each time series to an axis.
  dySeries("ratio", axis = 'y', label = "Copper/Gold (LHS)") %>% 
  dySeries("TenYear", axis = 'y2', label = "10-Year % Yield (RHS)")

Alright, that chart looks better now (though I encourage you to play with different settings for rollPeriod to compare different smoothness levels) and, incidentally, seems supportive of our hypothesized relationship between copper-gold and yields. There did seem to be a few periods of divergence, which would be a ripe area for further research. But we will move on to calculate and chart the rolling correlation between these two time series.

Recall way back to this post from January where we examined rolling correlations between sector ETFs. We’ll port our work from that project and use rollyapply to calculate the 90-day rolling correlation between copper-gold and yields. We will also rename the column since it will be displayed on the dygraph.

# Calculate the rolling correlation between copper-gold and Treasury yields. 

rolling_cor <- rollapply(tenYear_ratio, 90, 
                                       function(x) cor(x[, 1], x[, 2], use = "pairwise.complete.obs"), 
                                       by.column = FALSE)

names(rolling_cor) <- "Copper/Gold 10-Year Correlation"

Let’s graph that rolling correlation.

dygraph(rolling_cor, main = "Rolling 90-day Correlation Copper-Gold & 10-Year Yield")

It looks like the correlation has been above .4 for the last year and on the increase since the election in November. Peeking back up at the chart of the copper-gold ratio and yields, both those time series have been on the rise since the election as well. Investors seem bullish on the economy and an increase in interest rates.

We can add further context with the minimum, maximum and mean rolling correlations and an event label for the election. Nothing complicated here but it’s a good use of the dyLimit() and dyEvent() functions.

  avg <- round(mean(rolling_cor,  na.rm = T), 2)

  mini <- round(min(rolling_cor,  na.rm = T), 2)
 
  maxi <- round(max(rolling_cor,  na.rm = T), 2)
  
dygraph(rolling_cor, main = "Rolling 90-day correlations Copper-Gold & 10-Year Yield") %>% 
  dyLimit(avg, color = 'purple') %>% 
  dyLimit(mini, color = 'red') %>% 
  dyLimit(maxi, color = 'blue') %>% 
  dyEvent("2016-11-08", "Trump!", labelLoc = "bottom")

That’s all for today. We have examined a hypothesized relationship and found some confirmatory evidence, or at least visual indicators. We have also explored some new tools in the dygraphs world by using dyRoll() and dyAxis(). From a workflow and reproducibility perspective, I like this as a template for importing multiple time series and visualizing the relationship between them in different combinations. When we wrap this into a Shiny app next time, we will see how this can be useful for further exploration. Thanks and see you next time!

Solving Data Science's First Mile Problem

Fri, 07 Apr 2017 00:00:00 +0000

At data.world, we are out to solve the “first mile problem of data science”: helping people obtain and understand the data sets they need. Everybody starts here whether they are analyzing their fantasy football league or working on the Zika pandemic. But unfortunately, as Professor Eric Schwartz at the University of Michigan said to me recently: “for so many people, the first mile of data science is the last mile, because people just quit when they see the data.”

We have begun by asking these two questions:

What if people who work on data projects could adopt the principles and processes of the open source community?

We believe that with transparency, meritocracy and a bottoms-up approach to knowledge management, the world could benefit from the understanding of those who came before them.
What if we could democratize the semantic web and linked data concepts?

Tim Berners-Lee, the creator of HTML and inventor of the hyperlink, has envisioned a future where the web links not only documents, but data too. There’s tremendous power in this vision for linked data across the web. Unfortunately, these concepts are obtuse and not accessible to most people working on data (we are data people after all, and not knowledge engineers).

One group that is using data.world to effectively solve their first mile problem is Data for Democracy. This incredibly diverse group of over 1200 people aims “to be an inclusive community for data scientists and technologists to volunteer and collaborate on projects that make a positive impact on society”. Their Medicare/Medicaid drug spending project makes particularly good use of the unique features of data.world datasets by including files of many different formats, publishing queries, and then building dashboards using Shiny against the normalized data queried from our API. Now their first mile work is captured in an accessible format with clear context, so others will be able to easily and quickly build on it in the future.

Context is king in so many aspects of life, and it’s especially true when you’re working with data. data.world helps you understand the data you are working with by providing a context for datasets that quickly gets you to the who, what, where, when, why and how of the data. Once you understand the data, we want you to get that data into your tools as quickly as possible with our APIs and SDKs, and then get any analysis you’ve done back into data.world for further distribution and discussion.

Who - data.world shows you who the contributors are on a dataset, who has liked it and who is commenting. This allows you to see who is working on this data or project, and what other things they work on by browsing their profiles. You can also request to be a contributor if you have something to add!

What - What is this data, what can it be used for, what have people done with this data? Published queries, exploratory visualizations, shared R markdown and notebooks, documentation and analysis all help answer these three critical questions. Of course, you can use the discussions feature to ask the owners and contributors questions about the data as well.

Where - Where did this data come from? In the world of data, this is intrinsically tied to “who.” In addition to getting all the social context of “who” from our rich profiles, we encourage people to cite the sources of their data in the dataset summary or in further documentation (PDF or Markdown) included in the dataset. This helps users understand the provenance of the data.

When - When has this data changed and when has it been used? With data.world, you can tell when the data was last updated, when analyses were done, and whether people are actively discussing it. All this helps to determine if the data is suitable to your purpose and, if you’re new to a data, if it still fits in the context of the question you’re asking.

Why - Why does this data answer my questions or tell the story it does? Why should I use or trust this data? We love it when people upload a lot more than just the data and code. We specifically allow people to add images and documentation to their datasets to tell their data’s story.

How - How do I work with this data? How is my data being used by others? With our shareable and publishable queries and the ability to share source code alongside the raw data, you don’t have to recreate work that others have done. If you’re a dataset owner, providing the code that operates on the data or sharing a query that cleans or joins the data in interesting ways provides other people a jumping-off point or a way to reproduce your research and test your findings easily.

In a follow-up to this article, we’ll show you how to use our R SDK along with our data normalization features to discover, join and add features to data in RStudio. In the interim, if you want to jump in and start using data.world to power your R projects and analyses the way that Data for Democracy is, I recommend checking out this tutorial, in which we show how to link data from the Washington Post with data discovered in data.world using RStudio.

About the Author

Jon Loyens is the Co-Founder and Chief Product Officer of data.world. In a past life, he was the VP of Engineering for Traveler Products at HomeAway, and before that, a VP of Engineering and Director of the Labs group at Bazaarvoice. As a long-time technology executive in Austin, he has seen the rise of the trends towards data and analytics as a democratizing force. Jon brought A/B testing and data-driven product management to Bazaarvoice, and massively expanded the data programs at HomeAway. Jon introduced GitHub to Bazaarvoice and made OSS contribution a major part of their culture. He’s seen the benefits of openness and transparency in software and, at data.world, is bringing the same benefits to open data.

Organizing DataFest the tidy way

Wed, 05 Apr 2017 00:00:00 +0000

Organizing an event can be a full-time task in and of its own. I have been organizing ASA DataFest for six years at Duke, and over this time, the number of participants has grown from 23 students from Duke only, to 360 students from seven area schools this year!

First, a bit about ASA DataFest: ASA DataFest is a data “hackathon” for students around the U.S., Canada, and Germany (for now; this list has been growing each year). Students spend a weekend working in small teams, around the clock, to find insight and meaning in a large, messy, and rich data set. For almost all students, it is the most complex data they have encountered, and they push themselves to master new skills, resurrect forgotten knowledge, and bring everything they’ve got to compete for the honor of being declared the best by a panel of expert judges.

As an educator, statistician, and data scientist, growth in student interest in this event sounds fantastic to me. However, as the person responsible for running the event at Duke, it has also meant that for the couple months leading up to DataFest, I have almost an additional full-time job dealing with everything from student registrations to promoting the event to putting in food orders. While I have not found an R-based solution for ordering food (yet!), this year I incorporated R and R Markdown in my organization workflow for grabbing, processing, and reporting registration information.

This post highlights using Google Forms for data collection (e.g., registration), the googlesheets package to pull that data into R, and packages from the tidyverse to manipulate, summarise, and visualize that data. Then, we use RPubs for publishing documents to be shared with participants and other constituents.

Here is a list of all packages used in this article:

library(googlesheets)
library(tidyverse)
library(stringr)
library(DT)
library(knitr)

In an effort to make it easier for others organizing DataFests to replicate my workflow, I have created a Google Drive containing all forms needed for registering participants and collecting information from consultants (mentors), and judges. I have also populated these forms with randomly generated names to showcase how these data are processed to yield the rosters and reports that are useful for organizing the event and disseminating registration information. All Google Forms mentioned can be found in the DataFest Organization Google Drive, which is available for public viewing. You can make a copy for your own use.

Additionally, all R scripts and R Markdown documents used to process these data are available on the datafest GitHub repo.

Team sign ups

If a group of students has already formed a team, it makes sense for them to sign up as a team to ensure that they use the same team name and that everyone registers at once. This Google Form is used to sign such students up.

One issue with registering each team as a single entry is that we end up with what we call “wide” data: each row represents a team, and within that row we have information on all students in that team. However for most practical purposes (counting participants, plotting distributions of years and majors, figuring out how many of each size t-shirt to order, etc.) we need the data to be in “long” format, where each row represents a student.

To accomplish this transformation, we first read the data in using the googlesheets package:

part_wide <- gs_title("DataFest [YEAR] @ [HOST] - Team Sign up (Responses)") %>%
  gs_read()

Then, we realize that the variable names are a mess since they come directly from questions on the Google form!

names(part_wide)

##  [1] "timestamp"       "team_name"       "last_name_1"    
##  [4] "first_name_1"    "school_1"        "tshirt_size_1"  
##  [7] "class_year_1"    "major_1"         "email_1"        
## [10] "participation_1" "diet_1"          "last_name_2"    
## [13] "first_name_2"    "school_2"        "tshirt_size_2"  
## [16] "class_year_2"    "major_2"         "email_2"        
## [19] "participation_2" "diet_2"          "last_name_3"    
## [22] "first_name_3"    "school_3"        "tshirt_size_3"  
## [25] "class_year_3"    "major_3"         "email_3"        
## [28] "participation_3" "diet_3"          "last_name_4"    
## [31] "first_name_4"    "school_4"        "tshirt_size_4"  
## [34] "class_year_4"    "major_4"         "email_4"        
## [37] "participation_4" "diet_4"          "last_name_5"    
## [40] "first_name_5"    "school_5"        "tshirt_size_5"  
## [43] "class_year_5"    "major_5"         "email_5"        
## [46] "participation_5" "diet_5"          "photo"

Using stringr, we can get these variable names in concise snake_case shape:

names(part_wide) <- names(part_wide) %>%
  str_replace(" of team member", "") %>%
  str_replace(" in DataFest before", "") %>%
  str_replace(" Check all that apply.", "") %>%
  str_replace("Email address", "email") %>%
  str_replace("Dietary restrictions", "diet") %>%
  str_replace("Check if you agree", "photo") %>%
  str_replace("\\:", "") %>%
  str_replace("-", "") %>%
  str_replace_all(" ", "_") %>%
  tolower()

We can see that things look a lot better now:

names(part_wide)

##  [1] "timestamp"       "team_name"       "last_name_1"    
##  [4] "first_name_1"    "school_1"        "tshirt_size_1"  
##  [7] "class_year_1"    "major_1"         "email_1"        
## [10] "participation_1" "diet_1"          "last_name_2"    
## [13] "first_name_2"    "school_2"        "tshirt_size_2"  
## [16] "class_year_2"    "major_2"         "email_2"        
## [19] "participation_2" "diet_2"          "last_name_3"    
## [22] "first_name_3"    "school_3"        "tshirt_size_3"  
## [25] "class_year_3"    "major_3"         "email_3"        
## [28] "participation_3" "diet_3"          "last_name_4"    
## [31] "first_name_4"    "school_4"        "tshirt_size_4"  
## [34] "class_year_4"    "major_4"         "email_4"        
## [37] "participation_4" "diet_4"          "last_name_5"    
## [40] "first_name_5"    "school_5"        "tshirt_size_5"  
## [43] "class_year_5"    "major_5"         "email_5"        
## [46] "participation_5" "diet_5"          "photo"

Finally, then we use dplyr and tidyr to transform the data from wide to long:

participants <- part_wide %>%
  select(-photo) %>%
  gather(column, entry, last_name_1:diet_5, -timestamp, -team_name) %>%
  mutate(person_in_team = str_match(column, "[0-9]")) %>%
  mutate(column = str_replace(column, "_[0-9]", "")) %>%
  spread(column, entry) %>%
  filter(!is.na(last_name)) %>%
  arrange(team_name, last_name, first_name) %>%
  select(-person_in_team) %>%
  select(timestamp, team_name, first_name, last_name, email, school, 
         class_year, major, participation, diet, tshirt_size)    # reorder

Let’s take a peek:

participants

## # A tibble: 16 x 11
##             timestamp          team_name first_name last_name
##                 <chr>              <chr>      <chr>     <chr>
## 1   4/2/2017 22:20:05      Bae's Theorem   Adrienne    Fuller
## 2   4/2/2017 22:20:05      Bae's Theorem     Sylvia     Hicks
## 3   4/2/2017 22:20:05      Bae's Theorem       Toni   Simpson
## 4   4/2/2017 22:20:05      Bae's Theorem      Vicky     Water
## 5    4/4/2017 1:03:26      Bayes Anatomy   Meredith      Gray
## 6    4/4/2017 1:03:26      Bayes Anatomy      Derek  Shepherd
## 7   4/3/2017 16:14:00         Fake iid's    Carolyn      Byrd
## 8   4/3/2017 16:14:00         Fake iid's     Gordon   Hawkins
## 9   4/3/2017 16:14:00         Fake iid's    Cecilia   Pittman
## 10  4/3/2017 16:14:00         Fake iid's       Paul      Rios
## 11  4/3/2017 16:14:00         Fake iid's       Ryan      Rose
## 12 3/31/2017 23:55:00 Passive Regression     Amanda      Boyd
## 13 3/31/2017 23:55:00 Passive Regression       Rosa       Fox
## 14 3/31/2017 23:55:00 Passive Regression      Lucas  Gonzales
## 15  4/1/2017 20:14:05            The Pit      James   Andrews
## 16  4/1/2017 20:14:05            The Pit        Tom  Lawrence
## # ... with 7 more variables: email <chr>, school <chr>, class_year <chr>,
## #   major <chr>, participation <chr>, diet <chr>, tshirt_size <chr>

We can now easily look for duplicates (sometimes students sign up twice or more times) or use these data to explore the various features of participants.

Individual sign-ups

If a student is wanting to participate in DataFest but they don’t have a team in mind, we ask them to fill out a brief survey where they answer questions about their background as well as how much time they are wanting to commit to DataFest, ranging from “I’m in it to win it” to “I’m more interested in the experience, and am not really sure if I’ll submit a final presentation.”

Sometimes students find a team and register with that team after having filled out this survey. These students should be removed from the list of those looking for teammates, though there is no easy way for them to do so in Google Forms (they can’t go back and remove their response).

However we can easily do this with an anti_join. Suppose this data frame is called looking, and remember that the earlier data frame of students registering with teams was called participants.

looking <- anti_join(looking, participants, by = "email")

Then, the survey results are made available to the same students who are looking for teammates so that they can match up with others and form a team.

looking %>%
  select(first_name, last_name, participation_level, class_year, major, school, participation_before, email) %>%
  arrange(participation_level, class_year, major, school) %>%
  datatable()

Here we make use of the datatable function in the DT package to display the list of students in a pretty and easily sortable and searchable format.

Consultants and judges

Using a similar approach we can also grab, organize, and report lists of consultants and judges. All relevant code for this can be found in the datafest GitHub repo.

Participant summary

Now that we have our participant data in a tidy format, we can visualize distributions of majors, years, previous participation etc.

For example, we can count how many teams are participating from each school:

participants %>%
  distinct(team_name, .keep_all = TRUE) %>%
  count(school) %>%
  arrange(desc(n))

## # A tibble: 3 x 2
##                    school     n
##                     <chr> <int>
## 1           Faber College     2
## 2 Port Chester University     2
## 3     Harrison University     1

or visualize the distribution of class years per school:

ggplot(data = participants, aes(x = school, fill = class_year)) +
  geom_bar(position = "fill") +
  labs(title = "Schools and class years")

Information guides

We can also use R Markdown to create documents that are mostly text, that introduce the event to the participants, consultants, and judges. Then, summary statistics and visualizations of the participants can easily be included in these guides.

Sample guides for participants and consultants/judges can also be found on the GitHub repo.

And finally, all of these can be published on RPubs. However, note that these documents will be publicly available.

March '17 Tips and Tricks

Fri, 31 Mar 2017 00:00:00 +0000

This month’s Tips and Tricks focus is on file navigation. Many of these tips are straightforward - more tip than trick - but they can save quite a bit of time and frustration.

Open Recent

Please don’t spend time navigating through your folder structure to find a recent project or file.

Go to File/Function

RStudio automatically indexes all open files as well as files in the current project. The index includes function calls and objects within the files. To search the index, enter a search string in the Go to file/function window in the main toolbar or use the keyboard shortcut Ctrl + ..

The search supports fuzzy matching as long as the search string is in the correct order. For example, tiptric will match files and function names like march-tips-and-tricks.Rmd.

Find in File

In addition to the index, RStudio can quickly search the text of any file(s) for a string or regular expression. Open the Find in Files dialogue by navigating through the toolbar Edit -> Find in Files or using the keyboard shortcut Cmd / Ctrl + Shift + F.

Tabs

Close All and Close All Except Current

If you’re like me, you often end up with a whole bunch of files open and the number of tabs in the source pane causes anxiety. Don’t click through the tabs closing each individually.

Search for a Tab

If you don’t get anxiety from multiple tabs, you may find the keyboard shortcuts for switching between tabs helpful. My favorite is Ctrl + Shift + . which opens a search dialogue for the currently open tabs.

Set Working Directory

Sometimes I find myself inside a file and I have no idea where that file is. In this scenario, I often use the following trick to orient myself:

The other options, To Project Directory and To File Pane Location are also very helpful. These commands do not have key bindings by default, but you can create your own keyboard shortcuts for these actions: Tools -> Modify Keyboard Shortcuts...

File Pane: Go To Working Directory

After mastering tip 5, you can easily navigate from the file pane to the working directory:

File Pane: Go To Project Root Directory

Finally, it is really easy to go to the root of a project directory:

Be sure to follow @rstudiotips for more frequent updates. Tweet us @rstudio if there’s a file navigation trick you wish we’d implement!

R and Singularity

Wed, 29 Mar 2017 00:00:00 +0000

R (https://www.r-project.org) is a premier system for statistical and scientific computing and data science. At its core, R is a very carefully curated high-level interface to low-level numerical libraries. True to this principle, R packages have greatly expanded the scope and number of these interfaces over the years, among them interfaces to a large number of distributed and parallel computing tools. Despite its impressive breadth of sophisticated high-performance computing (HPC) tools, R is not often that widely used for “big” problems.

I believe the idiosyncrasies of most HPC technologies represent the major road block to their adoption (in any language or system). HPC technologies are often difficult to set up, use, and manage. They often rely on frequently changing and complex software library dependencies, and sometimes highly specific library versions. Managing all this boils down to spending more time on system administration, and less time on research.

How do we make things easier? One approach to help accelerate the adoption of HPC technology by the R community uses Singularity, a modern application containerization technique suited to HPC (http://singularity.lbl.gov/).

Containers

A container is a collection of the software requirements to run an application. Importantly, containers are defined and generated from a simple text recipe that can be easily communicated and versioned. Containers leverage modern operating system capabilities for virtualizing process and name spaces in a high-performance, low-overhead way. Container technology allows us to quickly turn recipes into runnable applications, and then deploy them anywhere.

The success of Docker, CoreOS, and related systems in enterprise business applications shows that there is a huge demand for lightweight, versionable, and portable containers. Notably, these technologies have not been all that widely successful in HPC settings, despite significant effort. Shifter (https://github.com/NERSC/shifter) is the most successful application of Docker to HPC, and while it is very impressive, it suffers from a few important drawbacks. The root-capable daemon program used by Docker is difficult to accommodate in many HPC environments. And the relatively heavy-weight nature of Docker virtualization can degrade the performance of high-performance hardware resources like Infiniband networking.

Singularity is a lightweight and very simple container technology that is particularly well-suited to HPC environments. Singularity virtualizes the minimum amount necessary to compute, allowing applications full access to fast hardware resources like Infiniband networks and GPUs. And Singularity runs without a server at all, eliminating possible server security exploits. The minimalist philosophy of Singularity makes it easy to install and run on everything from laptops to supercomputers, promoting the ability to quickly test containers before using them across large systems. Singularity is now widely available in supercomputer centers across the world.

Reproducible research

Publishing results with code and data that can be reproduced and validated by others is an obviously important concept that has seen increased urgency these days. The idea is an old one that has been supported by S, S+ and R from the beginning with ideas like Sweave and more recently knitr and R markdown. R even promotes reproducible simulation in distributed/parallel settings by including high-quality, reproducible, distributed random number generators out of the box.

However, as R integrates with an increasing number of external libraries and frameworks like cuDNN, Spark, and others, the ability to reproduce the software environment that R runs in is becoming both more important and more complex. Containers help us define these complex set ups with simple, versionable text files, and then portably run them in diverse environments.

Examples

The following examples assume that Singularity is installed on your system. See http://singularity.lbl.gov/ for details – it’s very easy to install. The examples can be run from nearly any modern Unix operating system, although the processor architecture must be supported by the container operating system.

Hello TensorFlow

The first example below shows a canonical “hello world” program. Instead of a completely trivial example, we print “Hello, TensorFlow!” using TensorFlow from R via Python (https://github.com/tensorflow/tensorflow, https://github.com/python/cpython), introducing a complex but typical software dependency chain. A test program validates operation by printing the “hello world” message from R through Tensorflow. The container generically will run any R program named main.R in its working directory.

Here is the Singularity container definition file for the example using the Ubuntu Xenial operating system. (Note that you can build a container from this definition file on any Singularity-supported operating system.)

BootStrap: debootstrap
OSVersion: xenial
MirrorURL: http://archive.ubuntu.com/ubuntu/

%post
  sed -i 's/main/main restricted universe/g' /etc/apt/sources.list
  apt-get update

  # Install R, Python, misc. utilities
  apt-get install -y libopenblas-dev r-base-core libcurl4-openssl-dev libopenmpi-dev openmpi-bin openmpi-common openmpi-doc openssh-client openssh-server libssh-dev wget vim git nano git cmake  gfortran g++ curl wget python autoconf bzip2 libtool libtool-bin python-pip python-dev
  apt-get clean
  locale-gen en_US.UTF-8

  # Install Tensorflow
  pip install tensorflow

  # Install required R packages
  R --slave -e 'install.packages("devtools", repos="https://cloud.r-project.org/")'
  R --slave -e 'devtools::install_github("rstudio/tensorflow")'

%test
  #!/bin/sh
  exec R --slave -e "library(tensorflow); \
                     sess  <- tensorflow::tf\$Session(); \
                     hello <- tensorflow::tf\$constant('Hello, TensorFlow!'); \
                     sess\$run(hello)"


%runscript
  #!/bin/bash
  Rscript --slave "main.R"

TIP If you’re running on Red Hat or CentOS, you’ll need the debootstrap program: sudo yum install debootstrap. See the Singularity documentation for more information.

Assuming that the above definition file is named tensorflow.def, you can bootstrap a Singularity container image named tensorflow.img with:

sudo rm -f tensorflow.img && \
sudo singularity create --size 4000 tensorflow.img && \
sudo singularity bootstrap tensorflow.img tensorflow.def

The %post section of the definition file installs R, Python, Tensorflow and miscellaneous utilities into the container. The %test section runs the “hello world” program as an example to verify things are working. The %run section of this example simply runs an arbitrary user R program named main.R in the container’s working directory.

Run the “hello world” %test script with:

singularity test tensorflow.img

I love Singularity’s ability to include unit tests in container definition files – it reminds me of building R packages! I encourage using the test section judiciously to confirm that the container will work as intended.

You can run an arbitrary R program in the container by creating a main.R file in the container working directory and running:

singularity run tensorflow.img

Full-genome variant Principal Components

The previous example illustrated a complex tool chain, but only running on a single computer. This example is closer to a complete distributed R application.

Genomic variants record differences in a genome relative to a reference. Many types of differences exist, see for instance https://en.wikipedia.org/wiki/Structural_variation. This example focuses on differences among the 2,504 whole human genomes curated by the 1000 Genomes Project (see: “A global reference for human genetic variation”, The 1000 Genomes Project Consortium, Nature 526, 68-74 (01 October 2015) doi:10.1038/nature15393). The example downloads whole genome data files in VCF 4.1 format. Although the 1000 Genome Project data files are used here, the example will work for any input set of VCF files (it processes all files named *.vcf.gz in the working directory).

The example constructs a sparse 2,504 row (people) by 81,271,844 column (genomic variants) R matrix from the VCF data files. The matrix entries are one if a particular variant occurs in the person, or a zero otherwise. Because not every person exhibits every variant, the matrix is very sparse with about 9.8 billion nonzero-elements, or about 2% fill-in. Rather than construct a single giant sparse matrix, the example partitions the data and saves many smaller sub-matrices each with CHUNKSIZE non-zero elements as R data files in the working directory, where CHUNKSIZE is an optional user-defined parameter that defaults to a value based on system memory size.

The example computes the first NCOMP principal components, where NCOMP is a user-specified environment variable specified by the user, of sparse genomic variant VCF files. The example is very general, requiring an arbitrary number of VCF data files as input and running on any number of computers. It uses MPI to coordinate parallel activity across computers, along with the Rmpi, doMPI, and foreach packages in R. The choice of MPI is well-suited to supercomputer deployment, and the example assumes that MPI is available along with the following assumptions:

Launched by MPI
One or more gzip-compressed variant files ending in “.vcf.gz” (the program will use all files matching this pattern)
The input variant files are split up among working directories across the worker computers – each worker will parse and process only the variant files in its local working directory
Optional CHUNKSIZE environment variable in number of variants per chunk
Optional NCOMP environment variable specifying the number of principal components to return, defaulting to 3

A successful run produces the following output:

A file ‘pca.rdata’ in serialized R format containing the largest NCOMP singular values and corresponding principal component vectors of the variant data

This example was designed for deployment with supercomputer systems in mind. See https://github.com/bwlewis/1000_genomes_examples for other implementations that don’t require MPI.

Singularity encapsulates the program logic and the external library dependency chain (MPI, etc.) required by the computation in the following definition file:

BootStrap: debootstrap
OSVersion: xenial
MirrorURL: http://archive.ubuntu.com/ubuntu/
Include: bash

%post
  sed -i 's/main/main restricted universe/g' /etc/apt/sources.list
  apt-get update

  # Install R, openmpi, misc. utilities:
  apt-get install -y libopenblas-dev r-base-core libcurl4-openssl-dev libopenmpi-dev openmpi-bin openmpi-common openmpi-doc openssh-client openssh-server libssh-dev wget vim git nano git cmake  gfortran g++ curl wget python autoconf bzip2 libtool libtool-bin
  apt-get clean

  # Install required R packages
  R --slave -e 'install.packages(c("irlba", "doMPI"), repos="https://cloud.r-project.org/")'

  # Install simple VCF parser helper
  wget https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/master/parse.c && cc -O2 parse.c && mv a.out /usr/local/bin/parsevcf && rm parse.c

  # Set up unit test
  mkdir -p /usr/local/share/R
  chmod a+rwx /usr/local/share/R
  wget https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/master/unit.R && mv unit.R /usr/local/share/R/

  # This is the main R program run by /singularity
  wget https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/master/pca-mpi.R && mv pca-mpi.R /usr/local/share/R/


%test
  #!/bin/sh
  exec Rscript --slave "/usr/local/share/R/unit.R"

%runscript
  #!/bin/bash
  Rscript --slave "/usr/local/share/R/pca-mpi.R"

Build and bootstrap a Singularity container using the variant_pca.def definition file with:

sudo rm -f variant_pca.img && \
sudo singularity create --size 4000 variant_pca.img && \
sudo singularity bootstrap variant_pca.img variant_pca.def

Unit test

The container includes a simple unit test that verifies MPI operation invoked by:

mpirun -np 4 singularity test variant_pca.img

Small example

A small, fast-running example computes principal components for the first 10,000 variants from the 1000 Genomes Project chromosomes 21 and 22 as follows:

wget https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/extra/chr21.head.vcf.gz
wget https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/extra/chr22.head.vcf.gz
LANG=C CHUNKSIZE=10000000 mpirun -x LANG -x CHUNKSIZE -np 2 singularity run -H $(pwd) variant_pca.img

Read the output pca.rdata file from R using readRDS(). The following code plots the first three estimated principal components.

x <- readRDS('pca.rdata')
library(lattice)
splom(x$v)

We see some obvious clusters in the data, but the clusters are not all that well-defined because we only use data from two smaller chromosomes (21 and 22) in this example. The clusters correspond to distinct genetic superpopulations. See the following example for a refined plot using the whole genomes.

Full-sized example

Finally, compute the whole genome principal components across all chromosomes and all 2,504 people in the 1000 Genomes project with:

# Remove small example files if they exist
rm -f chr21.head.vcf.gz chr22.head.vcf.gz

# Download the variant files
j=1
while test $j -lt 23; do
  wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/release/20130502/ALL.chr${j}.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz &
  j=$(( $j + 1 ))
done
wait

When running on more than one computer, first distribute the vcf.gz files by scattering them across working directories on each computer. Each computer will only process the files located in its working directory, so copy a subset of the files to each computer.

The Singularity container image must also be available to run on each computer, so copy the image to each one.

Now scatter the *.vcf.gz files across your MPI computers, for instance using scp. Let’s assume for this example that we have four total computers. Then we need to invoke the program on 4 + 1 = 5 total MPI hosts, as outlined in https://cran.r-project.org/web/packages/doMPI/vignettes/doMPI.pdf (the first listed host will operate as the R master program in a master/slave configuration).

Assume that our four host computers are listed in a comma-separated list by the environment variable HOSTS, for instance by

HOSTS=10.0.0.1,10.0.0.1,10.0.0.2,10.0.0.3,10.0.0.4

Then a typical openmpi invocation is (for our four hosts):

LANG=C CHUNKSIZE=10000000 mpirun -wd $(pwd) -x LANG -x CHUNKSIZE -np 5 -host $(HOSTS) singularity run -H $(pwd) variant_pca.img

Replace the host list and -np 5 with the number of computers available in your cluster plus one. Or, submit the job using an available cluster job manager like Slurm. See https://cran.r-project.org/web/packages/doMPI/vignettes/doMPI.pdf for more details on using MPI with R.

Example output

To give you an idea of performance, I ran this example on four Amazon EC2 r4-4xlarge instances. The parsing step completed in about 20 minutes, and principal component computation took about 11 minutes (680 seconds).

As with the small example above, we can read the output file and plot the principal components:

x <- readRDS('pca.rdata')
library(lattice)
splom(x$v)

The resulting clusters are much more highly defined, and split into four or five very well-defined data clusters, corresponding almost exactly to the NIH superpopulation categories for each person. Some of the data clusters themselves exhibit sub-cluster structure.

Additional Notes

The computation uses an R program downloaded from https://raw.githubusercontent.com/bwlewis/1000_genomes_examples/master/pca-mpi.R that we don’t reproduce here. See that file and https://github.com/bwlewis/1000_genomes_examples/blob/master/PCA_whole_genome.Rmd for additional notes.

The computation proceeds in two sequential phases, first processing the raw VCF files into chunks of sparse R matrices corresponding to the variant data, and then computing principal components on the R matrices. Parallel computation is used within each phase.

Sparse matrix chunk size is specified by the user with the environment variable CHUNKSIZE to indicate the maximum number of nonzero matrix elements per chunk. If unspecified, CHUNKSIZE is automatically determined based on a heuristic using the host computer’s memory size.

The first processing phase of the computation stores the R sparse matrix chunks corresponding to the input available VCF files for re-use iteratively by the algorithm. In particular, this algorithm process the chunked VCF data out of core – alternative versions of the program pin sparse matrix chunks in memory on each computer and avoid intermediate file system use. That can be obviously more efficient than using a file system. But, importantly, the file system approach scales easily. In particular, this program will run (slowly) on a single laptop even if the total variant sparse matrix size vastly exceeds available RAM size. Thus, this example trades best performance for flexibility. Despite this trade off, performance can be excellent in the example, thanks to the efficient algorithm used and the fact that files are cached in each computer’s buffer cache if memory permits.

Some Random Weekend Reading

Fri, 24 Mar 2017 00:00:00 +0000

Few of us have enough time to read, and most of us already have depressingly deep stacks of material that we would like to get through. However, sometimes a random encounter with something interesting is all that it takes to regenerate enthusiasm. Just in case you are not going to get to a book store with a good technical section this weekend, here are a few not-quite-random reads.

Deep Learning by Goodfellow, Bengio and Courville is a solid, self-contained introduction to Deep Learning that begins with Linear Algebra and ends with discussions of research topics such as Autoencoders, Representation Learning, and Boltzman Machines. The online layout extends an invitation to click anywhere and begin reading. Sampling the chapters, I found the text to be engaging reading; much more interesting and lucid than just an online resource. For some Deep Learning practice with R and H2O, have a look at the post Deep Learning in R by Kutkina and Feuerriegel.

However, if you are under the impression that getting a handle on Deep Learning will get you totally up to speed with neural network buzzwords, you may be disappointed. Extreme Learning Machines, which “aim to break the barriers between the conventional artificial learning techniques and biological learning mechanisms”, are sure to take you even deeper into the abyss. For a succinct introduction to ELMs with and application to handwritten digit classification, have a look at the recent paper by Pang and Yang. For more than an afternoon’s worth of reading, browse through the IEEE Intelligent Systems issue on Extreme Learning Machines here, and the other resources collected here. See the announcement of the 2014 conference for the full context of the quote above.

For something a little lighter and closer to home, Christopher Gandrud’s page on the networkD3 package is sure to set you browsing through Sankey Diagrams and Force Directed Drawing Alorithms.

library(networkD3)
# Load data
data(MisLinks)
data(MisNodes)

# Plot
forceNetwork(Links = MisLinks, Nodes = MisNodes,
            Source = "source", Target = "target",
            Value = "value", NodeID = "name",
            Group = "group", opacity = 0.8)

Finally, if you are like me and think that the weekends are for catching up on things that you should probably already know, but on which you might be a bit shaky, remember that you can never know enough about GitHub. Compliments of GitHub’s Carolyn Shin, here is some online GitHub reading: GitHub Guides, GitHub on Demand Training, and an online version of the Pro Git Book.

Reading recommendations go both ways. Please feel free to comment with some recommendations of your own.

February 2017 New Package Picks

Wed, 22 Mar 2017 00:00:00 +0000

One hundred and forty-five new packages were added to CRAN in February. Here are 47 interesting packages organized into five categories; Biostatistics, Data, Data Science, Statistics and Utilities.

Biostatistics

BaTFLED3D v0.1.7: Implements a machine learning algorithm to make predictions and determine interactions in data that varies along three independent modes. It was developed to predict the growth of cell lines when treated with drugs at different doses. The vignette shows an example with simulated data.
DClusterm v0.1: Implements methods for the model-based detection of disease clusters. Look at the JSS paper for details.

Data

ccafs v0.1.0: Provides access to data stored in Amazon S3 collected by the research program on Climate Change, Agriculture and Food Security. There is an introduction.
dataseries v0.1.0: Provides access to Swiss data series from dataseries.org.
fpp2 v2.0: Provides all of the data sets used in the online book Forecasting: Principles and Practice.
rnaturalearthdata v0.1.0: Contains the world vector map data used in Natural Earth.

Data Science

autothresholdr v0.2.0: Is an R port of the ImageJ image processing program. The vignette shows how to use it.
dlib v1.0: Provides an Rcpp interface to dlib, the C++ toolkit containing machine learning algorithms and computer vision tools.
liquidSVM v1.0.1: Provides several functions to support a fast support vector machine implementation. There is a demo vignette and supplemental installation documentation.
OOBCurve v0.1: Provides a function to calculate the out-of-bag learning curve for random forest models built with the randomForest or ranger packages.
opusminer v0.1-0: Provides an interface to the OPUS Miner algorithm for finding the top-k, non-redundant itemsets from transaction data.

Statistics

BayesCombo v1.0: Implements Bayesian meta-analysis methods to combine diverse evidence from multiple studies. The vignette provides a detailed example.
BayesianTools v0.1.0: Implements various Metropolis MCMC variants (including adaptive and/or delayed rejection MH), the T-walk, differential evolution MCMCs, DREAM MCMCs, and a sequential Monte Carlo particle filter, along with diagnostic and plot functions. The vignette will get you started.
FRK v0.1.1: Implements the Fixed Rank Kriging methods presented by Cressie and Johannesson in their 2008 paper. An extended vignette explains the math and provides several examples.

glmmTMB v0.1.1: Provides functions to fit Generalized Linear Mixed Models using the Template Model Builder (TMB) package. There are vignettes for getting started, Covariance Structures, post-hoc MCMC, simulation, and troubleshooting.
IMIFA v1.1.0: Provides flexible Gibbs sampler functions for fitting Infinite Mixtures of Infinite Factor Analysers and related models, introduced by Murphy et al. in a 2017 paper. The vignette shows examples.
ImputeRobust v1.1-1: Provides new imputation methods for the mice package, based on generalized additive models for location, scale, and shape (GAMLSS) as described in de Jong, van Buuren and Spiess.
lmvar v1.0.0: Provides functions to run linear regressions in which both the expected value and variance can vary by observation. Vignettes provide an introduction and explain the details of the math.
metaviz v0.1.0: Implements the rainforest plots proposed by Schild & Voracek as a variant of the forest plots used for meta-analysis. The vignette describes their use.

prophet v0.1: Implements a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly and weekly seasonality, plus holidays. There is a Quick Start guide.
robustarima v0.2.5: Provides functions for fitting a linear regression model with ARIMA errors, using a filtered tau-estimate.
rpgm v0.1.3: Provides functions that use the Ziggurat Method to generate Normal random variates quickly.
sarima v0.4-3: Provides functions for simulating and predicting with seasonal ARIMA models. The vignette presents a use case.
sppmix v1.0.0.0: Implements classes and methods for modeling spatial point patterns using inhomogeneous Poisson point processes, where the intensity surface is assumed to be analogous to a finite additive mixture of normal components, and the number of components is a finite, fixed or random integer.

Look here for documentation.

walkr v0.3.4: Provides functions to sample from the interaction of a hyperplane and an N Simplex. The vignette describes the math and the algorithms involved.

Utilities

AWR v1.11.88: Installs the compiled Java modules of the Amazon Web Services SDK for R packages interacting with AWS.
AWR.Kinesis v1.7.3: Provides functions for fetching data from Amazon Kinesis Streams using the Java-based MultiLangDaemon.
AWR.KMS v0.1: Provides functions to encrypt plain text and ‘decrypt’ cipher text using encryption keys hosted at Amazon Web Services Key Management Service.
drake v2.0.0: Provides a reproducible build system for R that understands the workflow dependency structure, builds out-of-date targets, skips up-to-date targets, and offers two forms of parallel computing. There is a quickstart guide.
filesstrings v0.3.1: Provides tools for manipulating strings and files on top of stringr.
ggraph v1.0.1: Provides functions to extend ggplot2 to graph visualizations. There are vignettes showing how to work with Edges, Layouts, and Nodes.

odbc v1.0.1: Uses the DBI interface to implement a connection to ODBC compatible databases.
sonify v0.1-0: Contains a function to transform univariate data, sampled at regular or irregular intervals, into a continuous sound with time-varying frequency. The function is intended as a substitute for R’s plot function to simplify data analysis for the visually impaired.
widgetframe v0.1.0: Provides two functions for working with htmlwidgets and iframes, which may be useful when working with Wordpress or R Markdown. There is a vignette.
wrapr v0.1.1: Contains the debugging functions DebugFnW to capture function context on error for debugging, and let to convert non-standard evaluation interfaces to standard evaluation interfaces.

Quandl and Forecasting

Fri, 17 Mar 2017 00:00:00 +0000

Welcome to another installment of Reproducible Finance with R. Today we are going to shift focus in recognition of the fact that there’s more to finance than stock prices, and there’s more to data download than quantmod/getSymbols. In this post, we will explore oil prices using data from Quandl, a repository for both free and paid data sources. We will also get into the forecasting game a bit and think about how best to use dygraphs when visualizing predicted time series as an extension of historical data. We are not going to do anything too complex, but we will expand our toolkit by getting familiar with Quandl, commodity prices, the forecast package, and highcharter. Our ultimate goal is to build a tool where an end user can explore and forecast commodities prices via a Shiny app. The final app can be viewed here.

Before we dive in, a few thoughts to frame this post.

We are using oil data from Quandl, but the original data is from FRED. There’s nothing wrong with grabbing the data directly from FRED, of course, and I browse FRED frequently to check out economic data, but I tend to download the data into my RStudio environment using Quandl. I wanted to introduce Quandl today because it’s a nice resource and it’s gaining in popularity. If you work in the financial industry, you might start to encounter it in your work.
This post marks our first foray into the world of predictive modeling, albeit in a very simple way. But the complexity and accuracy of the forecasting methodology we use here is almost irrelevant since I expect that most R coders, whether in industry or otherwise, will have their own proprietary models. Rather, what I want to accomplish here is a framework where models can be inserted, visualized, and scrutinized in the future. I harp on reproducible workflows a lot, and that’s not going to change today because one goal of this Notebook is to house a forecast that can be reproduced in the future (at which point, we will know if the forecast was accurate or not), and then tweaked/criticized/updated/heralded.

Let’s get to the data download! In the chunk below, as we import WTI oil prices, notice that Quandl makes it easy to choose types of objects (raw/dataframe, xts, or zoo), periods (daily, weekly, or monthly) and start/end dates.

library(Quandl)
library(tidyverse)
library(tidyquant)
library(timetk)
library(forecast)
library(highcharter)

# You might want to supply an API key. It's free to sign up.
# Quandl.api_key("your API key here")

# Start with daily data. Note that "type = raw" will download a data frame.
oil_daily <- Quandl("FRED/DCOILWTICO", 
                    type = "raw", 
                    collapse = "daily",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# Now weekly and let's use xts as the type.
oil_weekly <- Quandl("FRED/DCOILWTICO", 
                     type = "xts", 
                     collapse = "weekly",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# And monthly using xts as the type.
oil_monthly <- Quandl("FRED/DCOILWTICO", 
                      type = "xts", 
                      collapse = "monthly",  
                    start_date = "2008-01-01", 
                    end_date = "2018-01-01")

# Have a quick look at our three  objects. 
head(oil_daily)

##         Date Value
## 1 2017-12-29 60.46
## 2 2017-12-28 59.84
## 3 2017-12-27 59.67
## 4 2017-12-26 59.55
## 5 2017-12-22 58.25
## 6 2017-12-21 58.34

head(oil_weekly)

##             [,1]
## 2008-01-06 97.90
## 2008-01-13 92.74
## 2008-01-20 90.55
## 2008-01-27 90.37
## 2008-02-03 89.03
## 2008-02-10 91.77

head(oil_monthly)

##            [,1]
## Jan 2008  91.67
## Feb 2008 101.78
## Mar 2008 101.54
## Apr 2008 113.70
## May 2008 127.35
## Jun 2008 139.96

Note that we specified the start date as January 1, of 2008 and end date of January 1, 2018 so we will be working with 10 years of data.

Each of the oil data objects we created would work well for the rest of this project, but let’s stick with the monthly data. I don’t love the formate of the column so let’s use the seq() function and mdy() from the lubridate package to put the date into a nicer format.

index(oil_monthly) <- seq(mdy('01/01/2008'), mdy(last(index(oil_monthly))), by = 'months')

head(index(oil_monthly))

## [1] "2008-01-01" "2008-02-01" "2008-03-01" "2008-04-01" "2008-05-01"
## [6] "2008-06-01"

Now we have a cleaner date format and our price data object is in good shape. Let’s fire up highcharter and visualize our price history. Since we imported an xts object directly from Quandl, we can plug it straight into the hchart() function.

hchart(oil_monthly)

Or we can use the highchart(type = "stock") code flow to produce the same chart. Let’s also add a $ label to the y-axis with hc_yAxis.

highchart(type = "stock") %>% 
  hc_add_series(oil_monthly, color = "cornflowerblue") %>% 
  hc_yAxis(title = list(text = "monthly price"),
           labels = list(format = "${value}"),
           opposite = FALSE) %>% 
  hc_add_theme(hc_theme_flat())

Nothing too shocking here. We see a peak in mid-2008, followed by a precipitous decline through the beginning of 2009.

Now we’ll make things a bit more interesting and try to extract some meaning from that data. First, let’s fit an arima model to our time series using the auto.arima() function from the forecast package. This is for illustrative purposes only and probably isn’t the best model for oil prices.

auto.arima(oil_monthly)

## Series: oil_monthly 
## ARIMA(1,1,0) 
## 
## Coefficients:
##          ar1
##       0.3218
## s.e.  0.0871
## 
## sigma^2 estimated as 47.84:  log likelihood=-398.55
## AIC=801.1   AICc=801.2   BIC=806.65

Next we can use the forecast() function to predict what oil prices will look like over the next six months, based on the arima model we just fit.

auto.arima(oil_monthly) %>% 
  forecast(h = 6)

##     Point Forecast    Lo 80    Hi 80    Lo 95     Hi 95
## 121       61.44478 52.58047 70.30909 47.88799  75.00157
## 122       61.76170 47.06935 76.45406 39.29169  84.23172
## 123       61.86370 42.48557 81.24182 32.22741  91.49998
## 124       61.89652 38.60035 85.19270 26.26809  97.52495
## 125       61.90709 35.21664 88.59754 21.08757 102.72661
## 126       61.91049 32.19775 91.62322 16.46878 107.35219

The point forecast is around $62. It looks like the 95% confidence interval 6 months out has a high of $16 and a low of $107. We won’t dwell on these numbers because I imagine you will want to use your own model here - this code flow is more of a skeleton where other models can be inserted and then tested or evaluated at a later date.

Let’s move on to visualizing the results of the forecast along with the historical data. A great feature of highcharter is that it accepts the results of forecast directly. We can pipe the model and forecast results directly to hchart().

auto.arima(oil_monthly) %>% 
  forecast(h = 6) %>% 
  hchart() %>% 
  hc_title(text = "Oil historical and forecast") %>% 
  hc_yAxis(title = list(text = "monthly price"),
           labels = list(format = "${value}"),
           opposite = FALSE) %>% 
  hc_add_theme(hc_theme_flat()) %>% 
  hc_navigator(enabled = TRUE)

That’s all for today. We have gotten some familiarity with Quandl, used forecast() to predict the next six months of oil prices, and seen how smooth it is to pass forecasts to highcharter. Next time, we will wrap this into a Shiny app so that users can choose their own parameters, and choose different commodities. See you then!

Note: this post was updated on August 2, 2018.

Why I love R Notebooks

Wed, 15 Mar 2017 00:00:00 +0000

Note: R Notebooks requires RStudio Version 1.0 or later

I’m a big fan of the R console. During my early years with R, that’s all I had, so I got very comfortable with pasting my code into the console. Since then I’ve used many code editors for R, but they all followed the same paradigm – script in one window and get output in another window. Notebooks on the other hand combine code, output, and narrative into a single document. Notebooks allow you to interactively build narratives around small chunks of code and then publish the complete notebook as a report.

R Notebooks is a new feature of RStudio that combines the benefits of other popular notebooks (such as Jupyter, Zeppelin, and Beaker) with the benefits of R Markdown documents. As a long-time R user, I was skeptical that I would like this new paradigm, but after a few months I became a big fan. Here are my top three reasons why I love R Notebooks.

Number 3: Notebooks are for doing science

If scripting is for writing software, then notebooks are for doing data science. In my high school science class I used a laboratory notebook that contained all my experiments. When I conducted an experiment, I drew sketches and wrote down my results. I also wrote down my ideas and thoughts. The process had a nice flow which helped me improve my thinking. Doing science with physical notebooks is an idea that is centuries old.

Electronic notebooks follow the same pattern as physical notebooks, but apply the pattern to code. With notebooks, you break your script into manageable code chunks. You add narrative and output around the code chunk, which puts it into context and makes it reproducible. When you are done, you have an elegant report that can be shared with others. Here is the thought process for doing data science with notebooks:

I have a chunk of code that I want to tell you about.
I am going to execute this chunk of code and show you the output.
I am going to share all chunks of code with you in a single, reproducible document.

If you do data science with R scripts, on the other hand, you develop your code as a single script. You add comments to the code, but the comments tend to be terse or nonexistent. Your output may or may not be captured at all. Sharing your results in a report requires a separate, time consuming process. Here is the thought process for doing data science with scripts:

I have a thousand lines of code and you get to read my amazing comments!
Hold onto your hats while I batch execute this entire script!
You can find my code and about 50 plots under the project directory (I hope you have permissions).

Number 2: R Notebooks have great features

R Notebooks are based on R Markdown documents. That means they are written in plain text and work well with version control. They can be used to create elegantly formatted output in multiple document types (e.g. HTML, PDF, and Word).

R Notebooks have some features that are not found in traditional notebooks. These are not necessarily inherent differences, but differences of emphasis. For example, R Markdown documents give you many options when selecting graphics, templates, and formats for your output.

Feature	R Notebooks	Traditional Notebooks
Plain text representation	✓
Same editor/tools used for R scripts	✓
Works well with version control	✓
Create elegantly formatted output	✓
Output inline with code	✓	✓
Output cached across sessions	✓	✓
Share code and output in a single file	✓	✓
Emphasized execution model	Interactive & Batch	Interactive

When you execute a code chunk in an R Notebook, the output is cached and rendered inside the IDE. When you save the notebook, the same cache is rendered inside a document. The HTML output of R Notebooks is a dual-file format that contains both the HTML and the R Markdown source code. The dual format gives you a single file that can be viewed in a browser or opened in the RStudio IDE.

R Notebooks for data science

The following table summarizes the differences between notebooks and scripts.

Activity	R Notebook	R Script
Building a narrative	Rich text is added throughout the analytic process to describe the motivation and the conclusions for each chunk of the code.	Comments are added to the script, and a report that describes the entire analysis is drafted separately after the script is completed.
Organizing plots, widgets, and tables	All output is embedded in a single document and collocated with the narrative and code chunk to which it belongs.	Each individual output is sent to file and is collected later into a report.
Creating reports	Rendering the final report is instant. The same document can be published to multiple formats (e.g. HTML, PDF, Word). Since the document is based on code, future changes are easy to implement and the document is reproducible by others.	Creating a report is a separate, time-consuming step. Any changes to the report can be time-consuming and prone to error. Since the report is not tied to code, it is not reproducible.

R Notebooks are not designed for all the work you do in R. If you are writing software for a new package or building a Shiny app, you will want to use an R script. However, if you are doing data science you might try R Notebooks. They are great for tasks like exploratory data analysis, model building, and communicating insights. Notebooks are useful for data science because they organize narrative, code, and text around manageable code chunks; and creating quality, reproducible reports is easy.

References: For an introduction to R Notebooks see this video or blog post. For more detailed information, see this workflow presentation or the reference site.

Madrid R User Group, A Brief History

Fri, 10 Mar 2017 00:00:00 +0000

(Editors note: A Spanish verison of the post follows the English text)

In the first meeting we were 5, now we are consistently over 60. It was not difficult for us to start up the group of users of R of Madrid. Gregorio Serrano, Carlos Gil Bellosta, Pedro Concejero and I started our own help list R-help-es to regularly answers questions within this new community. It was in March 2012 when we first got together in a classroom of the Faculty of Economics of one of the Madrid’s public Universities. We discussed: a) How to model different trading strategies with R, b) How to model the number of bugs present in the R code, and c) the presentation of a library to create spatial graphs of the map of Spain.

The group of Users of R of Madrid was born as a group promoted by the Association of R of Spain. The Association allowed us to host the agendas of new calls and the content generated in the meetings in a space of its online portal, which helped to provide initial credibility to the group. From then on, we have held a monthly meeting on a regular basis, except the months of July and August and December - a total of about forty-five meetings in these five years.

In the beginning, we changed our headquarters several times. Then we found a place where we could meet on a regular basis, which was a classroom of the School of Sciences of the UNED. There we spent the first three years, approximately. It was there where the group forged its future in terms of creating a stable core team of members, and established a cadence for the meetings. We covered many of the emerging issues that were being addressed in the Community of R: Social Network Analysis, Spatial representation (maps), and different types of analysis on open sets.

From the first meeting, we agreed to record all the meetings in video and be very consistent in making accessible the presented material (video and presentations).

After this initial period, the leap that allowed us to achieve greater visibility and notoriety was to use the meetup.com portal and get a more central venue in Madrid. This put us in immediate contact with other people having similar interests in holding meetings in a space dedicated to new technologies. MediaLab Prado, promoted by the Madrid City Council, put us in touch with other groups with the same type of interest in everything related to the data: Data Journalists, Analytics, Python Users, etc. Interestingly, joining a Meetup was a must when we introduced ourselves to Revolution Analytics, a company that helped many other R user groups. This sponsorship has helped us keep the group in Meetup and be able to host all the videos on Vimeo.

Now that R has become a language of great success and traction, the group continues to evolve: The Big Data is here, we are sharing the experiences of having participated in Kaggle, Bayesian statistics, etc. This evolution has also occurred in the profile of the attendees: we started to see presentations of companies, and fortunately there are a lot of young people among the attendees and those who present.

Based on experience we’ve accumulated over the years, to those who want to create their own group, we would give these recommendations from Cultivating Communities of Practice:
* Keep the consistency, the monthly periodicity is the most valuable. Attendees appreciate maintaining regularity.
* Using Meetup has been shown to be a key element of success by the visibility it offers and by reaching out to other groups and users with the same interests.
* Another key element to become known and expand the scope of the group has been to participate in meetings of other communities: Data Journalists, Machine Learning, Python, SQLSaturdays, etc.
* Be rigorous in announcing the meetings in advance and sharing the material presented.
* Record the video sessions (we use a tripod and a Smartphone) has also helped to maintain a loyal community.

We have accomplished a lot, but certainly the best is yet to come. Long life to R.

EL GRUPO DE USUARIOS DE R DE MADRID (ESPAÑA)

En la primera reunión éramos 5, ahora somos más de 60 y de forma consistente. No nos costó mucho poner en marcha el grupo de usuarios de R de Madrid. A través de la propia lista de ayuda R-help-es los que solíamos responder de forma más habitual a las dudas de esta nueva comunidad (Gregorio Serrano, Carlos Gil Bellosta, Pedro Concejero y Carlos Ortega), acordamos juntarnos por primera vez. Fue en marzo de 2012, nos juntamos en un aula de la Facultad de Economia de uno de los Campus de Madrid. Hablamos de: a) Cómo modelizar diferentes estrategias de trading con R, b) Cómo modelizar el número de bugs presentes en el código R, y c) la presentación de una librería para crear gráficos espaciales del mapa de España.

El grupo de Usuarios de R de Madrid, nació como grupo promovido por la Asociación de R de España. La Asociación nos permitió alojar las agendas de nuevas convocatorias y el contenido generado en las reuniones en un espacio de su portal online. Este punto ayudó a dotar de un punto inicial de credibilidad sobre el grupo.

A partir de este momento, hemos mantenido una reunión mensual de forma constante, excepto los meses de julio y agosto y diciembre - un total de unas 45 reuniones.

Inicialmente cambiamos de sede varias veces, hasta que encontramos un sitio en donde reunirnos de forma estable en un aula de la Escuela de Ciencias de la UNED, donde estuvimos los tres primeros años aproximadamente. Fue aquí donde el grupo forjó su futuro. Cubrimos muchos de los temas emergentes que en la Comunidad de R se venían tratando: análisis de redes sociales, representación espacial (mapas), y diferentes tipos de análisis sobre conjuntos abiertos.

Desde un primer momento, acordamos grabar todas las reuniones en video y ser muy consistentes en hacer accesible el material presentado (video y presentaciones).

Después de esta primera época, el salto que nos permitió conseguir mayor visibilidad y notoriedad fue el utilizar el portal de Meetup y conseguir una sede más céntrica en Madrid.

Meetup por un lado, nos puso en contacto inmediato con otras personas con el mismo tipo de interés y el celebrar las reuniones en un espacio dedicado a las nuevas tecnologías (MediaLab Prado), promovido por el Ayuntamiento de Madrid nos puso en contacto con otros grupos con el mismo tipo de interés en todo lo relativo a los datos: Periodistas de Datos, Analytics, Usuarios de Python, etc.

Curiosamente, el entrar en Meetup fue también una necesidad cuando nos presentamos para conseguir la ayuda que a los diferentes grupos de usuarios de R recibían de Revolution Analytics. Esta esponsorización nos ha ayudado a mantener el grupo en Meetup y el poder alojar todos los videos en Vimeo.

Ahora que R se ha convertido en un lenguaje de gran éxito y tracción, el grupo sigue evolucionando: El Big Data ya está con nosotros, compartir las experiencias de haber participado en Kaggle, la estadística bayesiana, etc.

Igualmente esta evolución se ha producido en el perfil de los asistentes: comenzamos a ver presentaciones de empresas, y afortunadamente es mucha la gente joven entre los asistentes y los que presentan.

De la experiencia acumulada en estos años, a aquellos que quieran crear su propio grupo, les daríamos estas recomendaciones:

Mantener la consistencia, la periodicidad mensual es la más ajustada. Los asistentes valoran mantener la regularidad.
Utilizar Meetup se ha demostrado como un elemento clave del éxito por la visibilidad que ofrece y por el alcance a otros grupos y usuarios con los mismos intereses.
Otro elemento clave para darse a conocer y ampliar el alcance del grupo ha sido el participar en reuniones de otras comunidades: Periodistas de Datos, Analytics, Usuarios de Python, etc.
Ser rigurosos en el anuncio de las reuniones con cierta antelación y en compartir el material presentado.
El grabar las sesiones en video (usamos para ello un trípode y un Smartphone) también ha ayudado a mantener una comunidad fiel.

Hemos hecho mucho, pero sin duda lo mejor está todavía por llegar.

February '17 Tips and Tricks

Fri, 03 Mar 2017 00:00:00 +0000

If you spend time with an excellent programmer, one thing that immediately jumps out is how quickly she can write code. It often appears to be magic, the number of keystrokes simply can’t equal the number of characters on the screen. The secret: it doesn’t! Most programmers use a series of tricks to save the hassle of writing boilerplate code. This month, my personal goal was to spend less time typing, even if it required slowing down to master these tricks. In this post, I’ll focus on one of the easiest and most flexible tools at your disposal: code snippets. (Astute readers will note that I’ve mentioned code snippets before. They deserve the extra attention!)

Code snippets are a common programming tool similar to keyboard shortcuts. A snippet of text, usually a few characters long, is used as a shortcut to create a larger piece of boilerplate code. Tab navigation enables jumping through the boilerplate code to locations where customization is required.

For example, one of my favorite code snippets is for creating a function. The template of an R function looks like:

functionName <- function(arguments) {
  code 
}

The boilerplate - not worth typing - is:

 <- function( ) {
 
}

In RStudio, the code snippet for functions is fun. Typing fun and pressing Tab will provide an auto-complete suggestion for the snippet. Typing fun and pressing Shift + Tab inserts the snippet directly. As a general typing rule, keeping your hands on the middle row of the keyboard will speed things up. I hit Shift with my right pinky and Tab with my left.

After inserting the boilerplate code, the cursor is placed to left of the <-. You can then quickly type the name of the function. Pressing Tab places the cursor in between the parenthesis. Pressing Tab again places the cursor in between the brackets. This allows you to fill in the remaining portions of the function template quickly.

Type fun and Shift + Tab:

HERE <- function() {

}

Pressing Tab Once

 <- function(HERE) {
 
}

Pressing Tab Twice

 <- function() {
 HERE
 }

Accessing Snippets

RStudio comes with many built-in snippets. To access these snippets, navigate to Tools -> Global Options -> Code and press Edit Snippets.

The menu will show all of the defined snippets. Consider the definition for the function snippet:

snippet fun
	${1:name} <- function(${2:variables}) {
		${0}
	}

The snippet starts by defining the character shortcut fun. The definition includes the boilerplate code. The Tab behavior is defined using ${1} notation. The code snippet help page describes the syntax in more detail.

A Few Examples

`ts`

Adds a time-stamped comment.

# Mon Feb 20 07:34:03 2017 ------------------------------

`if`

Adds the skeleton of an if statement

if () {

}

`mat`

Adds the skeleton for defining a matrix.

matrix( , nrow = , ncol = )

There are code snippets for other languages as well.

Creating Snippets

Code snippets are powerful because they can be customized and created. If you find yourself repeatedly typing the same code, create a snippet. The menu of snippets in RStudio is editable. To create or modify a snippet, add your changes to the menu and click Save.

Once you’ve created a handful of useful code snippets, it would be generous to share. The snippr package makes this process easy.

This gist contains two of my favorites: a code snippet for creating a single file Shiny app, and a code snippet to start a ggplot.

Shiny:

snippet shiny
	library(shiny)
	ui <- fluidPage(
		${1:ui}
	)
	
	server <- function(input, output, session){
	  ${2:server}
	}
	shinyApp(ui = ui, server = server)

ggplot2:

snippet gg
	ggplot(${1:data}, aes(${2:aes})) + 
	  geom_${3:geom}()

What are your favorite snippets?

The R Formula Method: The Bad Parts

Wed, 01 Mar 2017 00:00:00 +0000

R’s model formula infrastructure was discussed in my previous post. Despite the elegance and convenience of the formula method, there are some aspects that are limiting.

Limitations to Extensibility

The model formula interface does have some limitations:

It can be kludgy with many operations on many variables (e.g., log transforming 50 variables via a formula without using paste)
The predvars aspect (discussed in my previous post) limits the utility of the operations. Suppose a formula had: knn_impute(x1) + knn_impute(x2). Do we embed the training set twice in predvars?
Operations are constrained to single columns or features (excluding interaction specifications). For example, you cannot do

lm(y ~ pca(x1, x2, x3), data = dat)
## or nested functions 
lm(y ~ pca(scale(x1), scale(x2), scale(x3)), data = dat)

I’ll use PCA feature extraction a few times here since it is probably familiar to many readers.

Everything Happens at Once

Some of our data operations might be sequential. For example, it is not unreasonable to have predictors that require:

imputation of a missing value
centering and scale
conversion to PCA scores

Given that the formula method operations happen (in effect) at once, this workflow requires some sort of custom solution. While caret::preProcess was designed for this sequence of operations, it does so in a single call, as opposed to a progression of steps exemplified by ggplot2, dplyr, or magrittr.

Allowing a series of steps to be defined in order is more consistent with how data analysis is conducted. However, it does raise the complexity of the underlying implementation. For example, caret::preProcess dictates the possible sequence of tasks to be: filters, single-variable transformations, normalizations, imputation, signal extraction, and spatial sign. This avoids nonsensical sequences that center the data before applying a Box-Cox calculation (which requires positive data).

No Recycling

As a corollary to the point above, there is no way to recycle the terms between models that share the same formula and data/environment. For example, if I fit a CART model to a data set with many predictors, the random forest model (theoretically) shouldn’t need to recreate the same terms information about the design matrix. If the model function has the non-formula interface (e.g., mod_func(x, y)), this can make it easier. However, many do not.

Also, suppose that one of the pre-processing steps is computationally expensive. We’d like to be able to store the state of the results and then add another layer of computations (perhaps as a separate object).

Formulas and Wide Datasets

The terms object saves a matrix with as many rows as formula variables and at least as many columns (depending on interactions, etc). Most of this data is zero and a non–sparse representation is used. The current framework was built in a time where there was more focus on interactions, nesting and other operations on a small scale.

It is unlikely that models would have hundreds of interaction terms, but now it is not uncommon to have hundreds or thousands of main effects. As the number of predictors increases, this takes up an inordinate amount of execution time. For simple randomForest or rpart calls, the formula/terms work can account for most of the execution time. For example, we can calculate how much time functions spend generating the model matrix relative to the total execution time. For rpart and randomForest, we used the default arguments and did the calculations with a simulated data set of 200 data points and varying numbers of predictors:

This is especially problematic for ensemble models. For example, ipred:::ipredbagg creates an ensemble of rpart trees. Since rpart only has a formula method, the footprint of the bagged model object can become very large if X trees are contained in the ensemble. Alternatively, randomForest.formula takes the approach of generating the terms once and feeding the model frame to randomForest.default. This does not work for rpart since there is no non-formula method exposed. Some functions (e.g., lm, survival::coxph) have arguments that can be used to prevent the terms and similar objects from being returned. This saves space but prevents new samples from being predicted. A little more detail can be found here.

One issue is the "factors" attribute of the terms object (discussed in the previous post). This is a non-sparse matrix that has a row for each predictor in the formula and a column for each model term (e.g. main effects, interactions, etc.). The purpose of this object is to know which predictors are involved in which terms.

The issue is that this matrix can get very large and usually has a high proportion of zeros. For example:

mod3 <- lm(Sepal.Width ~ Petal.Width + Petal.Length*Species, data = iris)
fact_mat <- attr(mod3$terms, "factors")
fact_mat

##              Petal.Width Petal.Length Species Petal.Length:Species
## Sepal.Width            0            0       0                    0
## Petal.Width            1            0       0                    0
## Petal.Length           0            1       0                    1
## Species                0            0       1                    1

As the number of predictors increases, the rate of ones is likely to approach a value close to zero very quickly. For example:

Again, it is doubtful that a model with a large number of predictors will have a correspondingly large number of high-level interactions (see the Pareto principle applied to modeling).

Variable Roles

Some packages have implemented extensions of the basic formula. There are cases when formula are needed for specific sub-models. For example, a random coefficient model can be fit with the lmer function. In this case, a model is specified for a particular clustering variable (e.g., a subject in a clinical trial). The code is an example of how lmer syntax works:

# ?lme4::lmer
lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)

Here Subject is important to the model-fitting routine, but not as a predictor. Similarly, the Bradley-Terry model can be used to model competitions and contests. A model on a set of boxers in a series of contests can include terms for their reach:

BTm(outcome = 1, player1 = winner, player2 = loser, 
    formula = ~reach[..] + (1 | ..), 
    data = boxers)

Another extension of basic formulas comes from the modeltools and mboost packages. The function mboost::mob fits a tree-based model with regression models in the terminal nodes. For this model, a separate list of predictors are used as splitting variables (to define the tree structure) and another set of regression variables that are modeled in the terminal nodes. An example of this call is:

# mboost::mob (using the modeltools package for formulas)
mob(diabetes ~ glucose | pregnant + mass +  age,
    data = PimaIndiansDiabetes)

The commonality between these three examples is that there are variables that are critical to the model but do not play the role of standard regression terms. For lmer, Subject is the independent experimental unit. For mob, we have variables to be used for splitting, etc.

There are similar issues on the left-hand side of the formula. When there are multivariate outcomes, different packages have different approaches:

# ?aggregate the mean of two variables by month
aggregate(cbind(Ozone, Temp) ~ Month, data = airquality, mean)

# grouped binomial data:
glm(cbind(events, nonevents) ~ x, family = binomial)

# pls::plsr, 
pls(sensory ~ chemical, data = oliveoil)
# sensory and chemical are 2D arrays in the oliveoil data frame

The overall point here is that, for the most part, the formula method assumes that there is one variable on the left-hand side of the tilde and that the variables on the right-hand side are predictors (exceptions are discussed below). One can envision other roles that columns could play in the analysis of data. Besides the examples given above, variables could be used for

outcomes
predictors
stratification
data for assessing model performance (e.g., loan amount to compute expected loss)
conditioning or faceting variables (e.g., lattice or ggplot2)
random effects or hierarchical model ID variables
case weights
offsets
error terms (limited to Error in the aov function)

The last three items on this list are currently handled in formulas as “specials” or have existing functions. For example, when the model function has a weights argument, the current formula/terms frame work uses a function (model.weights) to extract the weights, and also makes sure that the weights are not included as covariates. The same is true for offsets.

Summary

Some limitations of the current formula interface can be mitigated by writing your own or utilizing the Formula package.

However, there are a number of conceptual aspects (e.g., roles, sequential processing) that would require a completely different approach to defining a design matrix, and this will be the focus of an upcoming tidyverse package.

More January Package Picks

Fri, 24 Feb 2017 00:00:00 +0000

In a recent post, I highlighted several new packages that arrived on CRAN in January that provided R users with access to data. In this post, I present additional selections for interesting January packages, organized into the categories Miscellaneous, Machine Learning, Statistics and Utilities.

Miscellaneous

rcss v1.2: Provides functions for Solving Control Problems with Linear State Dynamics.
stormwindmodel v0.1.0: Provides functions to calculate wind speeds for hurricanes and tropical storms in the North American Atlantic basin. One vignette describes the package and another shows how to use it.

Machine Learning

crisp v1.0.0: Implements the convex regression with interpretable partitions (CRISP) method of predicting an outcome variable on the basis of two covariates.
BayesS5 v1.22: Implements Bayesian Variable Selection Using Shotgun Stochastic Search with Screening (S5) useful in settings where p >> n. For details, see the paper
classifierplots v1.3.2: Provides functions to generate a grid of binary classifier and diagnostic plots with a single function call. See the README for details.
eclust v0.1.0: Provides an algorithm for clustering high-dimensional data that can be affected by an environmental factor. See the paper for details.
EnsCat v1.1: Implements various clustering methods for categorical data. See the website for examples and the paper for the details.
MAVE v0.1.7: Implements the MAVE (Minimum Average Variance Estimation) method of dimension reduction. Look here for the math and here for examples.
mfe v0.1.0: Provides functions to extract meta-features from datasets to support the design of recommendation systems. The vignette provides examples.
rsparkling v0.1.0: extends sparklyr with an interface to the H2O Sparkling Water machine learning library. The README explains how to use the package.

Statistics

confSAM v0.1: Contains a function that computes estimates and confidence bounds for the false discovery proportion in a multiple testing environment. The vignette describes the theory and provides examples.
pdSpecEst v1.0.0: Implements a non-parametric, geometric wavelet method to estimate autocovariance matrix of a time series that preserves positive-definiteness of the estimator. This preserves the intrepretability of the estimate as a covariance matrix and helps with computational issues. The paper describes the theory and the vignette provides an example.
tsdecomp v0.2: Implements ARIMA model-based decompositions for quarterly and monthly time series data. The vignette describes the math.
TSeriesMMA v0.1.1: Provides a function to calculate the Hurst surface for a time series. Multiscale, multifractical analysis (MMA) is described in a paper by Gieraltowski et al.

Utilities

awsjavasdk v0.2.0: Provides a boilerplate of classes used to access the Amazon Web Services Java Software Development Kit via package rJava. The vignette shows how to use the package.
colr v0.1.900: Provides functions that use Perl regular expressions to select and rename columns in dataframes, lists and numeric types. The vignette contains examples.
flifo v0.1.4: Provides functions to create and manipulate FIFO (First In First Out), LIFO (Last In First Out), and NINO (Not In or Never Out) stacks in R. See the vignette for examples.
fst v0.7.2: Provides functions to read and write data frames at high speed, and compress data with type-optimized algorithms that allow random access of stored data frames.
manipulateWidget v0.5.1: Provides helper functions to add controls like sliders, pickers, checkboxes, etc. to interactive charts created with package htmlwidgets. The animated vignette will get you started.
msgtools v0.2.4: Provides utilities for error, warning, and other messages in R packages, including consistency checks across messages, spell-checking, and message translations for localization. See the vignette for examples.
padr v0.2.0: Provides functions to transform datetime data into a format ready for analysis, including aggregating data to a higher level interval (thicken) and imputing records where observations were absent (pad). There is an Introduction.
pbdPRC v0.1-1: Implements light, yet secure remote procedure calls with a unified interface via ssh (OpenSSH) or plink/plink.exe (PuTTY). The vignette provides examples.
reprex v0.1.1: Provides a way to send code snippets with rendered output to sites like stackoverflow and github. The README shows examples.
restfulr v0.0.8: Models a RESTful service as if it were a list.
sys v1.1: A replacement for base system2 with consistent behavior across platforms. Supports interruption, background tasks, and full control over STDOUT / STDERR binary or text streams. README provides some details.
textclean v0.3.0: Provides tools to clean and process text, such as replacing or removing substrings that are not optimal for analysis. The README shows how to use them.
tidyxl v0.2.1: Imports non-tabular data from Excel into R. The vignette shows how.
unpivotr v0.1.0: Provides tools for converting data from complex or irregular layouts into a columnar structure. There is one vignette showing how to unpivot pivot tables from a spreadsheet, and another that shows how to work with multiple, similar tables.
WVPlots v0.2.2: Provides examples of ggplot plots that can be generated from a standard calling interface. Here is the explanation of the concept, and here are some nice examples.

How to Teach R: Common mistakes

Wed, 22 Feb 2017 00:00:00 +0000

Would you like to teach people to use R? If so, I would like to jump-start your efforts.

I’m one half of RStudio’s education team, and I’ve taught thousands of people to use R, usually in face-to-face workshops. Over time, I’ve come to appreciate that teaching R in a short workshop is an unusual challenge that requires an unusual approach: you cannot teach a short workshop in the same way that you would teach a college course, and you should not teach R in the same way you would teach Python, UNIX or C.

In the next few blog posts, I’ll share the pedagogy that I’ve adopted for teaching R workshops. These ideas have made my life easier and my students happier (based on student feedback). I think they can do the same for you.

We’ll begin in this post by identifying common mistakes that ensnare new R teachers. Each of these mistakes seems like a good idea at first glance, but leads to an unsuccessful short workshop, and I’ll tell you why. To make things simple, I’ve recast each mistake as a principle to follow. Let’s examine them one by one:

DO NOT teach R as if it were a programming language. Why not? Because R is a programming language for doing data science. You can be confident that your students want to use R to make graphs, fit models, and impress their colleagues. Show them how to do these empowering things and then teach programming later, as a way to do these things even better. To be honest, if your students only wanted to learn how to program, they would be studying another language.
DO NOT assume that methods that work well in a college classroom will work well in a one-, two-, or half-day workshop. Active learning, peer-led instruction, group projects, flipped classrooms, and other techniques have improved college so much that I wish I could go back and do college over again. But these techniques take more time to convey information than you have in a workshop. They also work best with motivated students who are accustomed to learning. Do you have those? If your workshops are like mine, you have busy individuals who have set aside precious time and money to take your workshop. To be frank, they want to acquire more information than you can provide in a day of active learning or peer-led instruction. I’m not saying that you shouldn’t use these techniques (please try!), but expect to modify them heavily.
DO NOT avoid the lecture. Some teaching gurus will do somersaults to avoid lectures because lectures are too passive for students and too easy to do poorly. Some extremists even extend this notion to slides, claiming that good teachers should not use slides. If you adopt this mindset, you will fail at the one thing that your students expect you to do well: to convey large amounts of information in a short period of time. Not only should you embrace lecturing, i.e., presenting information, you should become an expert at it. Learn to present effectively, and learn to intermix presentations with activities that keep your workshop engaging.
DO NOT assume that you can teach someone else’s workshop out of the box, even if it is your own. A workshop is not like a video that you make once and then replay when needed. A workshop is more like a play that must be cast, costumed, and rehearsed each time you present it in a new venue. If you think you can reproduce a workshop quickly because “it already exists,” you are setting yourself up for failure. If you let your manager think this, you are setting yourself up for stress!
DO NOT let your workshop become a consulting clinic for installation bugs. Workshops make first impressions just like people do. You want to use the first minutes of your workshop to set an energetic tone, to engage your students, and to inspire them — not to hop from student to student debugging installation problems. Do what it takes to avoid this situation. My favorite solution is to provide a classroom RStudio Server for students to use.

But what if you feel that students deserve to leave your workshop with the software successfully installed on their computer? Then you are in good company! My mentor, Hadley Wickham, argues for this persuasively and enthusiastically. But make it happen in a way that does not torpedo your workshop. Hold a real clinic. Pass out instructions in advance and demand that any problems be reported ahead of time. Make successful installation a prerequisite for registering. Be sure that your students know that if they do not have permission to install necessary software on their work laptop, they should bring a different laptop. Be creative and cover the bases.

Whatever you do, remember that the hour immediately before class is less than ideal for installing software. You have other tasks to attend to, and inevitably some students will come late and bring bugs.

I’ll have more to say about each of these topics in the posts that follow. In those posts, I’ll try to layout a fun, inspiring vision for how an R workshop works; no more “thou shall nots.” See you there!

January New Data Packages

Fri, 17 Feb 2017 00:00:00 +0000

As forecast, the number of R packages hosted on CRAN exceed 10,000 in January. Dirk Eddelbuettel, who has been keeping track of what’s happeining on CRAN with his CRANberies site for a number of years, called [hurricaneexposure]() as the 10K package in a tweet on January 27th. hurricaneexposure was of two hundred and six new packages that arrived on CRAN in January. I thought many of them were of high quality so picking out recommendations was unusually difficult this month. Here are my picks for the top fifty organized into xxx categories.

Data

elevatr v0.1.1: Provides access to several databases that provide elevation data including Mapzen Elevation Service, Mapzen Terrain Service, Amazon Terrain Tiles, and the USGS Elevation Point Query Service. There is a vignette.
epidata v0.1.0: Provides tools to retrive data from the Economic Policy Institute README shows how to use the package.
europop v0.3: Contains a dataset giving the populations of all European cities with at least 10,000 inhabitants during the period 1500-1800.
fivethirtyeight v0.1.0: Provides the data, code and interactive visualizations behind FiveThirtyEight Stories. There is a vignette that privides an example of a data analysis, and a list of data sets that are included.
getCRUCLdata v.1.1: Provides functions that automate downloading and importing climatology data from University of East Anglia Climate Research Unit (CRU). There is vignette to get you started.
hurricaneexposure v0.0.1: Allows users to create time series of tropical storm exposure histories for chosen counties for a number of hazard metrics (wind, rain, distance from the storm, etc.). The vignette provides an overview.
metScanR v0.0.1: Provides functions for mapping, and gathering meteorological data from various US surface networks: COOP, USCRN, USRCRN, AL-USRCRN, ASOS, AWOS, SNOTEL, SNOTELLITE, SCAN, SNOW, and NEON
mglR v0.1.0: Provides tools to download and organize large-scale, publicly available genomic studies on a candidate gene scale. The vignette shows how to use the package.
nzpullover v0.0.2: Contains datasets of driving offences and fines in New Zealand between 2009 and 2016, originally published by the New Zealand Police.
owmr v0.7.2: Provides a wrapper for the OpenWeatherMap API
PeriodicTable v0.1.1: Contains a dataset of the properties of chemical elements.
pwt9 v9.0-0: Contains the Penn World Table 9 which provides information on relative levels of income, output, inputs, and productivity for 182 countries between 1950 and 2014.
rdwd v0.7.0: Provides functions to obtain climate data from the German Weather Service, Deutscher Wetterdienst, (DWD) There is a vignette on Weather Stations and another showing how to use the package.
rwars v1.0.0: Provides functions to retrieve and reformat data from the ‘Star Wars’ API SWAPI. The vignette shows how to use the package.
wikidataQueryServiceR v0.1.0: Provides an API Client Library for Wikidata Query Service which provides a way for tools to query Wikidata via SPARQL. See README for how to use it.
wikilake v0.1: Provides functions to scrape metadata about lakes from Wikipedia. The vignette fetches data from Michigan lakes.
worrms v0.1.0: Provides a client for the World Register of Marine Species. The vignette shows how to use the package.

Three Tips for Training Excel Users in R

Fri, 10 Feb 2017 00:00:00 +0000

“I’m not a coder” or “I was never good at math” is a frequent refrain I hear when I ask professionals about their data analysis skills. Through popular culture and stereotypes, most people who don’t have a background in programming automatically underestimate their ability to create amazing things with code. However, Data Society has proven that this is a false narrative through our training program – with students in over 20 countries and many government and enterprise clients, we’ve seen so-called “non-coders” proficiently put together automated data cleaning code scripts and analyses within a few weeks. So how do we do it? Well, we’ve singled out three key steps to get someone started on their journey to an amazing skill set and more powerful data analytics:

Start simple. My first time training up a group of instructors to teach R programming was a great experience (especially given that these R trainings will train up under-served adults), but it wasn’t without its challenges. Anyone with a non-programming background can feel intimidated when they get started, and it’s critical to start off by using R as a glorified calculator. Start with basic operations to get your students comfortable with the interface of RStudio (an awesome and intuitive interface for R), then you can move on to more advanced functions.

See? This doesn’t look so complicated.

Show the similarities. Excel and R do have many similarities in syntax that makes it easier for Excel users to transfer their skills over to R. For example, RStudio has a tabular data view that may look very familiar to spreadsheet users:

This looks like another program I’ve used before…

And this is not just limited to viewing data. There is a lot of syntax from Excel that is easily transferable to R. For example, using if-else statements in Excel looks like this:

And here is what it looks like in R:

The learning curve for R is a fast one, especially for Excel users. Highlighting those similarities puts new users at ease and gives them a way to connect R functionality to the functionality they already use in Excel.

Bring out the wow factor. One of the main advantages of R is its amazing visualizations and advanced analysis that are not even possible in many more basic tools. Our “wow factor” is the collaboration project we did with the White House initiative, The Opportunity Project, which aimed to connect federal agencies with tech companies to build out tools with their open data. In a couple of short weeks, our team combined several disparate open data sources and built an interactive application entirely in R to provide schools and superintendents with a community resource mapper.

Showing students how to eliminate duplicates from data sets with 1 – 2 lines of code or quickly manipulating data into a different format is a wow factor in the amount of time it can save regular Excel users. The applications are immediately apparent for those who have struggled to go through thousands of rows manually or upload a data set with millions of records.

R is gaining in popularity, with millions of users worldwide and growing. Not only that, but we’ve seen an increase in demand for data analysis skills across all job sectors. Adding R programming and data analysis to your resume can add $10,000 - $15,000 to your salary. With that type of incentive, both in pay and in time saved, there’s no better time to take the “not” out of “I’m not a coder”.

The Data Society is a data science training platform for professionals. Among other government and corporate clients, Data Society has trained staff at the Department of Commerce and the U.S. Army through their enterprise firm, Data Society Solutions, which provides customized corporate data science training and consulting services. If you’d like to learn more, please email solutions@datasociety.co.

Reproducible Finance with R: ETF Country Exposure

Wed, 08 Feb 2017 00:00:00 +0000

Today, we are going to tackle a project that has long been on my wishlist - a Shiny app to take a fund or portfolio, analyze its exposure to different countries and display those exposures on a world map. Now you know how exciting my wishlists are.

Before describing our data importing/wrangling work here in the Notebook, it mgight be helpful to look at where we’re headed. The final Shiny app is here. This is similar to a previous project because we are building a leaflet map, shading according to data that we add to the spatial dataframe, and including another HTML widget that is reponsive to the map. However, our current project differs in important ways and has a completely different use.

The previous project allowed user to click a country on the map and view the time series of returns. Our current project will allow the user to choose an ETF and view how that ETF is invested in different countries by how a world map is shaded.

From a substantive perspective, this app helps visualize country risks instead of returns - indeed, it’s the first in our series that does not import stock returns in any way. From an R perspective, in our current project the map is the responsive object according to user inputs, whereas before the dygraph was the responsive object according to user clicks on a map. They are related and require spatial dataframes but very different.

If you looked closely at the Shiny app, you noticed that we do have a data object that os responsive to a map click: we display a datatable of companies held by the ETF in whatever country is clicked. That is, if a user chooses an ETF and sees by the shading that the ETF is allocated X% to China, the user can click on the map to see which companis the ETF owns in China. That functionality is similar to the dygraphs functionality, except of course we have to wire up a datatable and do some filtering by country instead of passing an xts object to dygraphs. The fulcrum will still be the clicked map shape.

Alright, that app is what we’re ultimately building but, by way of what we’ll do in this Notebook, here’s the roadmap.

First, we are going to grab the data for one fund, the MSCI Emerging Markets ETF. Note that we are not going to get return data over time. Instead, we just want a snapshot of the ETF holdings: its constituents, their weights and their home countries. Our eventual app will include several ETFs but we are going to work with one ETF in this Notebook, with the foreknowledge that we want to reuse our steps when it’s time to build the Shiny app. In short, let’s get it right for this Emerging Markets ETF and then we can iterate over other ETFs when we move to building our Shiny app.

After we download the snapshot of the emerging markets fund, we will do some wrangling, some country weight aggregation, and then merge that data to our spatial dataframe. Adding that data will depend on the ETF using the same country naming convention as our spatial dataframe, so we’ll pay attention to that in the wrangling process.

Once we add the data to our spatial dataframe, we will recycle some old code, build a leaflet map and shade it according to the ETF’s country exposure. This is just a test to see how things will look in the Shiny app and we can even play around with different color palettes to get things just right.

Once we have the map aesthetics sorted, we’ll turn to part two: displaying the details of each country holding. Really this is just filtering our dataframe by country name - whatever country the user clicks - but we’ll go ahead and make sure things look how we want in this Notebook and then pass that object to our app eventually.

Let’s get to it!

First, let’s grab the fund data from MSCI’s homepage. We will use the read_csv() function from the readr package. We will title it emerging_markets_fund since we’ll be pulling in other funds later.

Note that we have to skip the first 10 rows, which is why the ‘skip = 10’ argument is included. That’s because this csv file is loaded with oddly formatted data in the first 10 rows. If we don’t skip those 11 rows, this import will be totally unhelpful. The ‘import dataset’ button in the IDE saved me minutes/hours of frustration here! (Author’s note: when this post was originally published, this argument was “skip = 11”. A few weeks later, ishares changed the structure of their spreadsheet. Now it’s 10 rows that need to be skipped!)

# Download the Emerging Markets ETF fund data from the ishares homepage.

emerging_market_fund <- read_csv("https://www.ishares.com/us/products/239637/ishares-msci-emerging-markets-etf/1467271812596.ajax?fileType=csv&fileName=EEM_holdings&dataType=fund", 
    skip = 10)

# Take a look at the fund snapshot we just imported. 
head(emerging_market_fund)

## # A tibble: 6 × 16
##   Ticker                               Name `Asset Class` `Weight (%)`
##    <chr>                              <chr>         <chr>        <dbl>
## 1 005930            SAMSUNG ELECTRONICS LTD        Equity       4.1165
## 2    700               TENCENT HOLDINGS LTD        Equity       3.6649
## 3   2330 TAIWAN SEMICONDUCTOR MANUFACTURING        Equity       3.5112
## 4   BABA ALIBABA GROUP HOLDING ADR REPRESEN        Equity       2.7304
## 5    NPN              NASPERS LIMITED N LTD        Equity       1.7600
## 6    939     CHINA CONSTRUCTION BANK CORP H        Equity       1.5572
## # ... with 12 more variables: Price <dbl>, Shares <dbl>, `Market
## #   Value` <chr>, `Notional Value` <chr>, Sector <chr>, SEDOL <chr>,
## #   ISIN <chr>, Exchange <chr>, Country <chr>, Currency <chr>, `Market
## #   Currency` <chr>, `FX Rate` <dbl>

Alright, we have our fund data and now the wrangling begins. We are actually going to use this initial object to create two other objects: one will be merged with the spatial dataframe and one will be a standalone object to be loaded in our Shiny app.

emerging_market_fund_country_weights <- emerging_market_fund %>% 
  select(Country, `Weight (%)`) %>%
  mutate(Country = replace(Country, Country == "Russian Federation", "Russia"), 
         Country = replace(Country, Country == "Korea (South)", "Korea"),
         Country = replace(Country, Country == "KO", "Korea"),
         Country = replace(Country, Country == "Czech Republic", "Czech Rep.")
         ) %>%
  group_by(Country) %>% 
  summarise(EEM = sum(`Weight (%)`, na.rm = TRUE)) %>% 
  rename(name = Country) %>% 
  filter(EEM > 0) %>% 
  filter(name != "-") %>% 
  arrange(desc(EEM))

# Let's just take a look at the data.

emerging_market_fund_country_weights

## # A tibble: 28 × 2
##            name     EEM
##           <chr>   <dbl>
## 1         China 26.2637
## 2         Korea 14.9220
## 3        Taiwan 12.1916
## 4         India  8.5008
## 5        Brazil  7.4449
## 6  South Africa  6.7359
## 7        Russia  3.7714
## 8        Mexico  3.6169
## 9     Indonesia  2.4890
## 10     Malaysia  2.4130
## # ... with 18 more rows

Those country weights are pretty striking. China + Korea + Taiwan equal 51% of this fund - quite concentrated in economies that are probably closely linked. Perhaps that’s by design? Perhaps the inter-economy correlation isn’t as high as I believe? A cross-border investment or trade Shiny app would be helpful here.

It’s worth a second to consider the definition of ‘emerging market’, a term that has become quite ubiquitous and has a know-it-when-we-see-it feel (if you’re not into political economy, feel free to skip this paragraph). The phrase was coined in 1981 by the World Bank’s Antoine Van Agtmael to help encourage investment in developing nations as he felt that ‘Third World’ country was both distasteful and stifling to investors. Learn more here. Today, the phrase connotes an economy that is growing and transitioning from developed to developing, though some commentators include a political transition as well. Since we are working with an MSCI fund, we should consider their definition. It wasn’t easy to track down, but according to the Financial Times, MSCI takes into account number of listed companies of a certain size (an economic measure) and openness to foreign capital (a political measure).

Back to our task at hand: we have downloaded the fund data and got it into shape to be added to our shapefile. That process is the exact same as in our previous post so before we do that, let’s use the original fund data to create one other object, to store country level detail on companies, weights and sectors. If that seems a bit confusing, head back to the Shiny app and click on a country. The datatable displays company names and details, and we need to create a dataframe to extract and hold that data.

# Wrangle for the datatable to show company names, sector, weights.

EEM <- emerging_market_fund %>% 
  select(Name, Country, Sector, `Weight (%)`, `Market Value`) %>% 
  mutate(Country = replace(Country, Country == "Russian Federation", "Russia"), 
         Country = replace(Country, Country == "Korea (South)", "Korea"),
         Country = replace(Country, Country == "Czech Republic", "Czech Rep.")
         ) %>% 
  filter(`Weight (%)` > 0) %>% 
  filter(Country != "-") 

# Let's test it on Brazil to make sure it works.

Brazil_companies <- EEM %>%
  filter(Country == "Brazil") %>% 
  ungroup() %>% 
  select(-Country)

Brazil_companies

## # A tibble: 59 × 4
##                                  Name           Sector `Weight (%)`
##                                 <chr>            <chr>        <dbl>
## 1       ITAU UNIBANCO HOLDING PREF SA       Financials       0.8775
## 2              BANCO BRADESCO PREF SA       Financials       0.6219
## 3                            AMBEV SA Consumer Staples       0.5881
## 4                        VALE PREF SA        Materials       0.3817
## 5         PETROLEO BRASILEIRO PREF SA           Energy       0.3808
## 6                           PETROBRAS           Energy       0.3028
## 7             CIA VALE DO RIO DOCE SH        Materials       0.2701
## 8   ITAUSA INVESTIMENTOS ITAU PREF SA       Financials       0.2668
## 9  BMF BOVESPA BOLSA DE VALORES MERCA       Financials       0.2312
## 10                BANCO DO BRASIL S/A       Financials       0.2039
## # ... with 49 more rows, and 1 more variables: `Market Value` <chr>

This is what a user of our Shiny app will see upon clicking on Brazil, it is the country level detail of how the fund is invested in Brazil. We will save that ‘EEM’ object in the .RDat file so it can be loaded into our Shiny app.

Okay, let’s go ahead and build that map of the world and add our fund country weights to it. This process is identical to how we did it here, but we’ll go through the steps again.

First, let’s download the spatial dataframe. We will also use the ms_simplify() function from rmapshaper to reduce the size of the dataframe. This function will reduce the number of longitude and latitude coordinates used to build each country. It will make loading faster in our Shiny app but won’t affect any of our logic.

library(rgdal)
library(rmapshaper)
library(httr)

# Load the spatial dataframe
tmp_zipped <- tempfile()
tmp_unzipped <- tempfile()

httr::GET("http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_0_countries.zip", write_disk(tmp_zipped))

## Response [http://naciscdn.org/naturalearth/50m/cultural/ne_50m_admin_0_countries.zip]
##   Date: 2017-03-24 17:31
##   Status: 200
##   Content-Type: application/x-zip-compressed
##   Size: 818 kB
## <ON DISK>  /var/folders/np/13n2s4g91vsgv7y432b5wf_00000gn/T//RtmplAXIku/file106cc57ce68bd

unzip(tmp_zipped, exdir = tmp_unzipped)

world <- readOGR(tmp_unzipped, 'ne_50m_admin_0_countries', verbose = FALSE)

# Use ms_simplify to reduce the size of the spatial dataframe.
world <- ms_simplify(world)

Now we will use the merge() function from the sp package to add our country weight data. Remember above where we made sure to use a consistent country naming convention when wrangling the ETF data? This is where it will come in handy - we use the ‘name’ column to perform the merge. After the merging, ETF exposures will be added for each country that has a match in the ‘name’ column. For those with no match, the EEM column will be filled with NA.

library(sp)
library(leaflet)

world_fund_country_weights <- merge(world, emerging_market_fund_country_weights, by = "name") 

# Let's take a look and make sure that merge accomplished what we want. There will be
# several NA values, because this fund is allocated only to emerging markets. The developed markets
# should be mostly NAs - Australia is immediately visible to us as an NA.
world_fund_country_weights@data[c('name', 'EEM')]

##                     name     EEM
## 1            Afghanistan      NA
## 4                 Angola      NA
## 2                Albania      NA
## 162 United Arab Emirates  0.7672
## 6              Argentina      NA
## 7                Armenia      NA
## 5             Antarctica      NA
## 8              Australia      NA
## 9                Austria      NA
## 10            Azerbaijan      NA
## 24               Burundi      NA
## 13               Belgium      NA
## 15                 Benin      NA
## 23          Burkina Faso      NA
## 11            Bangladesh      NA
## 22              Bulgaria      NA
## 18      Bosnia and Herz.      NA
## 12               Belarus      NA
## 14                Belize      NA
## 17               Bolivia      NA
## 20                Brazil  7.4449
## 21                Brunei      NA
## 16                Bhutan      NA
## 19              Botswana      NA
## 28  Central African Rep.      NA
## 27                Canada      NA
## 149          Switzerland  0.0814
## 30                 Chile  1.2210
## 31                 China 26.2637
## 35         Côte d'Ivoire      NA
## 26              Cameroon      NA
## 40       Dem. Rep. Congo      NA
## 33                 Congo      NA
## 32              Colombia  0.4224
## 34            Costa Rica      NA
## 37                  Cuba      NA
## 107            N. Cyprus      NA
## 38                Cyprus      NA
## 39            Czech Rep.  0.1802
## 58               Germany      NA
## 43              Djibouti      NA
## 42               Denmark      NA
## 44        Dominican Rep.      NA
## 3                Algeria      NA
## 45               Ecuador      NA
## 46                 Egypt  0.1191
## 49               Eritrea      NA
## 143                Spain      NA
## 50               Estonia      NA
## 51              Ethiopia      NA
## 54               Finland      NA
## 53                  Fiji      NA
## 52          Falkland Is.      NA
## 55                France      NA
## 56                 Gabon      NA
## 163       United Kingdom      NA
## 57               Georgia      NA
## 59                 Ghana      NA
## 63                Guinea      NA
## 64         Guinea-Bissau      NA
## 48            Eq. Guinea      NA
## 60                Greece  0.3025
## 61             Greenland      NA
## 62             Guatemala      NA
## 65                Guyana      NA
## 67              Honduras      NA
## 36               Croatia      NA
## 66                 Haiti      NA
## 68               Hungary  0.2818
## 71             Indonesia  2.4890
## 70                 India  8.5008
## 74               Ireland      NA
## 72                  Iran      NA
## 73                  Iraq      NA
## 69               Iceland      NA
## 75                Israel      NA
## 76                 Italy      NA
## 77               Jamaica      NA
## 79                Jordan      NA
## 78                 Japan      NA
## 136      Siachen Glacier      NA
## 80            Kazakhstan      NA
## 81                 Kenya      NA
## 85            Kyrgyzstan      NA
## 25              Cambodia      NA
## 82                 Korea 14.9220
## 83                Kosovo      NA
## 84                Kuwait      NA
## 86               Lao PDR      NA
## 88               Lebanon      NA
## 90               Liberia      NA
## 91                 Libya      NA
## 144            Sri Lanka      NA
## 89               Lesotho      NA
## 92             Lithuania      NA
## 93            Luxembourg      NA
## 87                Latvia      NA
## 104              Morocco      NA
## 101              Moldova      NA
## 95            Madagascar      NA
## 100               Mexico  3.6169
## 94             Macedonia      NA
## 98                  Mali      NA
## 106              Myanmar      NA
## 103           Montenegro      NA
## 102             Mongolia      NA
## 105           Mozambique      NA
## 99            Mauritania      NA
## 96                Malawi      NA
## 97              Malaysia  2.4130
## 108              Namibia      NA
## 111        New Caledonia      NA
## 114                Niger      NA
## 115              Nigeria      NA
## 113            Nicaragua      NA
## 110          Netherlands  0.3367
## 116               Norway      NA
## 109                Nepal      NA
## 112          New Zealand      NA
## 117                 Oman      NA
## 118             Pakistan      NA
## 120               Panama      NA
## 123                 Peru  0.3722
## 124          Philippines  1.1454
## 121     Papua New Guinea      NA
## 125               Poland  1.2310
## 127          Puerto Rico      NA
## 41       Dem. Rep. Korea      NA
## 126             Portugal      NA
## 122             Paraguay      NA
## 119            Palestine      NA
## 128                Qatar  0.7938
## 129              Romania      NA
## 130               Russia  3.7714
## 131               Rwanda      NA
## 169            W. Sahara      NA
## 133         Saudi Arabia      NA
## 145                Sudan      NA
## 132             S. Sudan      NA
## 134              Senegal      NA
## 137         Sierra Leone      NA
## 47           El Salvador      NA
## 141           Somaliland      NA
## 140              Somalia      NA
## 135               Serbia      NA
## 146             Suriname      NA
## 138             Slovakia      NA
## 139             Slovenia      NA
## 148               Sweden      NA
## 147            Swaziland      NA
## 150                Syria      NA
## 29                  Chad      NA
## 156                 Togo      NA
## 154             Thailand  2.2042
## 152           Tajikistan      NA
## 159         Turkmenistan      NA
## 155          Timor-Leste      NA
## 157              Tunisia      NA
## 158               Turkey  1.0363
## 151               Taiwan 12.1916
## 153             Tanzania      NA
## 160               Uganda      NA
## 161              Ukraine      NA
## 165              Uruguay      NA
## 164        United States  0.4507
## 166           Uzbekistan      NA
## 167            Venezuela      NA
## 168              Vietnam      NA
## 170                Yemen      NA
## 142         South Africa  6.7359
## 171               Zambia      NA
## 172             Zimbabwe      NA

We have our data added to the shapefile. Let’s go ahead and construct a map. First we’ll build a popup to show some detail, then we will create a green palette and a purple palette - for no other reason than to see which is more visually appealing.

# Create a popup to display the exact country weight.
EEMPopup <- paste0("<strong>Country: </strong>", 
                world_fund_country_weights$name,
                "<br><strong> Country Weight: </strong>", 
                world_fund_country_weights$EEM, "%")

# Let's create two palettes so we can look at two variants. This is pure aesthetics, but the Notebook is the
# place for aesthetics.

EEMPalGreens <- colorQuantile("Greens", world_fund_country_weights$EEM, n = 20)
EEMPalPurples <- colorQuantile("Purples", world_fund_country_weights$EEM, n = 20)

Let’s invoke leaflet! As before, we will use ‘layerId = ~name’. This is, again, massively important because when we create a Shiny app, we want to pass country names to our datatable and filter accoringly. The ‘layerId’ is how we’ll do that: when a user clicks on a country, we capture the ‘layerId’, which is a country name that can be used for filtering.

# Build a green shaded map.
leaf_world_emerging_greens <- leaflet(world_fund_country_weights) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7,
                  
      # The next line of code is really important for creating the map we want to use later.
      
      color =~EEMPalGreens(EEM), layerId = ~name, popup = EEMPopup)

# Build a purple shaded map.
leaf_world_emerging_purples <- leaflet(world_fund_country_weights) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7,
      color =~EEMPalPurples(EEM), layerId = ~name, popup = EEMPopup)

# Let's compare our green shaded map to our purple shaded map.
leaf_world_emerging_greens

leaf_world_emerging_purples

Both those maps look good to me but purple might be the way to go ultimately. That’s a decision for next time - see you then!

Reproducible Finance with R: Sector Correlations Shiny App

Fri, 03 Feb 2017 00:00:00 +0000

In a previous post, we built an R Notebook that pulled in data on sector ETFs and allowed us to calculate the rolling correlation between a sector ETF and the S&P 500 ETF, whose ticker is SPY. Today, we’ll wrap that into a Shiny app that allows the user to choose a sector, a returns time period such as ‘daily’ or ‘weekly’, and a rolling window. For example, if a user wants to explore the 60-day rolling correlation between the S&P 500 and an energy ETF, our app will show that. As is customary, we will use the flexdashboard format and reuse as much as possible from our Notebook.

The final app is here, with the code available in the upper right-hand corner. Let’s step through this script.

The first code chunk is where we do the heavy lifting in this app. We will build a function that takes as parameters an ETF ticker, a returns period, and a window of time, and then calculates the desired rolling correlation between that ETF ticker and SPY.

library(flexdashboard)
library(quantmod)
library(dygraphs)
library(dplyr)

# A function to build an xts object to hold both sector and index returns.
sector_correlations <- function(sector, period = "weekly", window = 10) {

# Make a data frame of the sector and S&P 500 ETF.
etf_ticker_sector <- data_frame(c(sector, "SPY"))

colnames(etf_ticker_sector) <- "tickers"

# Use getSymbols and Ad to get adjusted prices.

symbols <- getSymbols(etf_ticker_sector$tickers, auto.assign = TRUE, warnings = FALSE)

etf_prices <- do.call(merge, lapply(symbols, function(x) Ad(get(x))))

# We want log returns by the period chosen by the user. 

etf_returns <- do.call(merge, lapply(etf_prices, 
                                     function(x) periodReturn(x, period = period, type = 'log')))

# Create one xts object we can pass to rollapply.
merged_xts <- merge(etf_returns[, 1], etf_returns[, 2])

merged_xts$rolling_cor <- rollapply(merged_xts, window, 
                                         function(x) cor(x[, 1], x[, 2], use = "pairwise.complete.obs"), 
                                         by.column = FALSE)
# We care about the name because it will be displayed in the dygraph when a user hovers.

names(merged_xts) <- c(paste(sector, "Returns", sep = ""), "SPY Returns", paste(sector, "/SPY Correlation", sep = ""))

    
assign("sector_correlations", merged_xts, .GlobalEnv)
}

That function uses getSymbols() to pull in prices and periodReturns() to convert to log returns, either daily, weekly or monthly. Then we merge into one xts object and calculate rolling correlations, depending on the window parameter. It should look familiar from the Notebook, but honestly, the transition from the previous Notebook to this code chunk wasn’t as smooth as would be ideal. I broke this into two functions in the Notebook, but thought it flowed more smoothly as one function in the app since I don’t need the intermediate results stored in a persistent way. Combining the two functions wasn’t difficult, but it did break the reproducible chain in a way that I don’t love. In the real world, I would (and, in my IDE, I did) refactor the Notebook to line up with the app better. Enough self-shaming, back to it.

Next, we need to create a sidebar where our users can select a sector, a returns period and a rolling window. Nothing fancy here, but one thing to note is how we use selectInput to translate from the sector to the ETF ticker symbol. This means our users don’t have to remember those three-letter codes; they just choose the name of the desired sector from a drop-down menu.

helpText("Choose a sector")

# Don't make the user memorize the ETF ticker symbols! 
# Let them choose the intuitive sector name and then translate to the 
# ticker symbol in the background.

fluidRow(
  column(9,
  selectInput("sector", "Sector ETF",
                c(
                  "Energy" = "XLE",
                  "Financials" = "XLF",
                  "Health Care" = "XLV",
                  "Industrials" = "XLI", 
                  "Materials" = "XLB", 
                  "Technology" = "XLK", 
                  "Utilities" = "XLU",
                  "Cons Discretionary" = "XLY", 
                  "Cons Staples" = "XLP")))
)  

helpText("Choose a returns time period and a rolling window")

fluidRow(
  column(7,
  selectInput("period", "Time Period", c("daily", "weekly", "monthly"))),
  column(7,
  numericInput("window", " Rolling Window", 10, min = 5, max = 50, step = 5))
)

# I like giving the user the option of whether to display the mean, min and max. 
# It doesn't make a huge difference, of course, but at least enables the user to 
# notice the lines more consistently. 
checkboxInput("max", label = 'Display Max Rolling Correlation', value = FALSE)
checkboxInput("mean", label = 'Display Mean Rolling Correlation', value = FALSE)
checkboxInput("min", label = 'Display Min Rolling Correlation', value = FALSE)

Have a close look at the last three lines of code in that chunk. These are a new addition that let the user determine if the mean, max and/or min rolling correlation should be included in the dygraph. We haven’t built any way of calculating those values yet, but we will shortly. This is the UI component.

Those three lines of code create checkboxes and are set to default as FALSE, meaning they won’t be plotted unless the user chooses to do so. I wanted to force the user to actively click a control to include these, but that’s a purely stylistic choice. Perhaps you don’t want to give them a choice at all here?

Next, we create our reactive values that will form the substance of this app. First, we need to calculate and store an object of rolling correlations, and we’ll use a reactive that passes user inputs to our sector_correlations function.

Then, we build reactive objects to store mean, minimum and maximum rolling correlations. These values will help contextualize our final dygraph.

# Build our correlation time series object here
# so we can access it throughout the rest of the app.

sector_correlation <- reactive({
  sector_corr <- sector_correlations(input$sector, input$period, input$window)
  sector_corr[, 3]
})

# Let's calculate mean, minimum and maximum rolling correlations. 
# We'll use these on the graph and in the value boxes.
avg <- reactive({ 
  avg <- round(mean(sector_correlation(),  na.rm=T), 2)
  })
mini <- reactive({
  mini <- round(min(sector_correlation(),  na.rm=T), 2)
  })
maxi <- reactive({
  maxi <- round(max(sector_correlation(),  na.rm=T), 2)
  })

At this point, we have done some good work: built a function to calculate rolling correlations based on user input, built a sidebar to take that user input, and coded reactives to hold the values and some helpful statistics. The hard work is done, and really we did most of the hard work in the Notebook, where we toiled over the logic of arriving at this point. All that’s left now is to display this work in a compelling way. Dygraphs plus value boxes has worked in past; let’s stick with it!

# Dygraph time!

dygraphOutput("dygraphCorrelation")

output$dygraphCorrelation <- renderDygraph({
  
  dygraph(sector_correlation(), 
          main = paste("rolling", input$period, "correlation ", input$sector, "/SPY", sep = " ")) %>% 
  # A new wrinkle that hasn't been included in previous flexdashboards: let the user add lines to the 
  # dygraph.   
  dyLimit(if(input$mean == TRUE) {avg()}, color = 'black') %>% 
  dyLimit(if(input$min == TRUE) {mini()}, color = 'red') %>% 
  dyLimit(if(input$max == TRUE) {maxi()}, color = 'blue')
  
})

That dygraph code should look familiar from the Notebook and previous posts, except we have added a little interactive feature. By including if(input$mean == TRUE) {avg()}, we allow the user to change the graph by checking or unchecking the ‘mean’ input box in the sidebar. We are going to display this same information numerically in a value box, but the lines make this graph a bit more compelling.

Speaking of those value boxes, they rely on the reactives we built above, but, unlike the graph lines, they are always going to be displayed. The user doesn’t have a choice here.

valueBoxOutput("approvalBox1")
output$approvalBox1<-renderValueBox({
 
  # Display the mean we calculated in 'avg' reactive function above.
  
  valueBox(value = avg(), icon = "fa-line-chart", color = "primary")
  
})

Again, this just adds a bit of context to the graph. Note that the lines and the value boxes take their value from the same reactives. If we were to change those reactives, both UI components would be affected.

Our job is done! This a simple but powerful app: the user can choose to see the 60-day rolling correlations between the S&P 500 and an energy ETF, or the 10-month rolling correlations between the S&P 500 and a utility ETF, etc. I played around with this a little bit and was surprised that the 10-week rolling correlation between the S&P 500 and health care stocks plunged in April of 2016. Someone smarter than I can probably explain, or at least hypothesize, as to why that happened.

A closing thought about how this app might have been different: we are severely limiting what the user can do here, and intentionally so. The user can choose only from the sector ETFs that we are offering in the selectInput dropdown. This is a sector correlations app, so I included only a few sector ETFs. But, we could just as easily have made this a textInput and allowed the users to enter whatever ticker symbol struck their fancy. In that case, this would not longer be a sector correlations app; it would be a general stock correlations app. We could go even further and make this a general asset correlations app, in which case we would allow the user to select things like commodity, currency and housing returns and see how they correlate with stock market returns. Think about how that might change our data import logic and time series alignment.

Thanks for reading, enjoy the app, happy coding, and see you next time!

The R Formula Method: The Good Parts

Wed, 01 Feb 2017 00:00:00 +0000

Introduction

The formula interface to symbolically specify blocks of data is ubiquitous in R. It is commonly used to generate design matrices for modeling function (e.g. lm). In traditional linear model statistics, the design matrix is the two-dimensional representation of the predictor set where instances of data are in rows and variable attributes are in columns (a.k.a. the X matrix).

A simple motivating example uses the inescapable iris data in a linear regression model:

mod1 <- lm(Sepal.Width ~ Petal.Width + log(Petal.Length) + Species, 
           data = iris, subset = Sepal.Length > 4.6)

While the purpose of this code chunk is to fit a linear regression models, the formula is used to specify the symbolic model as well as generating the intended design matrix. Note that the formula method defines the columns to be included in the design matrix, as well as which rows should be retained.

Formulas are used in R beyond specifying statistical models, and their use has been growing over time (see this or this).

In this post, I’ll walk through the mechanics of how some modeling functions use formulas to make a design matrix using lm to illustrate the details. Note, however, that the syntactical minutiae are likely to be different from function to function, even within base R.

Formulas and Terms

lm initially uses the formula and the appropriate environment to translate the relationships between variables to creating a data frame containing the data. R has a fairly standard set of operators that can be used to create a matrix of predictors for models.

We will start by looking at some of the internals of lm (circa December 2016).

Preparing for the Model Frame

The main tools used to get the design matrix are the model.frame and model.matrix functions. The definition and first few lines of lm are:

function (formula, data, subset, weights, na.action, method = "qr", 
    model = TRUE, x = FALSE, y = FALSE, qr = TRUE, singular.ok = TRUE, 
    contrasts = NULL, offset, ...)  {
  
    ret.x <- x
    ret.y <- y
    cl <- match.call()
    mf <- match.call(expand.dots = FALSE)
    m <- match(x = c("formula", "data", "subset", "weights", "na.action", "offset"), 
               table = names(mf), nomatch = 0L)
    mf <- mf[c(1L, m)]
    mf$drop.unused.levels <- TRUE
    mf[[1L]] <- quote(stats::model.frame)
    mf <- eval(expr = mf, envir = parent.frame())

The goal of this code is to manipulate the formula and other arguments into an acceptable set of arguments to the model.frame function in the stats package. The code will modify the call to lm to use as the substrate into model.frame, which has many similar arguments (e.g. formula, data, subset, and na.action). However, there are arguments that are not common to both functions.

The object mf is initially created to mirror the original call. After executing match.call(expand.dots = FALSE), our original call generates a value of

lm(formula = Sepal.Width ~ Petal.Width + log(Petal.Length) + Species, 
   data = iris, subset = Sepal.Length > 4.6)

and class(mf) has a value of "call". Note that the first element of the call, mf[[1L]], has a value of lm with the class of name.

The next few lines remove any arguments to lm that are not arguments to model.frame, and adds another (drop.unused.levels). Finally, the call is modified by replacing the first element of the call (lm) to stats::model.frame. Now, mf, has a value of

stats::model.frame(formula = Sepal.Width ~ Petal.Width + log(Petal.Length) + Species, 
                   data = iris, 
                   subset = Sepal.Length > 4.6, 
                   drop.unused.levels = TRUE)

Creating the Model Frame and Terms

When this code is executed using eval(expr = mf, envir = parent.frame()), the model.frame function returns

A data.frame containing the variables used in formula plus those specified in .... It will have additional attributes, including “terms” for an object of class “terms” derived from formula, and possibly “na.action” giving information on the handling of NAs (which will not be present if no special handling was done, e.g. by na.pass).

For our particular call, the first six values of mf are

  Sepal.Width Petal.Width log(Petal.Length) Species
1         3.5         0.2             0.336  setosa
2         3.0         0.2             0.336  setosa
3         3.2         0.2             0.262  setosa
5         3.6         0.2             0.336  setosa
6         3.9         0.4             0.531  setosa
8         3.4         0.2             0.405  setosa

Note that :

all of the columns are present, predictors and response,
the filtering defined by the subset command is executed here (note the row names above),
the Petal.Length has been log transformed and the column name is not a valid name, and
the Species variable has not generated any dummy variables.

If weights or an offset was used in the model, the resulting model frame would also include these.

As alluded to above, mf has several attributes, and includes one that would not normally be associated with a data frame (e.g. "terms"). The terms object contains the data that defines the relationships between variables in the formulas, as well as any transformations of the individual predictors (e.g. log). For our original model:

mod1$terms

## Sepal.Width ~ Petal.Width + log(Petal.Length) + Species
## attr(,"variables")
## list(Sepal.Width, Petal.Width, log(Petal.Length), Species)
## attr(,"factors")
##                   Petal.Width log(Petal.Length) Species
## Sepal.Width                 0                 0       0
## Petal.Width                 1                 0       0
## log(Petal.Length)           0                 1       0
## Species                     0                 0       1
## attr(,"term.labels")
## [1] "Petal.Width"       "log(Petal.Length)" "Species"          
## attr(,"order")
## [1] 1 1 1
## attr(,"intercept")
## [1] 1
## attr(,"response")
## [1] 1
## attr(,".Environment")
## <environment: R_GlobalEnv>
## attr(,"predvars")
## list(Sepal.Width, Petal.Width, log(Petal.Length), Species)
## attr(,"dataClasses")
##       Sepal.Width       Petal.Width log(Petal.Length)           Species 
##         "numeric"         "numeric"         "numeric"          "factor"

The terms object will be used to generate design matrices on new data (e.g. samples being predicted).

Creating the Design Matrix

The lm code has some additional steps to save the model terms and generate the design matrix:

mt <- attr(x = mf, which = "terms")
x <- model.matrix(object = mt, data = mf, contrasts.arg = contrasts)

The model.matrix function uses the data in the terms object to generate any interactions and/or dummy variables from factors. This work is mostly accomplished by a C routine.

The Predictive Nature of the `terms`

In the previous example, the log transformation is applied to one of the columns. When using an inline function inside a formula, this transformation will be applied to the current data, as well as any future data points (say, via predict.lm). The same workflow is followed where a model frame is used with the terms object and model.matrix.

However, there are some operations that can be specified in a formula that require statistical estimates. Two examples:

a natural spline (splines::ns) takes a numeric variable, does some computations, and expands that variable into multiple features that can be used to model that predictor in a nonlinear fashion.
orthogonal polynomials (stats::poly) is a basis expansion that takes a single predictor and produces new columns that correspond to the polynomial degree.

As an example of natural splines:

library(splines)
pwidth_ns <- ns(iris$Petal.Width[1:100], df = 2)
tail(pwidth_ns)

##            1       2
##  [95,] 0.539  0.1326
##  [96,] 0.563  0.0260
##  [97,] 0.539  0.1326
##  [98,] 0.539  0.1326
##  [99,] 0.577 -0.0662
## [100,] 0.539  0.1326

ns returns multiple elements: the basis function spline results (shown just above) and the data required to generate them for new data (in the attributes).

attributes(pwidth_ns)

## $dim
## [1] 100   2
## 
## $dimnames
## $dimnames[[1]]
## NULL
## 
## $dimnames[[2]]
## [1] "1" "2"
## 
## 
## $degree
## [1] 3
## 
## $knots
## 50% 
## 0.8 
## 
## $Boundary.knots
## [1] 0.1 1.8
## 
## $intercept
## [1] FALSE
## 
## $class
## [1] "ns"     "basis"  "matrix"

It turns out that the only statistics required to produce new spline results are the knots, Boundary.knots, and intercept attributes. When new data are predicted, those statistical quantities are used:

tail(predict(pwidth_ns, iris$Petal.Width[101:150]))

##             1     2
## [45,] -0.0344 1.799
## [46,]  0.0671 1.513
## [47,]  0.2702 0.941
## [48,]  0.2194 1.084
## [49,]  0.0671 1.513
## [50,]  0.3210 0.798

Now, getting back to formulas, we can include a function like this inline:

mod2 <- lm(Sepal.Width ~ ns(Petal.Width, df = 2) + Species, data = iris)

The resulting terms object saves the model specification, and also the values required to reproduce the spline basis function. The terms object contains an attribute that is misleadingly named predvars that has this information:

attr(x = mod2$terms, which = "predvars")

## list(Sepal.Width, ns(Petal.Width, knots = 1.3, Boundary.knots = c(0.1, 
## 2.5), intercept = FALSE), Species)

## The part with `ns` is in the _third_ element:
attr(x = mod2$terms, which = "predvars")[[3]]

## ns(Petal.Width, knots = 1.3, Boundary.knots = c(0.1, 2.5), intercept = FALSE)

When predict.lm is invoked with a new data set, the terms are passed to model.frame and the predvars are evaluated on the new data, e.g.,

eval(attr(x = mod2$terms, which = "predvars")[[3]], envir = head(iris))

##           1       2
## [1,] 0.0635 -0.0422
## [2,] 0.0635 -0.0422
## [3,] 0.0635 -0.0422
## [4,] 0.0635 -0.0422
## [5,] 0.0635 -0.0422
## [6,] 0.1878 -0.1226
## attr(,"degree")
## [1] 3
## attr(,"knots")
## 50% 
## 1.3 
## attr(,"Boundary.knots")
## [1] 0.1 2.5
## attr(,"intercept")
## [1] FALSE
## attr(,"class")
## [1] "ns"     "basis"  "matrix"

In summary, R’s formula interface works by exploiting the original formula as a general expression, and the terms object is where most of the information about the design matrix is stored.

Finally, it is possible to include more complex operations in the formula. For example, two techniques for imputation in predictive models are using tree ensembles and K-nearest neighbors. In each case, a model can be created to impute a predictor, and this model also could be embedded into predvars as long as the prediction function can be exploited as an expression.

There are some severe limitations to formulas which may not be obvious and various packages have had to work around these issues. The second post on formulas (“The Bad”) will illustrate these shortcomings.

January '17 Tips and Tricks

Fri, 27 Jan 2017 00:00:00 +0000

This month’s collection of Tips and Tricks comes from an excellent talk given at the 2017 RStudio::Conf in Orlando by RStudio Software Engineer Kevin Ushey. The slides from his talk are embedded below and cover features from autocompletion to R Markdown shortcuts. Use the left and right arrow keys to change slides.

Enjoy!

R Markdown for the Enterprise

Wed, 25 Jan 2017 00:00:00 +0000

In the corporate world, spreadsheets and PowerPoint presentations still dominate as the tools used for analyzing and sharing information. So, it is not at all surprising that even when business analysts use R for the analytical heavy lifting, they frequently revert to using spreadsheets and slide decks to share their results. This may seem like the easiest way to communicate with colleagues, but any modestly complicated project is likely to be error-prone and generate hours of unnecessary rework.

An R-savvy analyst can harness R Markdown to develop reproducible business reporting and information sharing workflows in any business organization; all it takes is a little effort to master some basic R document preparation tools.

In this post, I would like to examine a scenario that represents some experiences I had as an analytics professional.

“The report is great but…” Scenario

A new R analysis is delivered in a PowerPoint presentation, and everyone thinks that the insights are very valuable. They all want more associates to see it, so almost immediately, the following three requests are made:

“…we need it broken out by” - The presentation needs to be split by a specific segment. The segment is normally geographical or managerial in nature.
“…they shouldn’t see each others data” - Since the results are not published in a central publishing platform, it is necessary to create multiple versions of the same report in order to secure the contents.
“…we need it every” - Satisfying requests 1 and 2 may not be too overwhelming if this were meant as a one-time analysis, but usually the analysis and its distribution need to be repeated on a regular interval.

Because we exported the findings into a presentation, sharing results becames more complex and time-consuming if we wish to satisfy the new requirements.

How can R Markdown help?

R Markdown combines the creation and sharing steps. The three requests can be satisfied using the following features of R Markdown:

Break out the reports - Using R Markdown’s Parameterized Reports feature, we can easily create documents for each required segment.
Automate the file creation - R Markdown can be run from code, so a separate R script can iteratively run the R Markdown and pass a different parameter for each iteration.
Create the slides inside R - Take advantage of R Markdown Presentation output to create a slide deck. Without having to learn a new scripting language, we can code the slide deck and use the same Parameter feature to automate its creation.
Keep the interactivity - In many cases, the end user needs a level of interactivity with the report. This interactivity can be achieved by using htmlwidgets inside the R Markdown document. For example, the Leaflet widget can be used for interactive maps, the Data Table widget for interactive tables, and the dygraphs widget for interactive time series charting.

Additional benefits

Accessible and easy to open - Any alternative tool needs to be as accessible as the current spreadsheet and presentation tool. R Markdown can output results in HTML, PDF, and Word. Additionally, the Presentation output uses the highly accessible HTML5 format.
Reproducibility - Copying-and-pasting files, text, or images inevitably introduces human error. In R, data import, wrangling and modeling are already automated, so why not take it to its natural conclusion by using R Markdown to automate the presentation end of the process, as well?
Creating a dashboard is easy - In a spreadsheet, this is normally accomplished with a combination of pivot tables and graphs. R Markdown uses flexdashboard to create visually striking dashboards that are self-contained. By using this in combination with htmlwidgets, the audience gains access to a very powerful tool.

Here is an example of a live parameterized R Markdown flexdashboard based on stock data:

How to get started

R Markdown is a free package, so if you have R (and ideally RStudio), you can start using it today. Also, there are a lot of resources available for learning how to use R Markdown; the package’s official website is a good place to start.

Here is a sample script that uses Parameterized R Markdown to create a slide deck based on a selected stock. In this case we used Google:

And here is the resulting deck. Press the left arrow key to see the next slide:

This simple script creates an nice-looking and interactive deck that needs no manual intervention if the data needs to be refreshed, and one small parameter change if a different stock is to be selected.

Final thought

We encourage you to try R Markdown yourself. The “start small and then build big” strategy rarely fails, so you could begin by automating a simple report first, and then start taking advantage of more advanced features as you grow comfortable with the tool.

Introducing sparklyr to the Madrid R User Group

Fri, 20 Jan 2017 00:00:00 +0000

In the last meeting of 2016, the 40th in Madrid’s R Users Group five-year history, we had the opportunity to listen (via Skype) to a very interesting talk by Javier Luraschi, the main author of the package sparklyr.

In our previous meeting, a colleague of the Community (José Luis Cañadas) made a first introduction to sparklyr. José Luis presented the processing capacities of sparklyr on a Spark instance, as well as an interface with the h2o package. His presentation was very well-received, and generated considerable excitement for Javier’s talk.

In his presentation, Javier presented background information and details on the sparklyr package, including how it was conceived and launched, its development scheme, and what is involved in setting up sparklyr on a production cluster. Javier showed how to deploy sparlkyr on Amazon EMR, and how to use RStudio Server to use the machine learning algorithms of the Spark MLlib libraries with dplyr syntax.

Javier covered all these items very thoroughly in his presentation, and also presented the new changes coming with the then forthcoming release (version 0.5 was officially released some days later).

During the Q&A section, Javier provided additional details: * The intention to continue to support Spark. * Clarified information about how Cloudera supports sparklyr (Cloudera recommends the use of sparkly with the version of Spark 1.6.2)

On behalf of Madrid’s R User Group, we want to thank Javier for his willingness to participate in our meeting, and for his excellent work leading the development with sparklyr.

Note: All sessions of the Madrid’s R User Group are recorded on video. The videos, presentations, and the code of these two meetings can be found (in Spanish) at the URLs given above.

Sector Correlations

Wed, 18 Jan 2017 00:00:00 +0000

Welcome to the first installation of reproducible finance for 2017. It’s a new year, a new President takes office soon, and we could be entering a new political-economic environment. What better time to think about a popular topic over the last few years: equity correlations. Elevated correlations are important for several reasons - life is hard for active managers and diversification gains are vanishing - but I personally enjoy thinking about them more from an inference or data exploration perspective. Are changning correlations telling us something about the world? Are sectors diverging? How much can be attributed to the Central Bank regime at hand? So many questions, so many hypotheses to be explored. Let’s get started.

Today, we will build a Notebook and start exploring the historical rolling correlations between sector ETFs and the S&P 500. That is, we want to explore how equity returns in different sectors have been correlated with the returns of the broader index. Perhaps they are all moving in lockstep, perhaps they have been diverging. Either way, this Notebook will be the first step toward an flexdashboard that lets us do more interactive exploration - choosing different sector ETFs and rolling windows.

We are going to accomplish a few things today. We will load up the sector ETF tickers, then build a function to download their price history and calculate weekly returns. We will save this to one xts object. Next we will build a function to calculate the rolling correlations between a chosen sector ETF and the S&P 500. Finally, dygraphs will make its usual appearance to help visualize the rolling correlation time series.

As usual, we will be living in the Rmarkdown world and, by way of disclaimer, the data import and return calculation functions here should be familiar from previous posts. That is by design and hopefully, it won’t be too boring for devotees of this series (I know you’re out there somewhere!). More importantly, I hope the usefulness of reproducible, reusable code is emerging. Some of the code chunks in previous posts might have seemed trivially simple, containing just a simple function and little else. But, the simplicity of those code chunks made it very easy to return to those previous scripts, understand the functions, and use them in this post.

Let’s load up a few packages.

library(dplyr)
library(quantmod)
library(dygraphs)

Now, we need the tickers and sectors for the sector ETFs. They are copied below and available here. I deleted the XLRE real estate ETF because it’s only been around since 2015 and I want look back several years in this Notebook.

# List of tickers for sector etfs. 
# Omit XLRE because its inception is 2015. Not really enough to check out long term monthly correlations. 

ticker <- c("XLY", "XLP", "XLE",    "XLF", "XLV",   
            "XLI", "XLB", "XLK", "XLU", "SPY")  

# And the accompanying sector names for those ETFs.

sector <- c("Consumer Discretionary", "Consumer Staples", 
            "Energy", "Financials", "Health Care", "Industrials", 
            "Materials", "Information Technology", "Utilities", "Index")

etf_ticker_sector <- data_frame(ticker, sector)

etf_ticker_sector

## # A tibble: 10 × 2
##    ticker                 sector
##     <chr>                  <chr>
## 1     XLY Consumer Discretionary
## 2     XLP       Consumer Staples
## 3     XLE                 Energy
## 4     XLF             Financials
## 5     XLV            Health Care
## 6     XLI            Industrials
## 7     XLB              Materials
## 8     XLK Information Technology
## 9     XLU              Utilities
## 10    SPY                  Index

We’ve got our dataframe of tickers and sectors. Let’s build a function to download price history and then convert those price histories to weekly returns. We’ll use a combination of getSymbols() and periodReturn() to accomplish that. If you want to change this script to use daily returns, change the argument below to period = ‘daily’, but be prepared to import quite a bit more data.

# A function to build an xts object of etf returns.

etf_weekly_returns <- function(ticker) {
    
symbols <- getSymbols(ticker, auto.assign = TRUE, warnings = FALSE)

etf_prices <- do.call(merge, lapply(symbols, function(x) Cl(get(x))))
    
etf_returns <- do.call(merge, lapply(etf_prices, 
                                     function(x) periodReturn(x, period = 'weekly', type = 'log')))

#Change the column names to the sector names from our dataframe above.

colnames(etf_returns) <- etf_ticker_sector$sector
    
etf_returns

}

# Let's pass in our ticker symbols and build an xts object of etf returns
etf_returns <- etf_weekly_returns(etf_ticker_sector$ticker)

##     As of 0.4-0, 'getSymbols' uses env=parent.frame() and
##  auto.assign=TRUE by default.
## 
##  This  behavior  will be  phased out in 0.5-0  when the call  will
##  default to use auto.assign=FALSE. getOption("getSymbols.env") and 
##  getOptions("getSymbols.auto.assign") are now checked for alternate defaults
## 
##  This message is shown once per session and may be disabled by setting 
##  options("getSymbols.warning4.0"=FALSE). See ?getSymbols for more details.

## pausing 1 second between requests for more than 5 symbols
## pausing 1 second between requests for more than 5 symbols
## pausing 1 second between requests for more than 5 symbols
## pausing 1 second between requests for more than 5 symbols
## pausing 1 second between requests for more than 5 symbols
## pausing 1 second between requests for more than 5 symbols

head(etf_returns)

##            Consumer Discretionary Consumer Staples       Energy
## 2007-01-05           -0.007034027     -0.005722007 -0.011733539
## 2007-01-12            0.026062578      0.014055350 -0.008620725
## 2007-01-19            0.008875499      0.003765102  0.020354264
## 2007-01-26           -0.019888442     -0.004519819  0.004760584
## 2007-02-02            0.022409825      0.011634547  0.030830343
## 2007-02-09           -0.006315545     -0.010502685 -0.004101903
##              Financials  Health Care  Industrials     Materials
## 2007-01-05 -0.011167160  0.005655589 -0.007693470 -0.0113323889
## 2007-01-12  0.013331718  0.017361037  0.017579216  0.0290876947
## 2007-01-19 -0.001081666  0.011312637  0.001123511  0.0152115609
## 2007-01-26 -0.003795098 -0.012772249 -0.011577145  0.0086291974
## 2007-02-02  0.015362168  0.014501414  0.027452705  0.0151288535
## 2007-02-09 -0.005901386 -0.005485765 -0.009439299 -0.0008194183
##            Information Technology    Utilities        Index
## 2007-01-05            0.007260335 -0.019471313 -0.005888436
## 2007-01-12            0.021886396 -0.006668599  0.019029484
## 2007-01-19           -0.027005903  0.005005572 -0.002936435
## 2007-01-26           -0.001283929  0.005532545 -0.004842978
## 2007-02-02            0.016985546  0.023177534  0.018680358
## 2007-02-09           -0.008881386  0.025021348 -0.006025965

This function has done some good work for us and it was refreshingly comfortable to put in place because we used very similar functionality in this post and this post.

A pattern seems to be emerging in these Notebooks: grab tickers, get price history, convert to returns and save new xts object. In an ideal world, that pattern of data import and conversion is getting so familiar as to be commonplace.

That said, enough with the commonplace stuff - let’s get on to something a little more dangerous: rolling correlations amongst etf returns. Correlations are important because high correlations make it hard to find diversification opportunities and they make it hard to deliver alpha - though I suppose it’s always hard to deliver alpha. Fortunately, we don’t have to worry about generating alpha today so let’s get to our function.

Calculating rolling correlations in R is pretty straightforward. We use the rollapply() function, along with the cor() function, pass in our data and a time window, and it’s off to the races. We’ll create our own function below to handle these jobs and return an xts object.

# A function that calculates the rolling correlation between a sector ETF and the SPY SP500 ETF. 

sector_index_correlation <- function(x, window) {
  
    merged_xts <- merge(x, etf_returns$'Index')

    merged_xts$rolling_test <- rollapply(merged_xts, window, 
                                         function(x) cor(x[,1], x[,2], use = "pairwise.complete.obs"), 
                                         by.column = FALSE)
    
    names(merged_xts) <- c("Sector Returns", "SPY Returns", "Sector/SPY Correlation")
    
    merged_xts
}

Notice that this function does something that seems unnecessary: it creates a new xts object that holds the sector returns, SPY returns and the rolling correlation. We don’t have much use for that separate object and could probably have just added columns to our original xts object. Indeed, if this were our final product we might spend more time eliminating its present. I choose not to do that here for two reasons. First, this Notebook is built to underlie a flexdashboard that could go into production. I want to get the logic right here, then focus more on efficiency in the final app.

Second, and relatedly, we are prioritizing clarity of workflow in this Notebook. It should be crystal clear how we are moving from an xts object of ETF returns to creating a new XTS object of two returns plus one correlation. The goal is for any collaborators, including my future self, to open this Notebook and see the workflow. If that collaboarator finds this step to be unnecessary and has a more clever solution - that’s fantastic because it means this document is intellegible enough to serve as the basis for more sophisticated work.

Let’s go ahead and use this function. We will pass in a time series of Information Technology ETF returns and a window of size 20 for the rolling correlation.

# Choose a sector ETF and a rolling window and pass them to the function we just build. 
# Let's go with a 5 month window and the Information Technology sector.
# We will now have a new xts object with 3 time series: sector returns, SPY returns
# and the rolling correlation between those return series.

IT_SPY_correlation <- sector_index_correlation(etf_returns$'Information Technology', 20)

# Have a peek. The first 20 rows in the correlation column should be 
# NAs. 

head(IT_SPY_correlation, n = 25)

##            Sector Returns   SPY Returns Sector/SPY Correlation
## 2007-01-05    0.007260335 -0.0058884356                     NA
## 2007-01-12    0.021886396  0.0190294841                     NA
## 2007-01-19   -0.027005903 -0.0029364349                     NA
## 2007-01-26   -0.001283929 -0.0048429782                     NA
## 2007-02-02    0.016985546  0.0186803576                     NA
## 2007-02-09   -0.008881386 -0.0060259646                     NA
## 2007-02-16    0.015177315  0.0123590071                     NA
## 2007-02-23    0.003341691 -0.0029549759                     NA
## 2007-03-02   -0.057501940 -0.0467035959                     NA
## 2007-03-09    0.015776007  0.0151013857                     NA
## 2007-03-16   -0.002176236 -0.0161114796                     NA
## 2007-03-23    0.026656737  0.0344812823                     NA
## 2007-03-30   -0.010663355 -0.0097411262                     NA
## 2007-04-05    0.022892097  0.0156515560                     NA
## 2007-04-13    0.001256461  0.0074596420                     NA
## 2007-04-20    0.014131615  0.0224544236                     NA
## 2007-04-27    0.015561370  0.0061043558                     NA
## 2007-05-04    0.016122840  0.0092528469                     NA
## 2007-05-11    0.007171385 -0.0003976208                     NA
## 2007-05-18    0.004752405  0.0115988780              0.8820099
## 2007-05-25   -0.004752405 -0.0061121613              0.8963962
## 2007-06-01    0.018096442  0.0156329832              0.8946237
## 2007-06-08   -0.012159397 -0.0199273050              0.9301269
## 2007-06-15    0.017214861  0.0133507228              0.9311254
## 2007-06-22   -0.009744766 -0.0166001040              0.9302817

Alright, the function seems to have succeeded in building that new xts object and storing the rolling correlation. Now we will use dygraphs to visualize this rolling correlation over time and see if anything jumps out as interesting or puzzling.

# Let's graph the sector/index rolling correlation using our old friend dygraphs. 
# I'll add an event for the financial crisis and shade the recession of 2007-2009 with 
# an unimposing pink hue.

dygraph(IT_SPY_correlation$'Sector/SPY Correlation', main = "Correlation between SP500 and Tech ETF") %>% 
    dyAxis("y", label = "Correlation") %>% 
    dyRangeSelector(height = 20) %>%
    dyShading(from = "2007-12-01", to = "2009-06-01", color = "#FFE6E6") %>% 
    dyEvent(x = "2008-09-15", label = "Fin Crisis", labelLoc = "top", color = "red")

The correlation between the Tech ETF and the S&P 500 ETF seems quite high. It dipped a bit in the middle of 2009 and again towards the end of 2016. It would be interesting to see if this was true of the other sector ETFs as well. In other words, were these periods of generally declining correlations, or was it limited to the technology/SP500 relationship?

The best way to do some exploratory analysis on that is, no surprise, build a shiny app that allows users to choose their own sectors and rolling windows. We’ll do that next time - see you in a few days!

R on R Views

Multiple Versions of R

A solid foundation for R

Building R from source

RStudio professional products

References

A Data Science Lab for R

Architecture

Instance Sizing

Open-Source R

RStudio Server Pro

1. Installation

2. Configuration

3. Management

Next Steps

WordR - A New R Package for Rendering Documents in MS Word Format

Motivation

How to use it

MS Word part

Example

R part

Back to our example

A few comments

Summary

Survival Analysis with R

Load the data

Kaplan Meier Analysis

Cox Proportional Hazards Model

Random Forests Model

Some Tutorials and Papers

References

Writing and Publishing my first R package

Inspired by the Community

Writing padr

How padr works

Lessons Learned

July 2017 New Package Picks

The July 2017 Top 40

Machine Learning

Numerical Methods

Science

Statistics

Time Series

Utilities

Visualization

Control Systems Toolbox – System Interconnection

Introduction

Laplace Transform

Transfer Function Expressions

Electric Motor Example

Open-Loop Response of the electric motor

Closed-Loop Response of the electric motor

System Interconnections

Append Connection using the append function

Series Connection using the series function

Parallel Connection using the parallel function

Feedback Connection using the feedback function

General block diagram building using connect function

Case Study – Unmanned Free Swimming Submersible (UFSS) Vehicle

Rationale:

Solution using series, parallel and feedback

Solution via algebraic operations

Feedback

Conclusion

References

Learning things we already know about stocks

Software used in the example

Getting data

Download daily closing price data from Google Finance

Clean up and transform data

Sample correlation matrix

Regularization

But how to choose a cut-off rank?

A precision matrix, finally

Other approaches

Put a package on it

Networks and clustering

Review

Appendix: threejs tricks

Precision matrix quantile threshold (adjust slider to change)

Writing `padr`

How `padr` works

Append Connection using the `append` function

Series Connection using the `series` function

Parallel Connection using the `parallel` function

Feedback Connection using the `feedback` function

General block diagram building using `connect` function

Solution using `series`, `parallel` and `feedback`

Introducing `dbplot`

Example²