RStudio on R Views

Greg Wilson Wins ACM Influential Educator Award

Mon, 04 May 2020 00:00:00 +0000

Recently, the Association for Computing Machinery’s (ACM’s) Special Interest Group on Software Engineering (SIGSOFT) recognized Greg Wilson as the 2020 recipient of its prestigious Influential Educator Award which is awarded annually to individuals or groups who have made significant contributions to software engineering through education, mentoring or policy.

JBR: Greg Congratulations, and thank you for agreeing to this interview.

Greg, you have been an educator for quite some time, first as a professor of computer science, then with Software Carpentry and now RStudio. During your career technology has affected the educational experience for both students and teachers alike. What do you think have been the most dramatic changes?

Greg: Technically, Stack Overflow is the biggest change in computing education in the last 20 years. It has a lot of problems—women frequently have to post under pseudonyms in order to avoid harassment, for example—but every professional programmer I know consults it many times a day, and I think educators have an obligation to include it rather than ignore it.

But the more important change has been cultural. Guys like me didn’t pay much attention to equity or inclusion 20 years ago, and when we did, it was to make snarky jokes. Today, most communities have a code of conduct, all-white or all-male panels at conferences are called out, and pay disparities are discussed openly. (This article by Julia Silge is a great example.) We still have a long way to go, but at least we’re now talking about things that really matter.

JBR: Economic pressures over recent years have led many colleges and universities to explore online learning as a cost effective alternative to the classroom experience, how do you see these efforts going? Have there been any notable successes? Do you think that online experience is changing the relationship between teachers and students?

Greg: There’s a joke that online learning is what everyone wants for someone else’s kids. Distance learning can be just as effective as in-person learning—the Open University in the UK, for example, has been proving that annually since the 1970s—but MOOCs were driven by “what technology do we have?” and “can we scale quickly enough to keep the VCs interested?” rather than “how do we teach effectively?” I think the next decade will see us backing away from recorded video and robo-graded exercises and emphasizing real-time peer-to-peer interaction instead. It’s pedagogically more effective, and learners who have grown up playing Minecraft with each other will wonder why we ever did anything else.

JBR: The COVID-19 pandemic is forcing educators at all levels from pre-school teachers to college professors to adapt, “cold turkey” to the reality of completely immersive online education. What advice do you have for teachers and students?

GREG: The first is “only change what you absolutely have to”. The second is to ask learners rather than making assumptions: their devices may be a lot less powerful than yours, or shared with other family members, or they may not have bandwidth or a quiet place for meetings, and so on. The third, which is probably most important, is to be kind: if there was ever a time to give learners partial marks or extensions on homework, it’s now.

JBR: Do you distinguish between education and training, and if so do, you think the online experience can be effective for both tasks?

Greg: I don’t think there’s a meaningful distinction. I don’t believe people can learn to think about computing in any meaningful way without learning how to program, or vice versa. I think (at least I hope) the biggest change we’ll see from moving programming education online is a much wider adoption of live coding, and with it, much more emphasis on how programs are written, not just what they look like when they’re done.

JBR: Is there anything special about teaching R? Does the deep association of R with statistical inference and methods pose any particular challenges for educators?

Greg: I think the biggest challenge in teaching R is getting people with computer science backgrounds like mine to take it seriously. Nobody would design a language that way today, but then, nobody would design C or JavaScript as they are either, and they’re all extraordinarily useful tools.

I think the other challenge is that we don’t really know how to test data analyses. The unit testing paradigm of software engineering isn’t really appropriate: it doesn’t tell us how close is close enough when we’re checking answers, and its focus is building reusable products, not answering questions. A data scientist constructing an R Markdown notebook step by step is doing something very different from what a developer building an e-commerce site does; I hope to learn a lot more about those differences in the next few years so that we can teach data scientists more effectively.

JBR: Is there anything we did not touch on that you would like to communicate to our R Views readers before we wrap up here?

Greg: Yes: please wash your hands, and please vote.

rstudio::conf 2020 Videos

Tue, 18 Feb 2020 00:00:00 +0000

rstudio::conf 2020 is already receding in the rear view mirror, but the wealth of resources generated by the conference will be valuable for quite some time. All of the materials from the workshops, and now all one hundred and four videos of conference talks are available. This unique video collection offers valuable insight into how developers, data scientists, statisticians, journalists, physicians, educators and other R savvy professionals are using their domain knowledge, analytical expertise and coding skills to make the world a better place. Talks range from very technical, R specific topics; to scientific contributions, best practices, applications of reproducible research, and the culture and sociology surrounding open source software.

There is a lot to see, but my advice is to is to begin with J.J. Allaire’s keynote which presents open source software for data science through the lens of RStudio’s history and mission.

J.J. Allaire's Keynote at rstudio::conf 2020, announcing RStudio, PBC

Setting up RStudio Server on a Cloud for Collaboration and Reproducibility

Wed, 17 Apr 2019 00:00:00 +0000

Roland Stevenson is a data scientist and consultant who may be reached on Linkedin.

When setting up R and RStudio Server on a cloud Linux instance, some thought should be given to implementing a workflow that facilitates collaboration and ensures R project reproducibility. There are many possible workflows to accomplish this. In this post, we offer an “opinionated” solution based on what we have found to work in a production environment. We assume all development takes place on an RStudio Server cloud Linux instance, ensuring that only one operating system needs to be supported. We will keep the motivation for good versioning and reproducibility short: R projects evolve over time, as do the packages that they rely on. R projects that do not control package versions will eventually break and/or not be shareable or reproducible¹.

Since R is a slowly evolving language, it might be reasonable to require that a particular Linux instance have only one version of R installed. However, requiring all R users to use the same versions of all packages to facilitate collaboration is clearly out of the question. The solution is to control package versions at the project level.

We use packrat to control package versions. Already integrated with RStudio Server, packrat ensures that all installed packages are stored with the project², and that these packages are available when a project is opened. With packrat, we know that project A will always be able to use ggplot2 2.5.0 and project B will always be able to use ggplot2 3.1.0. This is important if we want to be able to reproduce results in the future.

On Linux, packrat stores compiled packages in packrat/lib/<LINUX_FLAVOR>/<R_VERSION>, an R-version-specific path, relative to the project’s base directory. An issue arises if we are using R version 3.5.0 one week and then upgrade to R 3.5.1 the next week: a packrat project will not find the 3.5.0 libraries anymore, and we will need to rebuild all the packages to install them in the 3.5.1 path. packrat will automatically build all packages from source (sources are stored in packrat/src) if it notices they are missing. However, this process can take tens of minutes, depending on the number of packages being built. Since this can be cumbersome when collaborating, we also opt to include the packrat/lib path in version control, thereby committing the compiled libraries as well.

Our solution is to bind one fixed R version to an instance³ and release fixed-R instance images periodically. We prefer limited, consistent R-versions over continually upgrading to the most recent version of R. This approach helps to ensure reproducibility and make collaboration easier, avoids having to use docker containers⁴. While binding a fixed version of R to an instance may seem restrictive, we have found that it is in fact quite liberating. Since we only update the existing R version infrequently (think once a year), the barrier of agreeing on an R-version is removed and with it any need to agree on package versions at the user level. Instead, packages are distributed with the project via git. The benefits of fixing the R version for a particular instance are:

Sharing packrat projects and reproducing results are both made easier, since pre-compiled libraries are included with the projects.
Fixing the R-version on an instance doesn’t keep us from upgrading R for a project, as packrat will automatically build and install libraries if an upgraded version is detected. In this way, a project can be opened on an instance with an upgraded R version and have its libraries compiled. Our limited instance image release schedule means the overhead to handle this only occurs at a maximum of once each year.
It is very unlikely that results will be different across R-versions, however being able to tie project results to one R-version allows us to upgrade R for a project while ensuring that results remain as expected.

What we lose by not being on the bleeding edge of (thankfully relatively non-critical) bug fixes we gain in ease of collaboration. Here’s what we’ve done to accomplish this:

rstudio-instance contains branches with scripts to set up a Linux instance with fixed R and RStudio versions. We git clone the repo and git checkout the branch suitable for the Linux flavor, R-version, and RStudio version we want. The scripts also ensure R is not auto-updated in the future.
We then run the install script to set up the instance and archive an image of it for future use.
Once the fixed-R instance is set up, rstudio-project contains an R-version specific base project with pre-built, packrat-managed, fixed-versions of many popular data-science packages⁵.
We git clone rstudio-project to a new project directory locally and remove the existing .git directory so that it can be turned it into a new git repo with git init.
We open the project in RStudio and begin work. All packages are pre-built, so we don’t have to go through lengthy installs. We can upgrade packages in the packrat library of the “Packages” tab, and then run packrat::snapshot() to save any libraries and ugrades into the project’s packrat/ directory. We can then git add packrat to add any packrat updates to the project’s git repo.
If we ever need to duplicate results, we can always build the same fixed-R instance (or clone the image we stored earlier), clone the project on the instance, and know that it will work exactly the same as when we previously worked on it… sometimes years earlier.

Here is a quick example script showing the workflow:

git clone git@github.com:ras44/rstudio-instance.git
cd rstudio-instance
git checkout centos7_R3.5.0_RSS1.1.453
./install.sh
sudo passwd <USERNAME> # set user password for RStudio Server login
cd
git clone git@github.com:ras44/rstudio-project.git new-project
cd new-project
git checkout dev-linux-centos7-R3.5.0
rm -rf .git
git init

Finally, here are some issues with packrat that we have run into along with our solutions. Note that RStudio support has been very helpful in addressing issues while monitoring and providing solutions via their github issue tracker.

If R crashes and the packrat libraries are not accessible after the RStudio restarts the session, the project might need to be re-opened. Run .libPaths() to ensure the project library paths are correct. Verify libraries are accessible by looking at the “packages” tab in RStudio Server and ensuring a “Project Library” header exists with all packages(see above image). Follow issue discussion.
An issue can arise when some packages are updated but others aren’t. This can be challenging to troubleshoot and raises the question of what to do when package versions become incompatible with each other. This is not packrat, but version compatibility.
Installing packages directly from a private/internal github is evolving. An easy solution exists: simply clone the package to a local directory such as ~/local_repos/. Then use install_local() to install from the local_repos directory. See issue for details.
packrat can occasionally have very slow snapshots, particularly with projects that contains many R-Markdown files and packages. This is likely due to packrat dependency searches. As discussed in the issue, we resolve it by ignoring all of our source directories with packrat::set_opts(ignored.directories=c("all","my","R","src","directories") and then running packrat::snapshot(ignore.stale=TRUE, infer.dependencies=FALSE).

Unless you somehow exclusively use packages that are never updated, never implement version-breaking/major version updates, or always provide backwards-compatible version upgrades. Many R packages are in major version 0, meaning there is no guarantee that a future release will maintain the same API. ↩
In the packrat/ directory ↩
It is possible to have multiple R versions installed on a system. I have avoided that for simplicity. ↩
docker containers may be a good alternate solution, but in this case we are not using them. ↩
rstudio-project contains all packages in the anaconda distribution and more. ↩

R and TensorFlow Presentations

Tue, 03 Apr 2018 00:00:00 +0000

In early March, the Bay Area useR Group was able to hold an R and TensorFlow mini-conference on Google’s new Sunnyvale campus. Pete Mohanty, a Stanford researcher and frequent BARUG speaker, lead off with a talk on his recent kerasformula package, which allows R users to call a keras-based neural net with R formula objects. Pete’s slides show an example of using using a regression-style formula with the kerasformula::kms() function to fit a sequential TensorFlow model.

J.J. Allaire, RStudio’s founder and CEO, spoke for over an hour, delivering a polished and comprehensive presentation that ranged from big-picture vistas illuminating the merits and limitations of the Deep Learning methodology to the deep details of R based TensorFlow models. We were not able to record J.J.’s BARUG talk, but it was similar to his presentation at the January RStudio conference in San Diego, which is well worth watching.

We do have the slides for J.J.’s BARUG presentation. They comprise a comprehensive overview of Deep Learning and the TensorFlow Technology behind it. While most slide presentations are little more than collections of mnemonics, lifeless without the animation of the speaker, J.J.’s 113-slide deck stands on its own. Replete with links to current research papers and references that guide a reader to the frontiers of Deep Learning applications, is a very credible introduction and study guide. Because 113 slide are quite a bit to get through in one sitting, especially if you start following interesting links, I offer the following gloss:

Slide	Topic
3	What is TensorFlow?
4	Why should R users care?
6 - 10	What are tensors?
11 - 13	What is the “flow”?
14 - 16	Applications with links to examples
17 - 26	What is Deep Learning, how does it work, and what is deep about it?
27	Statistical modeling vs. machine learning with links to seminal papers
28 - 36	A technical overview of what goes on in a Deep Learning model
37 - 44	Applications on the frontiers of Deep Learning with links to recent papers
45 - 47	A perspective on problems, hype and the usefulness of Deep Learning
48 - 73	The details of the R interface to Keras and TensorFlow, with links to the
	technical documentation and R packages and code
73 - 84	A tour of the models in RStudio’s TensorFlow Gallery
85 - 92	Tools for running on GPUs, managing experiments, running in the cloud, and deploying TensorFlow models
93 - 101	cloudml: an interface to Google CloudML
102 - 112	R-based technology for deploying TensorFlow models

A Data Science Lab for R

Wed, 20 Dec 2017 00:00:00 +0000

In a previous post I described the role of analytic administrator as a data scientist who: onboards new tools, deploys solutions, supports existing standards, and trains other data scientists. In this post I will describe how someone in that role might set up a data science lab for R.

Architecture

A data science lab is an environment for developing code and creating content. It should enhance the productivity of your data scientists and integrate with your existing systems. Your data science lab might live on your premises or in the cloud. It might be built with hardware, virtual machines, or containers. You may use it to support a single data scientist or hundreds of R developers. Here is one reference architecture of a data science lab based on server instances.

Key components of this setup include: authentication; load balancing; a testing environment; data connectivity; and a publishing platform. In this server-based architecture, data scientists use a web browser to access the data science lab. High performance compute and live data reside securely behind a firewall.

Instance Sizing

The size of your server instance depends on how many concurrent sessions you run and how large your sessions are. Keep in mind that R is single threaded by default and holds data in memory. Here is a list of example server sizes:

Instance Size	Cores	RAM	Description
Minimum recommended	2	4G	This server will be for lightweight jobs, testing, and sandboxing.
Small	4	8G	This server will support one or two analysts with small data.
Large	16	256G	This server will support 15 analysts with a blend of large and small sessions. Alternatively, it will support dozens of analysts with small sessions.
Jumbo	32+	1T+	May be useful for heavier workloads.

Open-Source R

If you haven’t done so already, I recommend you make R a legitimate part of your organization by officially recognizing it as an analytic standard. You should be familiar with installing and managing R and its packages.

You can install R as a pre-compiled binary from a repository, or you can install R from source. Installing R from source allows you to install multiple versions of R side by side. If you compile R from source, I recommend you link to the BLAS libraries so that you can speed up certain low-level math computations.

Data science labs tend to require a modern toolkit. You should expect to upgrade R at least once a year. You should also keep your operating system up to date. New and improved R packages tend to work better when you use them with recent versions of R and updated system libraries.

RStudio Server Pro

Building a data science lab involves installing, configuring, and managing tools. In this section I will describe how to administer RStudio Server Pro which has features for authentication, security, and admin controls.

1. Installation

Once you have installed R, you can install RStudio Server Pro by downloading the binaries and following the instructions. You will need root privileges to install and run the software. You will also need to create local system accounts for all of your R developers.

2. Configuration

Authentication. The first thing you will want to do after you install RStudio Server Pro is to configure it with your authentication system. RStudio Server Pro supports LDAP via PAM sessions. If you use single sign on or another system, you can configure RStudio Server Pro to work in proxied auth mode. You can also authenticate via Google accounts and local system accounts.

Data Connectivity. Most data scientists use R with databases. The RStudio Pro Drivers are ODBC drivers that will connect R to some of the most popular databases today. These drivers are a free add-on for RStudio Server Pro. If you are using a data source that is not supported, or if you are using the open source version of RStudio Server, you can bring your own ODBC driver.

Load Balancing. If you want to load balance your server instances, you can use the load balancer that is built into RStudio Server Pro or you can bring your own load balancer. Load balancing is designed to balance user sessions seamlessly across the cluster and provide high availability. It requires a shared home drive that is mounted to each one of the instances.

More Features. RStudio Server Pro has a list of features that you can configure. You should decide which features you want to enable or disable. For more information on configuring each of these features, see the admin guide.

Feature	Description
Authentication	LDAP, Active Directory, Google Accounts and system accounts Full support for Pluggable Authentication Modules, Kerberos via PAM, and custom authentication via proxied HTTP header
Data Connectivity	RStudio Professional Drivers are ODBC data connectors that help you connect to some of the most popular databases.
Load Balancing	Load balance R sessions across two or more servers Ensure high availability using multiple masters
Enhanced security	Encrypt traffic using SSL and restrict client IP addresses
Administrative dashboard	Monitor active sessions and their CPU and memory utilization Suspend, forcibly terminate, or assume control of any active session Review historical usage and server logs
Auditing and monitoring	Monitor server resources (CPU, memory, etc.) on both a per-user and system-wide basis Send metrics to external systems with the Graphite/Carbon plaintext protocol Health check with configurable output (custom XML, JSON) Audit all R console activity by writing input and output to a central location
Advanced R session management	Tailor the version of R, reserve CPU, prioritize scheduling and limit resources by User and Group Provision accounts and mount home directories dynamically via the PAM Session API Automatically execute per-user profile scripts for database and cluster connectivity
Project sharing	Share projects & edit code files simultaneously with others

3. Management

Once RStudio Server Pro is installed and configured, you’ll need to manage it over time. RStudio Server Pro comes with a variety of tools for workspace and server management that will help keep your environment organized. For example, you can kill sessions, set session timeouts, and broadcast notifications to user sessions in real-time. You can manage product licenses for both online and offline environments. If your instances start and stop frequently you can opt for using a floating license manager.

Next Steps

Your data science lab for R should be designed to scale. That might mean adding more people, more systems, or more tools. It also might mean creating more content. Shiny is an R package that makes it easy to build interactive web apps straight from R. R Markdown is an R package that makes it easy to author reports and build dashboards. You can publish your Shiny apps or R Markdown reports with the push of a button to RStudio Connect. RStudio Connect lets you share and manage content in one convenient place. You can also publish Shiny apps to shinyapps.io, which allows you to share your Shiny apps online.

Database Queries With R

Wed, 18 Oct 2017 00:00:00 +0000

There are many ways to query data with R. This post shows you three of the most common ways:

Using DBI
Using dplyr syntax
Using R Notebooks

Background

Several recent package improvements make it easier for you to use databases with R. The query examples below demonstrate some of the capabilities of these R packages.

DBI. The DBI specification has gone through many recent improvements. When working with databases, you should always use packages that are DBI-compliant.
dplyr & dbplyr. The dplyr package now has a generalized SQL backend for talking to databases, and the new dbplyr package translates R code into database-specific variants. As of this writing, SQL variants are supported for the following databases: Oracle, Microsoft SQL Server, PostgreSQL, Amazon Redshift, Apache Hive, and Apache Impala. More will follow over time.
odbc. The odbc R package provides a standard way for you to connect to any database as long as you have an ODBC driver installed. The odbc R package is DBI-compliant, and is recommended for ODBC connections.

RStudio also made recent improvements to its products so they work better with databases.

RStudio IDE (v1.1). With the latest version of the RStudio IDE, you can connect to, explore, and view data in a variety of databases. The IDE has a wizard for setting up new connections, and a tab for exploring established connections. These new features are extensible and will work with any R package that has a connections contract.
RStudio Professional Drivers. If you are using RStudio professional products, you can download RStudio Professional Drivers for no additional cost. The examples below use the Oracle ODBC driver. If you are using open-source tools, you can bring your own driver or use community packages – many open-source drivers and community packages exist for connecting to a variety of databases.

Using databases with R is a broad subject and there is more work to be done. An earlier blog post discussed our vision. Part of that vision was to create a website where you can find everything about databases and R in one place. To learn more, visit our site at db.rstudio.com.

Example: Query bank data in an Oracle database

In this example, we will query bank data in an Oracle database. We connect to the database by using the DBI and odbc packages. This specific connection requires a database driver and a data source name (DSN) that have both been configured by the system administrator. Your connection might use another method.

library(DBI)
library(dplyr)
library(dbplyr)
library(odbc)
con <- dbConnect(odbc::odbc(), "Oracle DB")

1. Query using `DBI`

You can query your data with DBI by using the dbGetQuery() function. Simply paste your SQL code into the R function as a quoted string. This method is sometimes referred to as pass through SQL code, and is probably the simplest way to query your data. Care should be used to escape your quotes as needed. For example, 'yes' is written as \'yes\'.

dbGetQuery(con,'
  select "month_idx", "year", "month",
  sum(case when "term_deposit" = \'yes\' then 1.0 else 0.0 end) as subscribe,
  count(*) as total
  from "bank"
  group by "month_idx", "year", "month"
')

2. Query using dplyr syntax

You can write your code in dplyr syntax, and dplyr will translate your code into SQL. There are several benefits to writing queries in dplyr syntax: you can keep the same consistent language both for R objects and database tables, no knowledge of SQL or the specific SQL variant is required, and you can take advantage of the fact that dplyr uses lazy evaluation. dplyr syntax is easy to read, but you can always inspect the SQL translation with the show_query() function.

q1 <- tbl(con, "bank") %>%
  group_by(month_idx, year, month) %>%
  summarise(
    subscribe = sum(ifelse(term_deposit == "yes", 1, 0)),
    total = n())
show_query(q1)

<SQL>
SELECT "month_idx", "year", "month", SUM(CASE WHEN ("term_deposit" = 'yes') THEN (1.0) ELSE (0.0) END) AS "subscribe", COUNT(*) AS "total"
FROM ("bank") 
GROUP BY "month_idx", "year", "month"

3. Query using an R Notebooks

Did you know that you can run SQL code in an R Notebook code chunk? To use SQL, open an R Notebook in the RStudio IDE under the File > New File menu. Start a new code chunk with {sql}, and specify your connection with the connection=con code chunk option. If you want to send the query output to an R dataframe, use output.var = "mydataframe" in the code chunk options. When you specify output.var, you will be able to use the output in subsequent R code chunks. In this example, we use the output in ggplot2.

```{sql, connection=con, output.var = "mydataframe"}
SELECT "month_idx", "year", "month", SUM(CASE WHEN ("term_deposit" = 'yes') THEN (1.0) ELSE (0.0) END) AS "subscribe",
COUNT(*) AS "total"
FROM ("bank") 
GROUP BY "month_idx", "year", "month"
```

```{r}
library(ggplot2)
ggplot(mydataframe, aes(total, subscribe, color = year)) +
  geom_point() +
  xlab("Total contacts") +
  ylab("Term Deposit Subscriptions") +
  ggtitle("Contact volume")
```

The benefits to using SQL in a code chunk are that you can paste your SQL code without any modification. For example, you do not have to escape quotes. If you are using the proverbial spaghetti code that is hundreds of lines long, then a SQL code chunk might be a good option. Another benefit is that the SQL code in a code chunk is highlighted, making it very easy to read. For more information on SQL engines, see this page on knitr language engines.

Summary

There is no single best way to query data with R. You have many methods to chose from, and each has its advantages. Here are some of the advantages using the methods described in this article.

Method	Advantages
DBI::dbGetQuery	Fewer dependencies required
dplyr syntax	Use the same syntax for R and database objects No knowledge of SQL required Code is standard across SQL variants Lazy evaluation
R Notebook SQL engine	Copy and paste SQL – no formatting required SQL syntax is highlighted

You can download the R Notebook for these examples here.

RStudio::Conf 2018

Tue, 29 Aug 2017 00:00:00 +0000

It’s not even Labor Day, so it seems to be a bit early to start planning for next year’s R conferences. But, early-bird pricing for RStudio::Conf 2018 ends this Thursday.

The conference which will be held in San Diego between January 31st and February 3rd promises to match and even surpass this year’s event. In addition to keynotes from Di Cook (Monash University and Iowa State University), J.J. Allaire (RStudio Founder, CEO & Principal Developer), Shiny creator Joe Cheng, and Chief Scientist Hadley Wickham, a number of knowledgeable (and entertaining) speakers have already committed including quant, long-time R user and twitter humorist JD Long (@CMastication), Stack Overflow’s David Robinson (@drob) and ProPublica editor Olga Pierce (@olgapierce).

Making the deadline for early-bird pricing will get you a significant savings. The “full seat” price for the conference is $495 before midnight EST August 31, 2017 and $695 thereafter. You can register here.

For a good idea of the kinds of things you can expect to learn at RStudio::Conf 2018 have a look at the videos from this year’s event.

Some Ideas for your Internal R Package

Wed, 19 Jul 2017 00:00:00 +0000

At RStudio, I have the pleasure of interacting with data science teams around the world. Many of these teams are led by R users stepping into the role of analytic admins. These users are responsible for supporting and growing the R user base in their organization and often lead internal R user groups.

One of the most successful strategies to support a corporate R user group is the creation of an internal R package. This article outlines some common features and functions shared in internal packages. Creating an R package is easier than you might expect. A good place to start is this webinar on package creation.

Logos and Custom CSS

Interestingly, one powerful way to increase the adoption of data science outputs - plots, reports, and even slides - is to stick to consistent branding. Having a common look and feel makes it easier for management to recognize the work of the data science team, especially as the team grows. Consistent branding also saves the R user time that would normally be spent picking fonts and color schemes.

It is easy to include logos and custom CSS inside of an R package, and to write wrapper functions that copy the assets from the package to a user’s local working directory. For example, this wrapper function adds a logo from the RStudioInternal package to the working directory:

getLogo <- function(copy_to = getwd()){
      copy_to <- normalizePath(copy_to)
      file.copy(system.file(“logo.png”, package = “RStudioInternal”) , copy_to)
}

Once available, logos and CSS can be added to Shiny apps and R Markdown documents.

ggplot2 Themes

Similar to logos and custom CSS, many internal R packages include a custom ggplot2 theme. These themes ensure consistency across plots in an organization, making data science outputs easier to recognize and read.

ggplot2 themes are shared as functions. To get started writing a ggplot2 theme, see resource 1. For inspiration, take a look at the ggthemes package.

Data Connections

Internal R packages are also an effective way to share functions that make it easy for analysts to connect to internal data sources. Nothing is more frustrating for a first time R user than trying to navigate the world of ODBC connections and complex database schemas before they can get started with data relevant to their day-to-day job.

If you’re not sure where to begin, look through your own scripts for common database connection strings or configurations. db.rstudio.com can provide more information on how to handle credentials, drivers, and config files.

learnr Tutorials

RStudio recently released a new package for creating interactive tutorials in R Markdown called learnr. There are many great resources online for getting started with R, but it can be useful to create tutorials specific to your internal data and domain. learnr tutorials can serve as training wheels for the other components of the internal R package or teach broader concepts and standards accepted across the organization. For example, you might provide a primer that teaches new users your organization’s R style guide.

Wrap Up

If you are leading an R group, an internal R package is a powerful way to support your users and the adoption of R. Imagine how easy it would be to introduce R to co-workers if they could connect to real, internal data and create a useful, beautiful plot in under 10 minutes. Investing in an internal R packages makes that on boarding experience possible.

Printing From Flex Dashboard

Wed, 28 Jun 2017 00:00:00 +0000

Shiny applications of all stripes (including flexdashboard with runtime Shiny) are revolutionary in that they put the power of R directly in the end user’s hands without needing to interact directly with the language. A common way end-users wish to interact with their data is via a dashboard that they can manipulate on the fly. Flexdashboard streamlines the process of turning an R-based analysis into a dashboard, so R users can create good-looking output for their analyses - and deploy this output to the web - with very little additional effort.

There are various ways to extract information from flexdashboard, but no good way to programatically return a printable version of the dashboard itself. This is partially by design - responsive HTML dashboards aren’t meant to be printed. There are well-documented workflows about how to share information between a web-centric output and a complementary report that renders to a printable format.

That said, there are some reasons to want a printable output that looks similar to a related HTML report. Snapshots are valuable for documentation/training materials, for embedding in emails or other reports, or for the all-too-common case of meetings with executives who like dashboards but want to consume paper-based reports. Plus, flexdashboards are an incredibly quick way to create visually appealing (read: pretty) summary output of a combination of outputs on a single page.

Both Harrison and Aaron¹ have struggled with trying to print flexdashboards in their respective practices. In a recent instance, Harrison was working with a client whose end users would be carrying snapshots of the dashboard into an industrial setting where mobile devices were not desired. It was important to the client to maintain a similar look between the on-screen and paper representations of the dashboard. It was also important to make generating these snapshots an integrated part of the application; it was not sufficient to suggest that the client manually take a screenshot of their browser window. This turned out to be a blessing, because (as you will see below) it allows the creation of a web-friendly ‘tabbed’ report with a companion ‘flat’ printed rendering.

Specific Task

Our goal was to create a “printable” flexdashboard with runtime Shiny. Flexdashboard produces a beautiful HTML dashboard with very little effort. Like all R Markdown documents, it also allows Shiny objects to be embedded inside of it, so that when rendered locally or on a server, it produces a reactive web application with minimal developer effort. Like many HTML files, however, it does not print well.

The flexdashboard in question also had tabsets, which make perfect sense on a computer or mobile device, but don’t translate well to printed media. When printed, tabsets communicate that there is pertinent information hidden from the viewer; better to remove the tabset in the printed version. If the information is pertinent, it should be displayed in the printed document; if it is not, it should be removed.

PhantomJS and Headless Chrome

The approach we settled on as good enough and easy enough was to use a headless browser to take a screenshot of a non-reactive HTML document. It’s not a perfect solution: a PDF would be better than a png file, headless browsers add a code dependency, and there is some administrative overhead to create the document to print. It is, however, quick to implement, easy to understand, and robust.

Example²

The mechanics of building a screen-capture-style report from flexdashboard are neither difficult nor obvious. In general, it requires the following steps:

Add a downloadButton to a flexdashboard with runtime Shiny
When the download button is invoked, temporarily save non-reactive copies of any objects of interest
Knit a static HTML version of the flexdashboard, using the objects saved in the prior step
Use webshot::webshot or decapitated::chrome_shot to capture a .png image of the static dashboard³
Deliver this version to the client using the downloadHandler function

The solution requires two separate R documents. The process is as follows:

Dynamic Flexdashboard document


---
title: "Snapshot Example"
runtime: shiny
output:
  flexdashboard::flex_dashboard:
    orientation: rows
    vertical_layout: scroll
---

This is standard YAML header for a flexdashboard, including the command runtime: shiny to create a Shiny application from an R Markdown document. Let’s examine the next chunks:

gears = mtcars$gear %>% as.factor() %>% levels()
selectizeInput("grs", "GEARS",
               gears, selected = gears[1])
renderUI({
downloadButton("downloadFile", "Download")
})

In this chunk, we have built a selectizeInput object to choose the number of gears to select, as well as a downloadButton named ‘downloadFile’. Notice that we have wrapped the button in a renderUI() context. This is standard shiny / flexdashboard code.

mpgp = reactive({
  mtcars %>% filter(gear %in% input$grs) %>% ggplot(aes(x = hp, y = mpg)) + geom_point() + geom_smooth()
  
})
renderPlot({
  mpgp()
})

For veteran shiny programmers, this block may seem redundant. We’re creating a plot called mpgp in a reactive context and then rendering it. In a ‘standard’ shiny implementation, we would simply have wrapped the plot code in the renderPlot() context. However, as you will see below, we’re going to need to reuse the object mpgp.

output$downloadFile <- downloadHandler(filename = function() { 
return(paste('Cars', '.png', sep=''))
},
    content = function(file){
    to_save <- list(
    mpgg = mpgg(),
    mpgp = mpgp()
    )
      saveRDS(to_save, "config_data.RDS")
      rmarkdown::render("ShadowCars.Rmd")
      webshot::webshot("ShadowCars.html", file = file)
                        })

This chunk contains the secret sauce. When the downloadButton is invoked, the program saves all elements to appear in the downloaded report in a file called ‘config_data.RDS’. These elements are then used as arguments to render a shadow document, ‘ShadowCars.Rmd’. Finally, webshot::webshot converts the hidden document to a .png and delivers it to the client side.

The full dashboard looks like this:

Shadow Document

The key elements of the shadow document code are presented below:

---
title: "Snapshot Example"
output:
  flexdashboard::flex_dashboard:
    orientation: rows
---

Note that our YAML for the shadow document is similar to the dashboard, except it does not include runtime: shiny

The code blocks that render the objects created in the dynamic document are straightforward to the point of triviality; they simply reference the saved objects (which are retrieved using readRDS) and render them using whatever we choose, in this case kable() for the table and the native print() for the plot.

data <- readRDS("config_data.RDS")
data$mpgg %>% kable()

data$mpgp %>% print()

And of course, while mpgp was reactive in the original dashboard, it is static in the shadow doc.

The report delivered to the client is as follows:

Conclusion

In this short post, we have shown how to create a downloadable custom report by using existing tools to create a static flexdashboard. From the .png environment, the rendered image can be converted to other formats as required.

Harrison Schramm is an Operations Research Analyst at CANA Advisors. Aaron Berg is a Customer Success Representative at RStudio.↩
To recreate this minimal example for yourself, please download the files from this gist.↩
The webshot package requires phantomJS to be installed as an external dependency, but will work on all systems. decapitated will only work on systems with versions of Chrome that can operate in headless mode.↩

What is the tidyverse?

Thu, 08 Jun 2017 00:00:00 +0000

Last week, I had the opportunity to talk to a group of Master’s level Statistics and Business Analytics students at Cal State East Bay about R and Data Science. Many in my audience were adult students coming back to school with job experience writing code in Java, Python and SAS. It was a pretty sophisticated crowd, but not surprisingly, their R skills were stitched together in a way that left some big gaps. Many for example, didn’t fully understand the importance of CRAN Task Views as curated source for the best packages to support their work in machine learning, time series and the other areas of Statistics they were studying. So, it made sense that even though ggplot2 and dplyr were mentioned in some of the student’s questions, a faculty member present asked: “What is the tidyverse?” in an attempt to cover an area that he knew was one of those gaps.

There is an incredible amount of good material available online about the tidyverse, and I will point to some of that below. But here, I’ll elaborate on the answer I gave during the Q&A.

The Basics

The tidyverse is a coherent system of packages for data manipulation, exploration and visualization that share a common design philosophy. These were mostly developed by Hadley Wickham himself, but they are now being expanded by several contributors. Tidyverse packages are intended to make statisticians and data scientists more productive by guiding them through workflows that facilitate communication, and result in reproducible work products. Fundamentally, the tidyverse is about the connections between the tools that make the workflow possible.

It is also the case that the tidyverse is work in progress. You can find the current state of development at tidyverse.org. Clicking on the icon for each package on this website will bring you to detailed documentation for each package. The following figure illustrates a canonical data science workflow, and shows how the individual packages fit in.

If you have some experience with R, you ought to be able to jump right into the online documentation and find your way around. If you are new to R, and maybe new to data science as well, you can’t do any better than work through the book R for Data Science by Hadley Wickham and Garrett Grolemund.

Advantages

The advantages of the tidyverse include consistent functions, workflow coverage, a path to data science education, a parsimonious approach to the development of data science tools, and the possibility of greater productivity.

Consistency

The tidyverse aspires to consistency on multiple levels. Examples of “micro”-level consistency include the convention of having variable names glide along in snake_case, and the signatures of tidyverse functions follow a regular pattern. (The first formal argument is always a data frame that provides the function’s input.) Higher-level consistency includes the idea of tidy data - a data frame where each row is an observation and each column contains the value of a single variable - and the way in which the pipe operator, %>%, channels the flow of tidy operations. Under the covers, there are even more levels of structure that aid the pursuit of consistency, including uniform standards for package organization, testing procedures, coding style, etc.

Coverage

The workflow shown above, with tidyverse packages associated with the various steps, or more usually rendered with the following iconic tidyverse diagram, preceded and motivated the development of the tidyverse.

It is an abstraction of the canonical data analysis workflow that has always guided statisticians, but now informs data science as a map to organize, streamline, automate and optimize the various processes involved. The fact that tidyverse packages are associate with all of the processes indicates that it comprises enough fundamental building blocks to support the entire end-to-end workflow for a variety of data sources and analysis goals. Moreover, the relatively recent addition of the purrr package extends the reach of the tidyverse to support the creation of new data science tools.

Critical Mass

A great strength of the R language is that with over ten thousand user contributed packages on CRAN alone, it has a lot to offer. This kind of organic growth makes it inevitable that packages will offer overlapping features. Users have to make decisions about which package, or suite of packages, they will make the effort to learn. For many users, the decision hinges on whether a collection of packages visibly supports important work. Does it have a large community of users and is it backed by committed developers and maintainers? All of the signals indicate that (at least, among R-using data scientists) the tidyverse has reached critical mass. For example, the tidyverse package has been downloaded 50,000 times in the last month. Moreover, it appears that tidyverse principles are propagating into other application areas. The tidyquant package, for example, is a serious attempt to bring tidy principles to Finance.

Education

A typical R user gets involved with R in the first place through a desire to compute in some quantitative field. The path to R competency frequently begins with mastering a small number of relevant functions. Statisticians, for example, may learn to read in data from a .csv file and build a linear regression model with lm(). Financial analysts may be introduced to R through a package like quantmod, which enables a new user to do quite a bit of real work. The tidyverse provides the path of least resistance, or “pit of success”, for data scientists interested in R. For example, the small number of compatible building blocks provided by dplyr enable even a relatively inexperienced user to tidy up a messy data set quickly and easily.

Parsimony

The packages and functions of the tidyverse are the result of trial-and-error experimentation carried out over several years, to find a minimum set of functions that are sufficient to enable the canonical data science workflow. Those of you who have been following Hadley’s work will remember cast() and melt() from the reshape and reshape2 packages, and ddply() from the plyr package, which were early attempts to find a vocabulary for wrangling data frames. After several attempts to identify and construct the most advantages set of primitive building blocks, the tidyverse has matured into its present form.

Productivity

Hadley has always been clear that a major goal for the tidyverse - and indeed much of his work over the years - has been to help anyone who needs to analyze data work productively, and he is fond of quoting Hal Abelson: “Programs must be written for people to read and only incidentally for machines to execute”. My take is that a major reason for the popularity of tidyverse packages is that they help people achieve and maintain flow in their daily data analysis work.

Some Limitations

The tidyverse, of course, is not without limitations. Some of these are due to factors that are beyond the designer’s control, and others may be by design. Limitations of the first kind may arise from a lack of agreement as to whether some data can be, or should be, forced into a “rectangular” data structure. For example, although there are scientists and data scientists working in genomics that are fans of dplyr and ggplot2, much of the work done in the Bioconductor Project remains outside of the tidyverse workflow.

The need for the close coordination of tidyverse packages produces some limitations of the second sort. There are many high-quality R packages that are of great use to data scientists, but based on design goals that differ from those of the tidyverse. There will always be more than the tidyverse.

A Bigger Picture

A powerful, but perhaps under-appreciated, capability of the R language is its ability to support the design and programming of Domain Specific Languages. Joe Cheng highlighted this feature in an interview he gave to R Views last year. He described R as being “shockingly close to LISP”, of which Joe says: “it’s almost like you change the language itself to be a DSL for whatever problem you’re trying to solve … the elegant, terse syntax of dplyr and the pipe operator are possible because of how malleable a language R is, and how great it is for writing DSLs in it.”

So, from a wider perspective, the tidyverse can be seen as sub-dialect of the R language that is evolving to express ideas and tasks inherent in Data Science workflows and software development. This dialect may not be for everyone, but it does seem to be helping many R fluent data scientists frame their conversations.

Some Resources

The following are some resources that you may find helpful in learning and mastering the tidyverse.

The video of Hadley Wickham’s Keynote address at rstudio::conf 2017
The slides corresponding to the above video
R for Data Science by Hadley Wickham and Garrett Grolemund
Text Mining with R
Getting Started with the Tidyverse in R

Mapping Quandl Macroeconomic Data

Wed, 10 May 2017 00:00:00 +0000

In previous posts, we built a map to access global ETFs and a simple Shiny app to import and forecast commodities data from Quandl. Today, we will begin a project that combines those previous apps. Our end goal is to build an interactive map to access macroeconomic data via Quandl, allowing the user to choose an economic indicator and click on a country to access that indicator’s time series. For example, if a user wants to import and graph GDP-per-capita data from China, that user will be able to select GDP per Capita from a drop-down menu, and then click on China to display the time series. As usual, we’ll start with a Notebook for data import, test visualizations, and map building.

Careful readers will note that we are not simply reusing code from those previous projects; that is because we are going to introduce some new packages to our toolkit. Specifically:

Instead of downloading a spatial data frame from Natural Earth, we will use the new R package rnaturalearth and it’s function ne_countries() to import our geographic data.
Relatedly, we are not going work with a “spatial” data frame, but rather with a simple features data frame of class sf. Learn more here.
Finally, when we test our data import of several macroeconomic data sets, we will use the map() function from the purrr package instead of relying on lapply(), as we did previously when passing ticker symbols to getSymbols(). If you’re like me, you won’t be sad to be done with lapply().

Let’s get to it!

We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators (WDI). The Quandl code for WDI is WWDI, and thus we’ll prepend WWDI/ to each data set call.

One difference from our earlier work with Quandl is that previously we imported and visualized global time series, such as the price of oil, gold or copper. By global, I mean these time series were not associated with a particular country. That simplified our job a bit because we did not need to supply a country code to Quandl. Today, we do need to include a country code.

Let’s start with a simple example and import GDP-per-capita data for China, whose country code is CHN. We will pass the string WWDI/CHN_NY_GDP_PCAP_KN to Quandl, which consists of the WDI code WWDI/, appended by the China country code CHN, appended by the GDP-per-capita code _NY_GDP_PCAP_KN.

library(Quandl)
# Pass the code string to Quandl. 
China_GDPPC <- Quandl("WWDI/CHN_NY_GDP_PCAP_KN", type = 'xts') %>% 
  # Add a nice column name
  `colnames<-`("GDP Per Capita")

# Take a look at the result.
tail(China_GDPPC, n = 6)

##            GDP Per Capita
## 2010-12-31       30876.04
## 2011-12-31       33658.85
## 2012-12-31       36126.73
## 2013-12-31       38737.58
## 2014-12-31       41354.61
## 2015-12-31       43991.55

That looks pretty good, but note that the data only goes back to December of 2015 - not ideal, but fine for our illustrative purposes.

Our eventual Shiny app, though, will give the user a choice among many economic time series, and we want to test the import and visualization of all those choices. In addition to GDP-per-capita, let’s test GDP-per-capita growth, real interest rate, exchange rate, CPI, and labor force participation rate. Those offer a nice snapshot of a country’s economy.

To run our test in the code chunk below, we will first build a vector of data set codes. Note that each of them start with WWDI/, then CHN, and then the relevant code. Next, we will pipe that vector to Quandl using the map() function. map() takes a function, in this case the Quandl() function, and applies it to the elements in a vector, in this case our vector of data set codes. We also want to specify type = "xts".

If we stopped our pipe there, the output would be a list of six time series and that would suit our testing purposes, but I don’t love having to look at a list of six. Thus, we’ll keep on piping and use the reduce() function from the purrr package and call merge() to combine the six lists into one xts object. That is much easier to deal with, and easier to pass to dygraphs for our test visualization. The last piped function will invoke colnames<- to clean up the column names.

library(purrr)
# Create a vector of economic indicators that can be passed to Quandl via map().
# Include names and values for easy naming of xts columns.
econIndicators <- c("GDP Per Capita" = "WWDI/CHN_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "WWDI/CHN_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "WWDI/CHN_FR_INR_RINR",
                  "Exchange Rate" = "WWDI/CHN_PX_REX_REER",
                  "CPI" = "WWDI/CHN_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "WWDI/CHN_SL_TLF_ACTI_ZS")

# You might want to supply your Quandl api key. It's free to get one.
Quandl.api_key("d9EidiiDWoFESfdk5nPy")

China_all_indicators <- 
  # Start with the vector of Quandl codes
  econIndicators %>% 
  # Pass them to Quandl via map(). 
  map(Quandl, type = "xts") %>% 
  # Use the reduce() function to combine them into one xts objects.
  reduce(merge) %>% 
  # Use the names from the original vector to set nicer column names.
  `colnames<-`(names(econIndicators))
# Have a look.
tail(China_all_indicators, n = 6)

##            GDP Per Capita GDP Per Capita Growth Real Interest Rate
## 2011-12-31       33658.85              9.012854          -1.472149
## 2012-12-31       36126.73              7.332031           3.523236
## 2013-12-31       38737.58              7.226936           3.692594
## 2014-12-31       41354.61              6.755778           4.732429
## 2015-12-31       43991.55              6.376423           4.270386
## 2016-12-31             NA                    NA                 NA
##            Exchange Rate      CPI Labor Force Part. Rate
## 2011-12-31      102.6904 5.410850                   76.7
## 2012-12-31      108.4435 2.624921                   77.0
## 2013-12-31      115.2980 2.627119                   77.3
## 2014-12-31      118.9863 1.996847                   77.6
## 2015-12-31      131.6298 1.442555                     NA
## 2016-12-31      124.1864 2.007556                     NA

That seems to have done the job, and note that both CPI and Exchange have data up to December of 2016. Much better, and a sign that there will be date variation among countries and indicators. That won’t be crucial today, but it could be if we needed consistent dates for all of our data sets.

As a final step, let’s use the dygraph() function to visualize these time series to make sure things work properly before moving to Shiny.

library(dygraphs)

## Warning: package 'dygraphs' was built under R version 3.3.2

dygraph(China_all_indicators$`GDP Per Capita`, main = "GDP Per Capita")

dygraph(China_all_indicators$`GDP Per Capita Growth`, main = "GDP Per Capita Growth")

dygraph(China_all_indicators$`Real Interest Rate`, main = "Real Interest Rate")

dygraph(China_all_indicators$`Exchange Rate`, main = "Exchange Rate")

dygraph(China_all_indicators$`CPI`, main = "CPI")

dygraph(China_all_indicators$`Labor Force Part. Rate`, main = "Labor Force Part. Rate")

GDP-per-capita has been on a steady march higher, labor force participation plummeted in the 1990’s and 2000’s, probably as the labor market became more market-oriented, and real interest rates look quite choppy. Plenty to learn, and it signals that our Shiny app might be a nice exploratory tool.

For now, it’s on to map-building!

In previous posts, we have started map-building by grabbing a spatial object from the internet. Today, we will use the new rnaturalearth package and its fantastic function ne_countries(). We will specify type = "countries" and returnclass = 'sf'. We could have set returnclass = 'sp' if we wanted to work with a spatial data frame, but I have been migrating over to the sf package and simple features data frames for mapping projects. Lots to learn here, but in general, it seems to use more tidy concepts than the sp package.

library(rnaturalearth)
library(sp)
world <- ne_countries(type = "countries",  returnclass = 'sf')

# Take a peek at the name, gdp_md_est column and economy columns. 
# The same way we would peek at any data frame.
head(world[c('name', 'gdp_md_est', 'economy')], n = 6)

## Simple feature collection with 6 features and 3 fields
## geometry type:  MULTIPOLYGON
## dimension:      XY
## bbox:           xmin: -73.41544 ymin: -55.25 xmax: 75.15803 ymax: 42.68825
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
##                   name gdp_md_est                   economy
## 0          Afghanistan      22270 7. Least developed region
## 1               Angola     110300 7. Least developed region
## 2              Albania      21810      6. Developing region
## 3 United Arab Emirates     184300      6. Developing region
## 4            Argentina     573900   5. Emerging region: G20
## 5              Armenia      18770      6. Developing region
##                         geometry
## 0 MULTIPOLYGON(((61.210817091...
## 1 MULTIPOLYGON(((16.326528354...
## 2 MULTIPOLYGON(((20.590247430...
## 3 MULTIPOLYGON(((51.579518670...
## 4 MULTIPOLYGON(((-65.5 -55.2,...
## 5 MULTIPOLYGON(((43.582745802...

This output looks similar to an sp object, but it’s not identical. In particular, note the geometry column, which contains the multipolygon latitude and longitude coordinates. It’s a matter of personal preference, but I find this sf object more intuitive in terms of how the geometry corresponds with the rest of the data.

Now we’ll create our shading and popups. Note that this code is identical to when we were working with spatial data frames.

library(leaflet)
# Create shading by GDP. Let's go with purples.
gdpPal <- colorQuantile("Purples", world$gdp_md_est, n = 20)

# Create popup country name and income group so that something happens
# when a user clicks the map.
popup <- paste0("<strong>Country: </strong>", 
                world$name, 
                "<br><strong>Market Stage: </strong>", 
                world$income_grp)

Now, it’s time to build our leaflet map. This code is nearly identical to our code from previous projects, but with one major change. As we saw above, we need to pass in a country code to Quandl, not a country name and not a ticker symbol. Thus, we don’t want to set layerId = name, but we also don’t need to add anything special to our world object. It already contains a column called iso_a3, and that column contains the country codes used by Quandl. Fantastic!

We just need to set layerId = ~iso_a3 and we will be able to access country codes in our Shiny app when a user clicks the map. This is quite fortunate, but it also highlights a good reason to spend time studying the world object. It might contain other data that will be useful in future projects.

leaf_world <- leaflet(world) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7, 
      
      # Note the layer ID. Not a country name! It's a country code! 
      
      color = ~gdpPal(gdp_md_est), layerId = ~iso_a3, popup = popup)

leaf_world

Because Quandl uses the ISO 3-letter code to identify a country, and because the rnaturalearth sf object already contains a column with iso_a3 country codes, building our map wasn’t very hard. It’s worth pointing out that in various projects, we have now built maps with three different layerIDs. In the Global ETF Map project, we built one map with layerID = name, using the built-in name column in the rnaturalearth object, and we built another map with layerID = tickers, using a column of tickers that we added ourselves with the merge() function. Now, today, we have a map with layerID = iso_a3, using the built-in column of country codes. We are building up a nice portfolio of maps with different layerIDs should we ever want or need them in the future.

That’s all for today. We’ll save that leaflet_world object for use in our Shiny app! See you next time.

Shiny in Medicine

Wed, 03 May 2017 00:00:00 +0000

Shiny Apps are becoming ubiquitous as a way for data scientists to present the results of an analysis, and also to engage with information consumers who may not be coders. The trend I see is that the greater the variety of skills and interests of the information consumers for any particular project, the more valued are interactive visualizations that can be integrated into enterprise-wide communication workflows. So, it is not surprising to see Shiny apps popping up in all manner of healthcare and medical applications. If data scientists are going to bring predictive analytics into clinical workflows where doctors, nurses, scientists, technicians and administrators are all part of near real-time decision processes, they are going to have to be even more inventive in providing these multi-skilled teams with low-friction tools to ingest and manipulate information. Below are few interesting Shiny apps that are broadly related to Health Care and Medicine. My guess is that interactive visualizations like these will improve research and clinical workflows, and eventually change how all of us look at Health Care and Medicine.

The Genome viewer for ICGC cancer, built by the folks at Aridhia, is geared towards researchers. You can learn how to interpret the plot and learn the story behind its creation here.

The Emergency Department Simulation, built by a group of mathematicians and physicians, models patient flow information under different assumptions about emergency department case loads, and illustrates how predictive analytics and statistical analysis can be integrated into operational clinical workflows.

The Colorectal Cancer risk calculator from the Cleveland Clinic targets physicians and the general public to personalize the risk of this disease. I found working through different “what if” scenarios of great help in thinking about the risk factors that are under my control.

Finally, these two Shiny Apps that provide information about US hospitals should be of interest to public health planners, as well as to the general public. The Hospital Ranking App compares hospital outcomes for heart attack, heart failure, and pneumonia against national statistics.

The Access to Hospital Care Dashboard plots the density of hospitals throughout the United States, and indicates under-served areas.

If you or your team are doing this kind of work, we here at R Views would love to hear about it.

A Shiny App for Importing and Forecasting Commodities Prices from Quandl

Fri, 21 Apr 2017 00:00:00 +0000

In a previous post, we imported oil data from Quandl and applied a simple model to it. Today, we’ll port that work over to a Shiny app (by way of flexdashboard, of course) that allows a user to choose a commodity (oil, copper or gold), choose a frequency for the time series, and choose how many periods ahead to forecast. The app will display the price history and the forecasted price. To see it in action, have a look here.

As with the previous Notebook, the main purpose of this post is to build a template where a more sophisticated or proprietary model could be used for forecasting. By setting it up so that models could be selected as inputs, several different models could be included. We are not as concerned with the modeling as much as we are with providing a format that is friendly to both end users and any future collaborator that might want to take this app and expand upon it.

Let’s get to it! In the code chunk below, we are immediately faced with a few important decisions. The first of those is the format of the user input. At the extreme, we could use textInput and simply allow the user to enter the code for the desired data set. The benefit is that we would not limit the user in any way; he or she could choose any dataset on Quandl. The cost would be that the user would need to know - or go to Quandl and look up - the code for any data set.

For example, to import WTI oil prices, the user would have to type in “FRED/DCOILWTICO”. That’s no problem if most of the end users know that code, but it’s a big problem otherwise. We want to emphasize convenience and broad usability, so we are going with selectInput instead of textInput, meaning our app will show a drop-down of a few choices. The user just selects “WTI oil” instead of typing “FRED/DCOILWTICO”“, or selects”copper" instead of typing “ODA/PCOPP_USD”. But, if a user wants to work with a data set that we haven’t included, said user is out of luck.

Another big decision is how many choices to give the user. I have included only 3: oil, gold and copper. In industry, you would probably include several more, perhaps all of the industrial metals, but there’s is a cutoff somewhere. Or maybe we prefer one choice, because this app is just for oil analysis. Either way, the number of options in the drop-down menu is another trade-off between usability and flexibility.

The final decision is a bit more nuanced, and requires looking ahead to how these inputs will be used further down in the app. Have a peek at the object called dataChoices, and you might notice that we don’t strictly need that object. We could have put the vector of choices as an argument to selectInput, so that our code would have read choices = c("WTI oil" = "FRED/DCOILWTICO", ...) instead of choices = dataChoices. In that choice assignment, “WTI oil” is called the name and “FRED/DCOILWTICO” is called the value (together we can think of them as a name-value pair). The entire reason for building a separate dataChoices object is that we want the ability to extract either the name or the value of the name-value pair. Usually we would care only about the value, because we want to pass the value to Quandl and import the data, but that name is going to be useful as well when we label our graph.

The ability to extract names and values will become even more useful when we get to the frequency of the time series and forecasting. For now, let’s look at dataChoices and selectInput.

# Notice a tradeoff here: we're making it easy on our users because they don't need to 
# remember the naming conventions. But, we're also forced to severely limit their choices. 
# The dataChoices object is going to allow us to add a nicer label to the graph. 
# Notice also how easily we can include datasets from different sources, and not worry about
# their formats. Thanks, Quandl!

dataChoices <- c("WTI oil" = "FRED/DCOILWTICO", # oil data from Fred
                 "Copper" = "ODA/PCOPP_USD", # copper data from ODA
                 "Gold" = "CHRIS/CME_GC1") # gold data from CME

selectInput("dataSet",
            "Commodity",
            choices = dataChoices,
            selected = "WTI oil")

Alright, we have given the user the ability to choose a commodity. Next we want to ask about the frequency, and this gets a bit more complicated. We need to tell Quandl the frequency of the time series to import using the “daily”, “weekly”, “monthly” convention, so we set those to the values. But, further down the app when we want to store the forecast results, we’ll need to use the “days”, “weeks”, “months” phrasing and we’ll need to pull out the names from the name-value pair in frequencyChoices. We knew that this would be necessary because when we built our Notebook for importing, wrangling, and testing, we noted the different conventions and started thinking about how to deal with them in the Shiny context.

# The frequencyChoices object is going to allow us to pass different period conventions
# to different places further down the app. 

frequencyChoices <- c("days" = "daily",
                      "weeks" = "weekly", 
                      "months" = "monthly")

selectInput("frequency",
            "freq",
            choices = frequencyChoices, 
            selected = "months")

The remainder of the inputs should look familiar as they are in standard format.

dateRangeInput("dateRange",
               "Date range",
               start = "1980-01-01",
               end   = "2016-12-31")

numericInput("periods", "Periods to Forecast", 6, min = 1, max = 100)

Now that we’ve built the sidebar inputs, let’s put them to use. First, we will import the commodity time series data from Quandl. This will be familiar from the Notebook, but note in particular that we will use the value from the input$frequency choice because Quandl uses the “daily/weekly/monthly” frequency format.

# Let's pull in the Quandl data. 
# Nothing fancy here except we are going to use the reactive inputs.
# We will pass in the value from the input$dataSet key-value pair.
# We will also pass in the value from the input$frequency key-value pair. 

commodity <- reactive({
  # It might be a good idea to include your quandl api key if this will be used more than 50 times in a day. 
  Quandl.api_key("your_apikey_here")
  
  commodity <- Quandl(input$dataSet,
                      start_date = format(input$dateRange[1]),
                      end_date = format(input$dateRange[2]),
                      order = "asc",
                      type = "xts",
                      collapse = as.character(input$frequency)
  )
})

We have imported a time series object for the dataset, date range and frequency chosen by the user. Now we want to do some forecasting and create a visualization. We’ll first use the forecast() function, then we’ll combine the forecasted prices and the historical prices into one xts object that can be passed to dygraph. Let’s handle this in one reactive.

First, we’ll call forecast and pass it the periods input from the user.

combined_xts <- reactive({
  
  # Just like the Notebook, except periods is a reactive input.
  
  forecasted <- forecast(commodity(), h = input$periods)

Now we need to combine that forecasted object with the commodity object that was created in a previous code chunk. Here we will finally thank ourselves for thinking about the name-value issue when assigning frequencyChoices. When we create a dataframe to hold the forecasted time series, we build a ‘date’ column using the seq() function. The column starts on the end date of the historical data (which the user selected with the dateRange input) and runs for as many periods as the user chose in the periods input. But, we need to supply a value to the by = ... argument of the seq() function so that it knows if we want to move 6 days, 6 weeks, 6 months, etc. To do that, we need to extract the name (“days/weeks/months”) from the frequencyChoices name-value pair and pass it to seq. The way we extract the name is with this selection statement: names(frequencyChoices[frequencyChoices == input$frequency]).

  forecast_dataframe <- data.frame(
    date = seq(input$dateRange[2], 
               # The next line is very important, and it's the reason we 
               # created the frequencyChoices object.
               by = names(frequencyChoices[frequencyChoices == input$frequency]),
               length.out = input$periods),
    Forecast = forecasted$mean,
    Hi_95 = forecasted$upper[,2],
    Lo_95 = forecasted$lower[,2])

Now we convert the forecast_dataframe object to an xts and combine that new object with the commodity xts object. Remember, we imported the commodity data from Quandl in the form of an xts object, which saved us from having to do a conversion. This chunk should look very similar to the Notebook.

  forecast_xts <- xts(forecast_dataframe[,-1], order.by = forecast_dataframe[,1])
  
  combined_xts <- cbind(commodity(), forecast_xts)
  
  # Add a nicer name for the first column.
  
  colnames(combined_xts)[1] <- "Actual"
  
  # This is the combined object that will be passed to dygraphs below. 
  combined_xts
})

Next, we will create a chart of the actual price history. Nothing fancy here, except again we are going extract the ‘name’ portion of a name-value pair, this time from the dataSet object (the first one we created in the first code chunk). We want to label the graph with ‘WTI oil’ and not the Quandl code, so we select the name with names(dataChoices[dataChoices==input$dataSet]).

dygraphOutput("commodity")

output$commodity <- renderDygraph({
  dygraph(commodity(),
          # We pull out the name of the selected name-value input$dataSet like so: 
          # names(dataChoices[dataChoices==input$dataSet])
          main = paste("Price history of", names(dataChoices[dataChoices==input$dataSet]), 
                       sep = " ")) %>%
    dyAxis("y", label = "$") %>%
    dyOptions(axisLineWidth = 1.5, fillGraph = TRUE, drawGrid = TRUE)
})

Last but not least, let’s graph the historical and forecasted time series together on one graph. We could have ported the code directly from the Notebook, but I couldn’t help tinkering just a bit. I wanted to focus the graph on the time period around where the actual data ends and the forecast begins, because that is probably what’s of most interest to the user. To do that, we’ll use the dyRangeSelector() function and pass two values to the dateWindow variable: a start date and end an date. This is pure aesthetics, but I think it’s worth the effort here.

We are going to use seq() and names(frequencyChoices[frequencyChoices == input$frequency]) once again.

dygraphOutput("forecasted")

output$forecasted <- renderDygraph({

# We want the user to be able to see the forecasted area, so let's focus in on that 
# by truncating the view of the dygraph.
# We need to give the graph a start date and an end date. 
start_date <- tail(seq(input$dateRange[2], by = "-1 months", length = 6), 1)
end_date <- tail(seq(input$dateRange[2], 
                     by = names(frequencyChoices[frequencyChoices == input$frequency]), 
                     length = input$periods), 1)

Now we will supply those date objects to a piped dygraphs chain, and that will be a wrap. Have a close look at the chunk below and spot where we again extract the name from the input$dataSet name-value pair.

dygraph(combined_xts(), 
        # Name about to be extracted!
        main = paste(names(dataChoices[dataChoices==input$dataSet]), 
                     ": Historical and Forecast", sep = "")) %>%
  # Add the actual series.
  dySeries("Actual", label = "Actual") %>%
  # Add the three forecasted series.
  dySeries(c("Lo_95", "Forecast", "Hi_95")) %>% 
  # A range selector to focus on the where historical data ends and 
  # foracated data begins.
  # Note the user can still use the range selector to zoom out if so desired.
  dyRangeSelector(dateWindow = c(start_date, end_date))
})

That’s all for today! We’ve built an app that can be extended in many directions - more data set choices, an input selector so the user can choose different models, more visualizations - and in that sense it can serve as a template for ourselves and anyone that wants to build upon this work.

This app also relies on and highlights one of the tremendous benefits of Quandl: the ability to import datasets from different sources and have them arrive in the IDE in a consistent format. This makes using Quandl with Shiny quite enjoyable after the initial skeleton is built. For example, if we or a colleague did want to change this app and substitute in different commodities, there would be no worry beyond the finding the right code on Quandl and adding another name-value pair to dataChoices.

R for Enterprise: Understanding R’s Startup

Wed, 19 Apr 2017 00:00:00 +0000

R’s startup behavior is incredibly powerful. R sets environment variables, loads base packages, and understands whether you’re running a script, an interactive session, or even a build command.

Most R users will never have to worry about changing R’s startup process. In fact, for portability and reproducibility of code, we recommend that users do not modify R’s startup profile. But, for system administrators, package developers, and R enthusiasts, customizing the launch process can provide a powerful tool and help avoid common gotchas. R’s behavior is thoroughly documented in R’s base documentation: “Initialization at Start of an R Session”. This post will elaborate on the official documentation and provide some examples. Read on if you’ve ever wondered how to:

Tell R about a local CRAN-like repository to host and share R packages internally
Use a different version of Python, e.g., to support a Tensorflow project
Define a proxy so R can reach the internet in locked-down environments
Understand why Packrat creates a .Rprofile
Automatically run code at the end of a session to capture and log sesssionInfo()

We’ll also discuss how RStudio starts R. Spoiler: it’s a bit different than you might expect!

.Rprofile, .Renviron, and R*.site oh my!

R’s startup process follows three steps: starting R, setting environment variables, and sourcing profile scripts. In the last two steps, R looks for site-wide files and user- or project-specific files. The R documentation explains this process in detail.

Common Gotchas and Tricks:

The Renviron file located at R_HOME/etc is unique and different from Renviron.site and the user-specific .Renviron files. Do not edit the Renviron file!
A site-wide file, and either a project file or a user file, can be loaded at the same time. It is not possible to use both a user file and a project file. If the project file exists, it will be used instead of the user file.
The environment files are plain-text files in the form name=value. The profile files contain R code.
To double check what environment variables are defined in the R environment, run Sys.getenv().
Do not place things in a profile that limit the reproducibility or portability of your code. For example, setting options(stringsAsFactors = FALSE) is discouraged because it will cause your code to break in mysterious ways in other environments. Other bad ideas include: reading in data, loading packages, and defining functions.

Where to put what?

The R Startup process is very flexible, which means there are different ways to achieve the same results. For example, you may be wondering which environment variables to set in .Renviron versus Renviron.site. (Don’t even think about calling Sys.setenv() in a Rprofile…)

A simple rule of thumb is to answer the question: “When else do I want this variable to be set?”

For example, if you’re on a shared server and you want the settings every time you run R, place .Renviron or .Rprofile in your home directory. If you’re a system admin and you want the settings to take affect for every user, modify Renviron.site or Rprofile.site.

The best practice is to scope these settings as narrowly as possible. That means if you can place code in .Rprofile instead of Rprofile.site you should! This practice complements the previous warnings about modifying R’s startup. The narrowest scope is to setup the environment within the code, not the profile.

Quiz

What is the best way to modify the path? The answer depends on the desired scope for the change.

For example, in an R project using the Tensorflow package, I might want R to use the version of Python installed in /usr/local/bin instead of /usr/bin. This change is best implemented by reordering the PATH using PATH=/usr/local/bin:${PATH}. This is a change I only want for this project, so I’d place the line in a .Renviron file in the project directory.

On the other hand, I may want to add the JAVA SDK to the path so that any R session can use the rJava package. To do so, I’d add a line like PATH=${PATH}:/opt/jdk1.7.0_75/bin:/opt/jdk1.7.0_75/jre/bin to Renviron.site.

R Startup in RStudio

A common misconception is that R and RStudio are one in the same. RStudio runs on top of R and requires R to be installed separately. If you look at the process list while running RStudio, you’ll see at least two different processes: usually one called RStudio and one called rsession.

RStudio starts R a bit differently than running R from the terminal. Technically, RStudio doesn’t “start” R, it uses R as a library, either as a DLL on Windows or as a shared object on Mac and Linux.

The main difference is that the script wrapped around R’s binary is not run, and any customization to the script will not take affect. To see the script try:

cat $(which R)

For most people, this difference won’t be noticeable. Any settings in the startup files will still take affect. For user’s that build R from source, it is important to include the --enable-R-shlib flag to ensure R also builds the shared libraries used by RStudio.

R Startup in RStudio Server Pro

RStudio Server Pro acts differently from R and the open-source version of RStudio. Prior to starting R, RStudio Server Pro uses PAM to create a session, and sources the rsession-profile. In addition, RStudio Server Pro launches R from bash, which means settings defined in the user’s bash profile are available.

In short, RStudio Server Pro provides more ways to customize the environment used by R. You might ask why you’d ever want more options. Recall our rule of thumb: “When else do I want this variable to be set?”

In server environments, there are often environment variables set every time a user interacts with the server. These environment variables are placed in a user’s bash profile by a system admin. Normally R wouldn’t pick up these settings. RStudio Server Pro allows R to make use of the work the system admin has already done by picking up these profiles.

Likewise, there may be some actions that take place on the server when a user logs in that have to happen before R starts. For example, a Kerberos ticket used by the R session to access a data source must exist before R is started. RStudio Server Pro uses PAM sessions to enable these actions.

There may also be actions or variables that should only be defined for RStudio, and not any other time R is run. To facilitate this use case, RStudio Server Pro provides the rsession-profile. For example, if your environment makes use of RStudio Server Pro’s support for multiple versions of R, you’d place any environment variables that should defined for all versions of R inside of rsession-profile.

Examples:

Define proxy settings in Renviron.site

Renviron.site is commonly used to tell R how to access the internet in environments with restricted network access. Renviron.site is used so the settings take affect for all R sessions and users. For example,

http_proxy=http://proxy.mycompany.com

This article contains more details on how to configure RStudio to use a proxy.

Add a local CRAN repository for all users

Organizations with offline environments often use local CRAN repositories instead of installing packages directly from a CRAN mirror. Local CRAN repositories are also useful for sharing internally developed R packages among colleagues.

To use a local CRAN repository, it is necessary to add the repository to R’s list of repos. This setting is important for all sessions and users, so Rprofile.site is used.

old_repos <- getOption("repos")
local_CRAN_URI <- paste0("file://", normalizePath("path_to_local_CRAN_repo"))
options(repos = c(old_repos, my_repo = lcoal_CRAN_URI))

More information on setting up a local CRAN repository is available here.

Record sessionInfo automatically

Reproducibility is a critical part of any analysis done in R. One challenge for reproducible scripts and documents is tracking the version of R packages used during an analysis.

The following code can be added to a .Rprofile file within an RStudio project to automatically log the sessionInfo() after every RStudio session.

This log could be referenced if an analysis needs to be run at a later date and fails due to a package discrepancy.

.Last <- function(){
  if (interactive()) {
    
    ## check to see if we're in an RStudio project (requires the rstudioapi package)
    if (!requireNamespace("rstudioapi"))
      return(NULL)
    pth <- rstudioapi::getActiveProject()
    if (is.null(pth))
      return(NULL)
    
    ## append date + sessionInfo to a file called sessionInfoLog
    cat("Recording session info into the project's sesionInfoLog file...")
    info <-  capture.output(sessionInfo())
    info <- paste("\n----------------------------------------------",
                  paste0('Session Info for ', Sys.time()),
                  paste(info, collapse = "\n"),
                  sep  = "\n")
    f <- file.path(pth, "sessionInfoLog")
    cat(info, file = f, append = TRUE)
  }
}

Automatically turn on packrat

Packrat is an automated tool for package management and reproducible research. Packrat acts as a super-set of the previous example. When a user opts in to using packrat with an RStudio project, one of the things packrat automatically does is create (or modify) a project-specific .Rprofile. Packrat uses the .Rprofile to ensure that each time the project opens, Packrat mode is turned on.

To Wrap Up

R’s startup behavior can be complex, sometimes quirky, but always powerful. At RStudio, we’ve worked hard to ensure that R starts and stops correctly whether you’re running RStudio Desktop, serving a Shiny app on shinyapps.io, rendering a report in RStudio Connect, or supporting hundreds of users and thousands of sessions in a load balanced configuration of RStudio Server Pro.

March '17 Tips and Tricks

Fri, 31 Mar 2017 00:00:00 +0000

This month’s Tips and Tricks focus is on file navigation. Many of these tips are straightforward - more tip than trick - but they can save quite a bit of time and frustration.

Open Recent

Please don’t spend time navigating through your folder structure to find a recent project or file.

Go to File/Function

RStudio automatically indexes all open files as well as files in the current project. The index includes function calls and objects within the files. To search the index, enter a search string in the Go to file/function window in the main toolbar or use the keyboard shortcut Ctrl + ..

The search supports fuzzy matching as long as the search string is in the correct order. For example, tiptric will match files and function names like march-tips-and-tricks.Rmd.

Find in File

In addition to the index, RStudio can quickly search the text of any file(s) for a string or regular expression. Open the Find in Files dialogue by navigating through the toolbar Edit -> Find in Files or using the keyboard shortcut Cmd / Ctrl + Shift + F.

Tabs

Close All and Close All Except Current

If you’re like me, you often end up with a whole bunch of files open and the number of tabs in the source pane causes anxiety. Don’t click through the tabs closing each individually.

Search for a Tab

If you don’t get anxiety from multiple tabs, you may find the keyboard shortcuts for switching between tabs helpful. My favorite is Ctrl + Shift + . which opens a search dialogue for the currently open tabs.

Set Working Directory

Sometimes I find myself inside a file and I have no idea where that file is. In this scenario, I often use the following trick to orient myself:

The other options, To Project Directory and To File Pane Location are also very helpful. These commands do not have key bindings by default, but you can create your own keyboard shortcuts for these actions: Tools -> Modify Keyboard Shortcuts...

File Pane: Go To Working Directory

After mastering tip 5, you can easily navigate from the file pane to the working directory:

File Pane: Go To Project Root Directory

Finally, it is really easy to go to the root of a project directory:

Be sure to follow @rstudiotips for more frequent updates. Tweet us @rstudio if there’s a file navigation trick you wish we’d implement!

February '17 Tips and Tricks

Fri, 03 Mar 2017 00:00:00 +0000

If you spend time with an excellent programmer, one thing that immediately jumps out is how quickly she can write code. It often appears to be magic, the number of keystrokes simply can’t equal the number of characters on the screen. The secret: it doesn’t! Most programmers use a series of tricks to save the hassle of writing boilerplate code. This month, my personal goal was to spend less time typing, even if it required slowing down to master these tricks. In this post, I’ll focus on one of the easiest and most flexible tools at your disposal: code snippets. (Astute readers will note that I’ve mentioned code snippets before. They deserve the extra attention!)

Code snippets are a common programming tool similar to keyboard shortcuts. A snippet of text, usually a few characters long, is used as a shortcut to create a larger piece of boilerplate code. Tab navigation enables jumping through the boilerplate code to locations where customization is required.

For example, one of my favorite code snippets is for creating a function. The template of an R function looks like:

functionName <- function(arguments) {
  code 
}

The boilerplate - not worth typing - is:

 <- function( ) {
 
}

In RStudio, the code snippet for functions is fun. Typing fun and pressing Tab will provide an auto-complete suggestion for the snippet. Typing fun and pressing Shift + Tab inserts the snippet directly. As a general typing rule, keeping your hands on the middle row of the keyboard will speed things up. I hit Shift with my right pinky and Tab with my left.

After inserting the boilerplate code, the cursor is placed to left of the <-. You can then quickly type the name of the function. Pressing Tab places the cursor in between the parenthesis. Pressing Tab again places the cursor in between the brackets. This allows you to fill in the remaining portions of the function template quickly.

Type fun and Shift + Tab:

HERE <- function() {

}

Pressing Tab Once

 <- function(HERE) {
 
}

Pressing Tab Twice

 <- function() {
 HERE
 }

Accessing Snippets

RStudio comes with many built-in snippets. To access these snippets, navigate to Tools -> Global Options -> Code and press Edit Snippets.

The menu will show all of the defined snippets. Consider the definition for the function snippet:

snippet fun
	${1:name} <- function(${2:variables}) {
		${0}
	}

The snippet starts by defining the character shortcut fun. The definition includes the boilerplate code. The Tab behavior is defined using ${1} notation. The code snippet help page describes the syntax in more detail.

A Few Examples

`ts`

Adds a time-stamped comment.

# Mon Feb 20 07:34:03 2017 ------------------------------

`if`

Adds the skeleton of an if statement

if () {

}

`mat`

Adds the skeleton for defining a matrix.

matrix( , nrow = , ncol = )

There are code snippets for other languages as well.

Creating Snippets

Code snippets are powerful because they can be customized and created. If you find yourself repeatedly typing the same code, create a snippet. The menu of snippets in RStudio is editable. To create or modify a snippet, add your changes to the menu and click Save.

Once you’ve created a handful of useful code snippets, it would be generous to share. The snippr package makes this process easy.

This gist contains two of my favorites: a code snippet for creating a single file Shiny app, and a code snippet to start a ggplot.

Shiny:

snippet shiny
	library(shiny)
	ui <- fluidPage(
		${1:ui}
	)
	
	server <- function(input, output, session){
	  ${2:server}
	}
	shinyApp(ui = ui, server = server)

ggplot2:

snippet gg
	ggplot(${1:data}, aes(${2:aes})) + 
	  geom_${3:geom}()

What are your favorite snippets?

January '17 Tips and Tricks

Fri, 27 Jan 2017 00:00:00 +0000

This month’s collection of Tips and Tricks comes from an excellent talk given at the 2017 RStudio::Conf in Orlando by RStudio Software Engineer Kevin Ushey. The slides from his talk are embedded below and cover features from autocompletion to R Markdown shortcuts. Use the left and right arrow keys to change slides.

Enjoy!

Get Ready for RStudio::Conf

Fri, 23 Dec 2016 00:00:00 +0000

The 2017 R Conference season will get off to an early start on January 13th and 14th with RStudio::Conf 2017 in Orlando, Florida. The schedule promises an intense but collegial experience with plenty of hands-on practice working with R and the RStudio tool chain of packages and products. To prepare for the conference, I thought I would try to extract the major themes from the list of talks and workshops and point out some resources to study ahead of time. Even if you are not going to the conference, you may find these themes and resources helpful in improving your R experience.

The first big theme might be described as R for Data Science. Conference talks will take you through operational issues using GitHub and working with databases, text mining with the Tidyverse of data acquisition, manipulation and exploration packages, writing readable code with pipes, functional programming using purrr, profiling code performance with the RStudio IDE, writing high performance code with Rcpp, and running on Spark clusters with sparklyr.

A second prominent theme is about leveraging the reach, interactivity and visualization potential of the web from R. Talks will describe new Shiny capabilities like bookmarking Shiny State and dynamic user interfaces. There will also be presentations on htmlwidgets, mapping with Leaflet, interactive plotting with rbokeh, and using web APIs from R.

The third big theme comprises talks about using Shiny and RMarkdown to facilitate reproducible research and publishing. Speakers will describe RStudio Connect, a new enterprise platform for “push-button” publishing of your team’s Shiny apps, RMarkdown documents, notebooks and plots. There will be a tutorial on advanced RMarkdown features and talks on using RMarkdown to make dashboards and web applications, extending and customizing RMarkdown and on R Notebook workflows and techniques.

This will be our final R Views post of the year. Our first post of 2017 is planned for January 4th. We are very thankful for your readership, and we will go into the new year with the mission to expand our coverage of all things interesting to the R Community. If you have something you would like to contribute pleased write to me at joseph.rickert@rstudio.com. We welcome contributions from guest bloggers and the possibility of giving voice to diverse views from the Community.

On behalf of all of RStudio we wish you: Happy Holidays and a bright and prosperous New Year!

December '16 RStudio Tips and Tricks

Fri, 09 Dec 2016 00:00:00 +0000

Here is this month’s collection of RStudio Tips and Tricks. Thank you to those who responded to last month’s post; many of your tips are included below! Be sure to subscribe to @rstudiotips on Twitter for more.

This month’s tips fall into two categories: Keyboard Shortcuts and Easier R Markdown.

Keyboard Shortcuts

The RStudio IDE is built upon “hooks”. Hooks are actions that the IDE can take. For instance, there is a hook to create a new file. Most users interact with hooks with point-and-click interactions. (RStudio toolbar -> new file or File -> New File). But, there is an alternative! All of these hooks have been surfaced to end users and can be bound to a keyboard shortcut. (Some of these actions are “secret” - they aren’t exposed through point-and-click options.)

Custom Keyboard Shortcuts

To view the complete list of actions, the current keybindings, and to customize keybindings, go to: Tools -> Modify Keyboard Shortcuts.

Define shortcuts for code chunk navigation using the previous tip. For example, Alt+Cmd+Down for Next Chunk and Alt+Cmd+Up for Previous Chunk.

Assignment Operator

Use Alt+- (press Alt at the same time as pressing -). This adds the assignment operator and spacing.

Pipe Operator

Use Cmd+Shift+m (for Mac) or Ctrl+Shift+m (for Windows). This adds the pipe operator %>% and spacing.

Easier R Markdown

R Markdown Options

R Markdown output formats include arguments specified in the YAML header. Don’t worry about remembering all of the key-value pairs; in RStudio, you can access and change the most common through a user-interface.

Spell Checker

Use the built-in spell checker when writing a R Markdown document. (Code chunks are automatically ignored.)

SQL Code Chunks

Execute SQL queries against database connections directly in R Markdown chunks.

R Markdown Websites

Are you building a website with R Markdown? Any RStudio project with an R Markdown website will include a Build Website option in the build pane.

What’s your favorite RStudio Tip?

RStudio IDE Easy Tricks You Might've Missed

Fri, 11 Nov 2016 00:00:00 +0000

The RStudio IDE reached version 1.0 this month. The IDE has come a long way since the initial release 5 and a half years ago. Many major features have been built: projects, package building tools, notebooks. During that same period, often hidden in the shadows, a growing list of smaller features has been changing lives. In celebration of version 1.0 this post hopes to spread fanfare for a few of these easy-to-miss tools.

Tearable Panes

Tearable panes are anything but terrible. This feature allows users to tear off data view panes and source panes facilitating the use of multiple screens.

Command History

In the console it is possible to scroll through the command history by clicking Ctrl/Cmd and ↑. The command history will be filtered as code is typed into the console:

History Pane

The history pane shows a searchable list of commands that have been run. Commands can be written to the source pane or the console. No more copy and paste from the console to a script!

Rename in Scope

This feature makes it easy to rename all instances of a variable. The tool is context aware; changing ’m’ to ‘m1’ won’t change ‘mtcars’ to ‘m1tcars’.

Gallery and Satellite View in Notebooks

A new feature built into R Notebooks, a code chunk that produces multiple plots will produce a gallery. The plots can be viewed by toggling between thumbnails. The gallery can be expanded into a new satellite window for closer inspection.

Code Outline

Save time scrolling with the code outline. This feature works for R Notebooks and traditional R scripts. In R Notebooks sections are delimited by the R Markdown headers. In R scripts sections are delimited by section comments (Try Code -> Insert Section).

Code Snippets

Code snippets are a shortcut to insert common boilerplate code. For instance, type fun and then Tab to insert the skeleton code for a function definition. Then hit Tab to replace the necessary components. In addition to a rich set of defaults, custom code snippets can also be created. Learn more.

Many people know of RStudio’s rich set of tab complete options for functions and function arguments. Tab complete can also help find files and remove the hassle of writing out long path locations. Hit tab in between two double quotes (” “) to open a file explorer.

Jump To Function Definition

Want to dig into the innards of a function? With the cursor on a function press F2 to jump to the function definition, even for functions in a package.

These are just a few of my favorite tricks. A comprehensive list is available in the RStudio IDE Cheatsheet or follow @rstudiotips on Twitter to learn a few tricks every week.

What’s your favorite RStudio tip?

Interview with J.J. Allaire

Wed, 12 Oct 2016 00:00:00 +0000

Welcome to “R Views”, the new R Community blog from RStudio. For this first post, I sat down with J.J. Allaire, RStudio’s founder and CEO, to discuss RStudio’s history, its mission and JJ’s vision for its future. In a short time, we touched on a wide range of subjects including RStudio’s business, the growth of the R language, the importance of the R Consortium to the R Community and J.J.’s advice to anyone coming to R for the first time. We hope you enjoy this “snapshot” of RStudio’s place in the R world.

JBR: How did you first become involved with R? What were you doing when R first came to your attention, and why R?

J.J.: As an undergraduate I studied economics and political science and as a result got very interested in quantitative research. As a student I worked with SAS, SPSS, and Excel. This was my first serious exposure to software and computing, which I also became quite fascinated with.As it turned out, my career ended up developing more around software than around quantitative analysis. Then, about eight years ago I was between jobs and looking for the next thing to work on. I was very much of the mindset that I wanted my work to be open source, because I wanted to create software that would be free and accessible to everyone as well as long lasting.

That’s when I learned about R. It was really wonderful to hear that there was a robust and broadly used open source statistical computing environment. I thought that this might be a great place to apply my energy. Quantitative analysis was my original interest in school, and in the meantime, I had acquired considerable skill in writing software systems. I thought I could take some of that, apply it to R, and make a meaningful contribution.

JBR: Was there any specific event that motivated you to start RStudio? When you did start, what kind of company did you envision then?

J.J.: RStudio really wasn’t started as a company. It began as an open source project that I just worked on by myself. Initially, I started working on the concept of a web based IDE for R. About nine months or so into the project I realized that it would be a really huge undertaking for me to try to build this IDE all by myself.

So, I recruited Joe Cheng to work with me. Joe and I had worked together in a couple of previous companies. At the time I told him: “Joe, I make no representation or promise that this is going to be a company. It might just be an open source project, is that okay with you?” He said, “That’s fine”. It was really just the two of us working on an open source project at the beginning. Again, there was no notion that it would have to become a company. My feeling was that if there was a company that could evolve out of it that would be fine, but that was not required.

JBR: Are you surprised about how RStudio turned out?

J.J.: I didn’t know how it was going to turn out. I definitely knew that open source software has the potential to be very widely used. I’m not surprised at all that R is a really big deal, but in no way did I presume or know whether there would be a company that you could build around R. I honestly would have been surprised by any outcome. I really didn’t know.

JBR: What is your role at RStudio? Do you run the company? Are you responsible for product development? What gets you up in the morning?

J.J.: What I principally do is write software. That’s what I love to do. I’m the CEO of RStudio but I don’t really run the company per se. I collaborate with other people who are engaged in running the company. In particular I spend a lot of time with with Tareef Kawaf who is the President of RStudio and is primarily responsible for running it.

I really spend the vast majority of my time, over eighty percent, writing software. I started off as the principal developer on RStudio IDE and then Joe and I worked on it together for a number of years. I continue to spend time on the IDE, but I’m also the principal developer on quite a few R packages including the R Markdown package. I’m the maintainer of a few other packages like htmlwidgets and the dygraphs package, and I have also contributed to several other packages like shiny and Rcpp and others.

JBR: What does a typical work day look like for you?

J.J.: On a typical work day, I look at my email from overnight and see if there’s anything urgent. I may give feedback on some pull requests or take a look at a few bugs. I probably spend about an hour just taking stock of things that are in my inbox. Then I’ll work for five or six hours, writing code for various products and packages.

After that, I may check in with RStudio’s Seattle office when those guys come online at midday, East Coast time. Then, maybe I’ll do some more coding until the end of the day. Various things come up, but I have very few meetings that I regularly attend. I try to keep things clear for working on product development.

JBR: From the outside it looks like RStudio has committed a considerable amount of resource to developing open source packages and free products. Many people wonder how RStudio can even be a business. Can you talk a little bit about RStudio’s business plan, how does RStudio make money?

J.J.: Well, first of all, the mission of the company, is to create open source software for data analysis and statistical computing. So, if you see that we’re creating a lot of open source software that should never surprise you: that is our principal mission. At the same time, in order to fulfill our mission we need to bring together as many talented people as we can. For that to happen obviously we also need to develop and grow a business around R.

I’d say the commercial business has two major facets, the first is concerned with organizations that want to deploy R at scale; scale here meaning either computing scale or just adopting R within a larger organization. For these requirements, we have enhanced, commercial versions of some of our server products, for example RStudio Server Pro and Shiny Server Pro, that have features aimed at deploying R at scale. Our open source products can typically be adopted by many, many people without buying our professional products, and that’s how we want it to be. When customers get more serious about R, or they are deploying R in a larger environment, they tend to buy our professional products. So that’s the main way we make money, those professional server products.

We also have a cloud business where we provide versions of our products over the web. For example, we have shinyapps.io for deploying Shiny applications. It’s not a big a part of our business yet, but we expect it to become more and more significant, and we also expect to have other cloud offerings in the future.

JBR: What do RStudio’s long range goals look like?

J.J.: One core set of goals is centered around creating an elegant and productive environment for using R. Mostly here I’m talking about the RStudio IDE. A really nice environment for working with R not only needs to provide the tools to make users very productive, but it also needs to surface all of the richness that is inside R so that it’s conceptually clear to people what’s possible, and then makes it easy for them to do what they want to do.

Other goals are centered on creating APIs for R that are really intuitive and work well together. That’s the tidyverse, if you will, the family of packages that Hadley and others have developed like ggplot2, dplyr, tidyr, lubridate, stringr and so on.

We have also focused quite a bit on helping people communicate what they do in R. Here, I’m talking about R Markdown which is all about creating documents and web content from R, and Shiny which is about creating interactive applications with R.

In the future, we want to continue to excel along those three axes and make all of our open-source products and packages even better. We also continue to learn more about the needs of companies who are deploying R at scale, whether it be trying to achieve high performance, ensuring robust security, or integrating seamlessly with other enterprise software systems. As we continue to learn we will continue to develop products along this axis as well. Right now, we’ve solved some of the problems, but there are a whole lot more to address, and I don’t think we’re nearly done on any of these fronts. Building up our professional server product line is a big focus as is building hosted products to let people access and use R in the cloud.

Also, we want to do a better job of educating people about what we already have. So we are trying to create a well-thought-through curriculum that people can access through different mediums including books, videos, websites and interactive self-paced tutorials. You’ll see lots more investment in education from us in the future.

JBR: So far, from what you’ve said, RStudio projects all seem to be related to producing tools to help others develop R software and do analytics work, but what would a complete R based tool chain look like?

J.J.: That’s interesting. I think what we have been talking about, the things that we’ve traditionally focused on like helping to deploy R at scale are horizontal tools. They are very, very helpful, but there’s actually another layer of tools that we haven’t talked about. In any given domain of application, you want to have a suite of tools that are specific to that domain. These are also extremely important, but we are more of a facilitator of those tools than a creators, since we’re not domain experts.

Over time, I think that the whole R community is engaged in a really big collective project to build out all the tools required to let people analyze data in the different domains. Some of the work that we are doing is very much focused on enabling this. For example, Hadley tries both to build tools, and to teach people how to build their own great tools for R. As a result, people have built some wonderful packages to make it easier to do specialized types of analyses. The work I’ve done on Rcpp is all about letting people access the highest performance capabilities within their domain whether that’s reading data really fast, or analyzing it quickly, or optimizing certain algorithms. Also, many of the tools that we’ve developed in data visualization are about providing an extensible platform to let people do custom static and interactive visualizations for different domains. So what does a complete tool chain look like? I think part of it is made up of the tools that we build, and the rest of it is made up of the tools other people build on top of our stuff.

JBR: Does sparklyr represent a new direction for RStudio, or is this just part of the tool chain you described?

J.J.: I don’t think it’s a new direction. In his keynote address at the 2014 UseR! conference in UCLA, John Chambers spoke about how the original concept for R was to create an interface language. One of the motivators was that people wanted to use the robust and high performance FORTRAN libraries for doing statistical analysis that already existed, but they wanted to use them in a flexible and interactive way. So the idea was to create a language that was really good at providing interfaces to other computing systems. In the beginning, it was interfacing to an existing body of FORTRAN code, but over time that has been extended to run just about everything. R is a very good interface language.

John gave several examples during his talk. One was Rcpp which I’ve already talked about. The other was H2O, which is a distributed computing system for doing machine learning. H2O is written in Java, not R, but the developers have provided it with a very nice R interface. I think that’s how we view just about everything that might be relevant to statisticians or data analysts. If it’s not already in R you can use R to create a fluent and expressive interface for it.

That’s how we view sparklyr. Spark provides a set of big data capabilities that people want to take advantage of, and we’re creating a rich interface in R so the tools and techniques, the working environment and the ability to communicate, things like R Markdown and Shiny, just automatically come with you when you go to work with Spark. John mentioned H2O, but R’s been doing things with distributed processing for a long time, so I think we’re just fulfilling the vision and mission of R against the latest and greatest distributing computing platforms.

JBR: What do you think about R and big data in general? Do you think what RStudio is doing is going to help overcome the perceived limitations of R?

J.J.: Again, you should understand R as a user interface, or a language that’s capable of providing very good user interfaces. Developers don’t implement core algorithms or core distributed computing primitives in R. They use R as a way of interacting with those things.Once people understand this, and they see how good R is, how they can use dplyr to interact with a Spark cluster, I think the light bulb will go off and they’ll say: “Okay, I understand where R fits now. It’s the world’s best interactive environment for data analysis and now I can use it directly against Spark”.

JBR: RStudio is a founding platinum member of the R consortium, seems from the outside really looks like a really big commitment for a small company.

J.J.: Yes

JBR: Why is the R Consortium important? What are your hopes for its future, and what do you think the R Consortium can accomplish?

J.J.: What’s really interesting about the R consortium is that it represents all of R’s stakeholders. There are the original stakeholders, the R Foundation and R Core, the creators of R. There’s a whole community of package developers and users, and then there’s the vendors who are in various ways making a business around R. The R Consortium is an alliance of all these people that says “Hey, we all have a stake in this being as good as it can be. We all have a stake in the long term integrity of the system, so let’s get together in one venue that will allow us to have a dialog about things and fund projects that can help the R Community”.

The R Consortium is a really profound and important group. It has the potential to be the nexus point for everyone in the R Community. That’s why we’re so enthusiastic about the R Consortium and why we’ve got folks within RStudio spending considerable time on it. If it works well, the R Consortium is going to provide a lot of ballast for R, provide a great forum for communicating about R, and help establish trust in R. We want to support it as much as we can.

JBR: In addition to supporting the R community it looks like RStudio packages are geared toward reproducible research and improving scientific research in general. Do you have specific goals or aspirations with respect to science itself?

J.J.: There is a broader movement in science around reproducible research that we can contribute to. Reproducible research is all about the integrity of the work you do. It’s all about the fact that people rely on scientific software to make very important decisions and to gain a clear understanding about how the world works. The integrity and the trustworthiness of the process by which that software yields its results is critical.

Our goal is to make sure that the tools for working reproducibly within the R ecosystem are excellent. In the past, people may have felt that working reproducibly was going to require more work, more effort, and thought “I don’t have time to do that”. We’re trying to make it so that reproducible work actually takes less effort. It’s sort of that pit of success idea where we provide a great tool chain for creating compelling documents which has the very pleasant side effect that everything that people do is reproducible.

Also, reproducibility is one of the reasons we focus on teaching people how to write and work with code to do data analysis. We don’t have visual abstractions for doing data analysis because those types of techniques are not reproducible, not easy to inspect, not easy to share, and not easy to automate. We really focus on code. It’s a core principle that, no matter what, we’d rather make code easier to work with than try to eliminate it.

JBR: Do you have any advice for people coming to R for the first time, either as students or as experienced statisticians and researchers?

J.J.: I would suggest that they get a copy of the R for Data Science book written by Hadley Wickham and Garrett Grolemund. The book explains the tidyverse and fundamental principles for working with data in R. There are a lot of great tools for working in R but not everybody who comes to R encounters them right away. We’re trying to change that. I would also recommend that newcomers to R take a careful look at all the tools and packages that we have developed at RStudio.

Also, when you have questions or run into problems don’t give up. There’s a lot of great activity around R on stackoverflow and other places and there’s an excellent chance you’re going to find the answers to your questions if you look carefully for them.

JBR: Do you have any thoughts on what statisticians and data scientists ought to know about computer science in general?

J.J.: That’s an interesting question. A lot of the popular philosophy about R revolves aroundBo Cowgill’s famous quote: “The best thing aboutRis that it was written by statisticians. The worst thing about R is that it was written by statisticians.“The fact is, however, that the folks who made R knew quite a bit about computing and computer science. Their goal was, and continues to be, to provide an expressive language that lets you accomplish what you want to accomplish with a minimal requirement for mastering computer science. R is trying to provide an interface and a vocabulary that allows non-computer science people to still do sophisticated things with data.

I think there will always be sets of problems that you can’t solve with a high level vocabulary. So, becoming more educated about general programming principles, doing some lower level computations or lower level text analysis or parsing and those sorts of things is useful. But, the whole point of R is to not force everyone to become a computer scientist.

JBR: Last question: where do you think R and RStudio will be in ten years?

J.J.: Currently, R is quite prominent in the universities. It’s taught to undergraduates and researchers and, more and more it’s becoming part of the core curriculum. However, in the world of practitioners, there’s still an awful lot people using proprietary software that in many cases is less capable, less flexible, less powerful than R.

I’m hopeful that over the next ten years we’ll start to see the kind of predominance R now has in universities really take hold in the world of practicing statisticians. I’m hopeful that will happen, and I’m hopeful that RStudio can be a big part of making the transition to R a positive experience. We want to continue to make R easier to learn and easier to use so that when organizations decide that they’re going to commit to R for the long term and replace their proprietary statistics software, they’ll come to an environment that is really refined, really powerful, really easy to learn, and that they can actually deploy R at scale and be successful with it. I hope that RStudio can facilitate all those things in the next ten years.

JBR: Thank you J.J.!