Packages on R Views

reticulate, virtualenv, and Python in Linux

Mon, 10 Jun 2019 00:00:00 +0000

Roland Stevenson is a data scientist and consultant who may be reached on Linkedin.

reticulate is an R package that allows us to use Python modules from within RStudio. I recently found this functionality useful while trying to compare the results of different uplift models. Though I did have R’s uplift package producing Qini charts and metrics, I also wanted to see how things looked with Wayfair’s promising pylift package. Since pylift is only available in python, reticulate made it easy for me to quickly use pylift from within RStudio.

In the article below, I’ll show how I worked through the following circumstances:

Since pylift has only been tested on Python >= 3.6, and my system version of Python was 2.7, I needed to build and install Python 3.6 for myself, preferably within a self-contained virtual environment.
I wanted to install pylift in the virtual environment and set up reticulate in my R Project to work within that environment.
Finally, I needed to access pylift from an R Markdown document via the reticulate interface.

Setting up Python, virtualenv, and RStudio

Note: for consistency, I always use an instance created via r-studio-instance and a base project from r-studio-project.

Python 2.7 is the default on the systems I use (CentOS 6/7). Since I did not want to modify the system-level Python version, I installed Python 3.6.x at the user level in $HOME/opt and created a virtual environment using Python 3. I then activated the Python 3 environment and installed pylift. Finally, I ensured RStudio-Server 1.2 was installed, as it has advanced reticulate support like plotting graphs in line in R Markdown documents.

Below is a brief script that accomplishes the tasks in bash on CentOS 7:

cd ~
mkdir tmp
cd tmp
wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz
tar -xzvf Python-3.6.2.tgz
cd Python-3.6.2
./configure --prefix=$HOME/opt/python-3.6.2 --enable-shared
make
make install
cd ~
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/opt/python-3.6.2/lib
virtualenv -p $HOME/opt/python-3.6.2/bin/python3 pylift
source pylift/bin/activate
cd pylift
git clone https://github.com/wayfair/pylift
cd pylift
pip install .
pip install -r requirements.txt
cd
wget https://s3.amazonaws.com/rstudio-ide-build/server/centos6/x86_64/rstudio-server-rhel-1.2.1335-x86_64.rpm
sudo yum install -y --nogpgcheck rstudio-server-rhel-1.2.1335-x86_64.rpm
sudo rstudio-server start

Some notes:

the --enable-shared option is required when building Python in order for reticulate to work
the LD_LIBRARY_PATH library also needs to be set prior to creating the virtual environment
we use virtualenv to create a virtual environment called “pylift” and then ensure that all Python packages are installed to that environment only (so as not to pollute any other environments we are working with)
we then clone the pylift source and install pylift along with all of its requirements via pip install -r requirements.txt
finally, we install the RStudio Server 1.2 Preview version in order to leverage its advanced reticulate features

Using Python from within RStudio via reticulate

Switching from bash to RStudio, we load reticulate and set it up to use the virtual environment we just created. Finally, and specific to pylift, we set matplotlib parameters so that we can plot directly in R.

library(reticulate)

Sys.setenv(LD_LIBRARY_PATH = paste0(Sys.getenv("HOME"),"/opt/python-3.6.2/lib"))
Sys.getenv("LD_LIBRARY_PATH")
use_virtualenv("/home/rstevenson/pylift", required=TRUE)
py_config()

# Currently this must be run in order for R-markdown plotting to work
matplotlib <- import("matplotlib")
matplotlib$use("Agg", force = TRUE)

Test that it works

The following replicates the first part of pylift tutorial: simulated data

import matplotlib.pyplot as plt
import numpy as np
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t,s)

When run, the above code chunk should display a sinusoidal graph below it.

import numpy as np, matplotlib as mpl, matplotlib.pyplot as plt, pandas as pd
from pylift import TransformedOutcome
from pylift.generate_data import dgp
# Generate some data.
df = dgp(N=10000, discrete_outcome=True)

# Specify your dataframe, treatment column, and outcome column.
up = TransformedOutcome(df, col_treatment='Treatment', col_outcome='Outcome', stratify=df['Treatment'])

# This function randomly shuffles your training data set and calculates net information value.
up.NIV()

The above Python chunk uses reticulate from within RStudio to interact with pylift in the context of a custom virtual environment, using a custom version of Python. This degree of customization and functionality should be useful to users who:

want to use a different Python version than they typically use while not affecting their typical setup by way of a virtual environment
want to install a Python module like pylift within a virtual environment so as not to affect any of their user- or system-level Python module installations
want to use reticulate from RStudio to access a custom virtual environment, Python version, and Python modules
wants to be able to delete the virtual environment and R-Project and have everything go back to the way it was
wants to be able to reproduce or share the environment exactly so that the workflow can be shared with others

Reproducible Environments

Mon, 22 Apr 2019 00:00:00 +0000

Great data science work should be reproducible. The ability to repeat experiments is part of the foundation for all science, and reproducible work is also critical for business applications. Team collaboration, project validation, and sustainable products presuppose the ability to reproduce work over time.

In my opinion, mastering just a handful of important tools will make reproducible work in R much easier for data scientists. R users should be familiar with version control, RStudio projects, and literate programming through R Markdown. Once these tools are mastered, the major remaining challenge is creating a reproducible environment.

An environment consists of all the dependencies required to enable your code to run correctly. This includes R itself, R packages, and system dependencies. As with many programming languages, it can be challenging to manage reproducible R environments. Common issues include:

Code that used to run no longer runs, even though the code has not changed.
Being afraid to upgrade or install a new package, because it might break your code or someone else’s.
Typing install.packages in your environment doesn’t do anything, or doesn’t do the right thing.

These challenges can be addressed through a careful combination of tools and strategies. This post describes two use cases for reproducible environments:

Safely upgrading packages
Collaborating on a team

The sections below each cover a strategy to address the use case, and the necessary tools to implement each strategy. Additional use cases, strategies, and tools are presented at https://environments.rstudio.com. This website is a work in progress, but we look forward to your feedback.

Safely Upgrading Packages

Upgrading packages can be a risky affair. It is not difficult to find serious R users who have been in a situation where upgrading a package had unintended consequences. For example, the upgrade may have broken parts of their current code, or upgrading a package for one project accidentally broke the code in another project. A strategy for safely upgrading packages consists of three steps:

Isolate a project
Record the current dependencies
Upgrade packages

The first step in this strategy ensures one project’s packages and upgrades won’t interfere with any other projects. Isolating projects is accomplished by creating per-project libraries. A tool that makes this easy is the new renv package. Inside of your R project, simply use:

# inside the project directory
renv::init()

The second step is to record the current dependencies. This step is critical because it creates a safety net. If the package upgrade goes poorly, you’ll be able to revert the changes and return to the record of the working state. Again, the renv package makes this process easy.

# record the current dependencies in a file called renv.lock
renv::snapshot()

# commit the lockfile alongside your code in version control
# and use this function to view the history of your lockfile
renv::history()

# if an upgrade goes astray, revert the lockfile
renv::revert(commit = "abc123")

# and restore the previous environment
renv::restore()

With an isolated project and a safety net in place, you can now proceed to upgrade or add new packages, while remaining certain the current functional environment is still reproducible. The pak package can be used to install and upgrade packages in an interactive environment:

# upgrade packages quickly and safely
pak::pkg_install("ggplot2")

The safety net provided by the renv package relies on access to older versions of R packages. For public packages, CRAN provides these older versions in the CRAN archive. Organizations can use tools like RStudio Package Manager to make multiple versions of private packages available. The “snapshot and restore” approach can also be used to promote content to production. In fact, this approach is exactly how RStudio Connect and shinyapps.io deploy thousands of R applications to production each day!

Team Collaboration

A common challenge on teams is sharing and running code. One strategy that administrators and R users can adopt to facilitate collaboration is shared baselines. The basics of the strategy are simple:

Administrators setup a common environment for R users by installing RStudio Server.
On the server, administrators install multiple versions of R.
Each version of R is tied to a frozen repository using a Rprofile.site file.

By using a frozen repository, either administrators or users can install packages while still being sure that everyone will get the same set of packages. A frozen repository also ensures that adding new packages won’t upgrade other shared packages as a side-effect. New packages and upgrades are offered to users over time through the addition of new versions of R.

Frozen repositories can be created by manually cloning CRAN, accessing a service like MRAN, or utilizing a supported product like RStudio Package Manager.

Adaptable Strategies

The prior sections presented specific strategies for creating reproducible environments in two common cases. The same strategy may not be appropriate for every organization, R user, or situation. If you’re a student reporting an error to your professor, capturing your sessionInfo() may be all you need. In contrast, a statistician working on a clinical trial will need a robust framework for recreating their environment. Reproducibility is not binary!

To help pick between strategies, we’ve developed a strategy map. By answering two questions, you can quickly identify where your team falls on this map and identify the nearest successful strategy. The two questions are represented on the x and y-axis of the map:

Do I have any restrictions on what packages can be used?
Who is responsible for managing installed packages?

For more information on picking and using these strategies, please visit https://environments.rstudio.com. By adopting a strategy for reproducible environments, R users, administrators, and teams can solve a number of important challenges. Ultimately, reproducible work adds credibility, creating a solid foundation for research, business applications, and production systems. We are excited to be working on tools to make reproducible work in R easy and fun. We look forward to your feedback, community discussions, and future posts.

Alternative Design for Shiny

Tue, 13 Mar 2018 00:00:00 +0000

Shiny’s Design

Most Shiny apps out there have a similar design style. It is usually easy for a seasoned Shiny developer to tell the difference between a Shiny app and a standard website. Why is this? Shiny apps ARE websites for all intents and purposes. Why do they not vary as greatly as the rest of the sites we encounter when surfing the web?

This is partly due to the fact that many Shiny developers leverage pre-written code (e.g., the layout was presented to them). There is another major factor here, though: the web framework that powers Shiny, called Bootstrap.

Bootstrap

Shiny implements the most widely used and established CSS, JavaScript, and HTML framework that exists today. Bootstrap was originally written for internal use at Twitter, and has since blossomed into the most utilized web framework across the web.

People with experience in Shiny will recognize these examples (think actionButton and selectInput) from the Bootstrap website:

Alternatives

While Bootstrap is the most widely used web framework, other frameworks do exist that are also appealing. There are too many good ones to name here, so I will go directly to the one which is the basis for this article: Material Design.

Material Design

Material Design is an extremely popular set of design standards created by Google. Anyone who uses Google web products (or Android) will quickly recognize some of their work.

Here is an example of some UI elements that adhere to Material Design:

Standards vs. Framework

Material Design is not the same thing as Bootstrap. Bootstrap is the design standards, along with the CSS, JavaScript, and HTML code written to implement those standards. Material Design is only the standards. While Google has the resources to implement the standards themselves, for an average developer this would be extremely difficult to take on. Thankfully, open-source development has a solution: an excellent framework called Materialize CSS.

Materialize CSS implements Google’s Material Design with such precision that is indistinguishable from Google’s own implementation. The UI elements found in Materialize CSS are slick and are suitable for any web site or application.

Material Design in Shiny

Let’s assume someone wants to implement Material Design in a Shiny app. Where does one start? Yes, Materialize CSS exists, but that is code written for the web and not for native R users (hence Shiny). This leads to, yes, an R package. The package is called shinymaterial, and it allows Shiny developers to implement Material Design using only R code. The package contains many of the standard types of Shiny inputs, but with its own API (e.g., material_select vs. selectInput). Shiny developers will quickly pick up the similarities between shinymaterial inputs and standard shiny inputs.

Along with inputs, the package also has more broad UI functionality, including the ability for Shiny developers to create Material dashboards, as shown here, along with other unique features such as parallax scrolling, shown here.

Conclusion

Shiny changed the game. It turned R programmers into web developers overnight. What shinymaterial aims to do is to provide Shiny developers with a little more design flexibility, while maintaining the spirit of Shiny: simple APIs that R developers can use to turn their scientific analyses into great web apps.

Recent R Data Packages

Wed, 01 Nov 2017 00:00:00 +0000

It has never been easier to access data from R. Not only does there seem to be a constant stream of new packages that access the APIs of data providers, but it is also becoming popular for package authors to wrap up fairly large datasets into R packages. Below are 44 R packages concerned with data in one way or another that have made it to CRAN over the past two months.

alphavantager v0.1.0: Implements an interface to the Alpha Vantage API to fetch historical data on stocks, physical currencies, and digital/crypto currencies. See the README to get the key.

AmesHousing v0.0.2: Contains raw and processed versions of the Ames Iowa Housing Data. See De Cock (2011).

billboard v0.1.0: Contains data sets regarding songs on the Billboard Hot 100 list from 1960 to 2016, including ranks for the given year, musical features, and lyrics.

BIS v0.1.0: Provides an interface to data provided by the Bank for International Settlements. There is a vignette.

bomrang v0.0.8: Provides functions to interface with Australian Government Bureau of Meteorology data. There is an Introduction and an Example.

brazilmaps v0.1.0: Enables users to obtain Brazilian map spatial objects of varying region types, e.g., cities, states, microregions, and mesoregions.

census v0.2.0: Provides functions to scrape US Census data from the American Community Survey (ACS) data and metadata. There is a brief vignette.

cptec v0.1.0: Retrieves data from the CPTEC/INPE, Centro de Previsão de Tempo e Estudos Climáticos Instituto Nacional de Pesquisas Espaciais, weather and climate forecasting center in Latin America.

coinmarketcapr v0.1 Provides functions to extract and monitor price and market cap of ‘Crypto currencies’ from Coin Market Cap.

copulaData v0.0-1: Contains data sets used for copula modeling in addition to those in the package copula.

CytobankAPIstats v1.0: Provides tools to access and process cytometry data from Cytobank.

data360r v1.0.1: Provides an interface to the API for the World Bank’s TCdata360 and Govdata360 platforms.

DescriptiveStats.OBeu v1.2.1: Provides functions to estimate and return the needed parameters for visualizations designed for OpenBudgets.eu datasets. There is a Getting Started Guide and two other vignettes.

dobson v0.1: Contains example data sets from the book An Introduction to Generalised Linear Models by Dobson and Barnett.

ensemblepp v0.1-0: Contains temperature and precipitation ensemble weather forecasts and observations taken at Innsbruck, Austria, which are contained in the forthcoming book (Elsevier) “Statistical Post processing of Ensemble Forecasts” by Vannitsem, Wilks, and Messner.

fedreporter v0.2.1: Implements an API to the Federal RePORTER, allowing users to search job projects from different government agencies.

fingertipsR v0.1.3: Provides an interface to the API for Fingertips, which contains data for many indicators of public health in England. There are vignettes for plotting Life Expentancy and for Interactively Selecting Indicators.

GetITRData v0.6: Provides functions to read quarterly and annual financial reports - including assets, liabilities, income, and cash flow statements - from Bovespa’s ITR (informacoes trimestrais) system. There is a vignette.

GetLattesData v0.6: Implements an API for downloading and reading XML data directly from Lattes. There is a vignette.

IIS v1.0: Contains the datasets and functions form the book Intuitive Introductory Statistics.

jaod v0.1.0: Provides a client for the Directory of Open Access Journals (DOAJ). See the API documentation and README file.

LAGOSNE v1.00: Provides a client for programmatic access to the Lake Multi-scaled Geospatial and Temporal database, with functions for accessing lake water quality and ecological context data for the US. There is a vignette on the Structure of LAGOSNE and another for Working with LAGOSNE. The following map shows chlorophyll concentrations in Pennsylvania.

lexiconPT v0.1.0: Provides access to Portuguese lexicons for sentiment analysis.

mapsapi v0.1.0: Provides an interface to the Google Maps API. See the vignette to get started.

matchbook v1.0.7: Provides a wrapper for the Matchbook API.

microdemic v0.1.0: Provides programmatic access to scholarly articles in the Microsoft Academic Graph.

mozzie v0.1.0: Contains weekly dengue cases in 25 districts of Sri Lanka from 2008/ week-52 to 2014/ week-21.

nyctaxi v0.0.1: Provides an interface to New York City’s Taxi and Limousine Commission Trip Data. The vignette describes how to get started.

opendotaR v0.1.4: Provides access to the OpenDota API.

PakPC2017 v0.3.0: Provides data sets and functions for exploration of the Pakistan Population Census 2017.

PakPMICS2014Ch v0.1.0: Provides a data set and function for exploring the Multiple Indicator Cluster Survey: 2014 Children Punjab, Pakistan.

PakPMICS2014HH v0.1.0: Provides a data set and function for exploring the Multiple Indicator Cluster Survey: 2014 Households Punjab, Pakistan.

PakPMICS2014HL v0.1.0: Provides a data set and function for exploring the Multiple Indicator Cluster Survey: 2014 Household Listings Punjab, Pakistan.

PakPMICS2014Wm v0.1.0: Provides a data set and function for exploring the Multiple Indicator Cluster Survey: 2014 Women (age 15-49 years) Punjab, Pakistan.

pder v1.0-0: Provides data sets for the Panel Date Econometrics with R book.

realestateDK v0.1.0: Provides quarterly information on Denmark Housing Market Statistics, including average square meter prices and the number of free trades for parcel and terraced houses, condominiums, and holiday homes in Denmark since 1992.

rcongresso v0.1.3: Wraps the API for the Brazilian Chamber of Deputies. There is an Introduction and a vignette for using the package with purrr.

repurrrsive v0.1.0: Contains recursive lists in the form of R objects, ‘JSON’, and ‘XML’, for use in teaching with examples, including color palettes, Game of Thrones characters, GitHub repositories, entities from the Star Wars universe, and more.

spatstat.data v1.1-1: Contains datasets for the spatstat package.

statsDK v0.1.1: Provides a wrapper for the API to Statistics Denmark. Have a look at the API Console and the vignette to get started with the package.

SPYvsSPY: Contains historical data from the legendary Mad magazine comic strip.

UdderQuarterInfectionData v1.0.0: Provides the udder quarter infection data set, which contains infection times of individual cow udder quarters with Corynebacterium bovis. See Laevens et al. 1997.

USCF v0.1.3: Provides a function to retrieve information from the U.S. Chess Federation website.

XKCDdata v0.1.0: Provides a function to download data from individual XKCD comics, by Randall Munroe.

Control Systems Toolbox – System Interconnection

Thu, 24 Aug 2017 00:00:00 +0000

Introduction

Dynamic systems are usually represented by a model before they can be analyzed computationally. These dynamic systems are systems that change, evolve or have their states altered or varied with time based on a set of defined rules. Dynamic systems could be mechanical, electrical, electronic, biological, sociological, and so on. Many such systems are usually defined by a set rules that are represented as a set of nonlinear differential equations. A generic first-order differential form is as shown below: \[ \frac{dx}{dt} = f ( x(t), u(t), t ) ........... (1) \]

where x(t) is the state vector representing the configuration (or state) of the system at the time, t, u(t) represents the external control input at time, t, and f is a function that gives the rate of change of the state vector for a specific input, state and time.

The system of (1) could be said to be time invariant if we assume that f does not explicitly depend on time. This implies that the relation of (1) becomes:

\[ \frac{dx}{dt} = f (x, u) ........... (2) \]

For such time-invariant systems, the control input may remain time-variant, but the coefficients of the function f are constant since the underlying physical laws themselves are not usually time-dependent. Over a sufficiently small operating range, the dynamics of most systems are approximately linear, and the following relation entails a linear time-invariant (LTI) system:

\[ \frac{dx}{dt} = Ax + Bu ........... (3) \]

Several engineering challenges till today are being solved using LTI techniques, while most results obtained from control theory are based on the described assumptions associated with LTI systems.

Laplace Transform

For those systems that are described by differential equations, it is possible to convert the system’s time-domain model into a frequency-domain (s-domain) representation. Two striking benefits for doing this are:

the differential equations that define the system are transformed into algebraic equations, which are often easier to analyze
The frequency response of the system can be obtained

This frequency-domain output / input representation is called the transfer function. Many control system models are available in transfer function representation, and this model is mostly used in the block diagram representation of control systems.

Figure 1 shows the block diagram for the pitch-control system of an Unmanned Free Swimming Submersible (UFSS) vehicle. This system and its block diagram shall be detailed in the Case Study section of this article.

Transfer Function Expressions

A transfer function expression is an algebraic expression in the frequency-domain. It is usually the ratio of the output to the input of a control system. Evaluating such expressions while retaining their mathematical form (as in symbolic mathematics) should be possible using a scientific computing language. An example of such expression is shown below:

\[ G(s) = \frac{Output}{Input} = \frac{s^2 + 3s + 4}{2s^3 + 4s^2 + 5s + 1} \]

Models of physical systems often appear as transfer functions with the coefficients of state as electrical, mechanical, chemical, or other physical quantities. Users would find it convenient to simply input these transfer functions as-is into a computer, and quickly analyze the system.

In the following mechanical example of a mass-spring-damper system of [1], the transfer function contains symbols m, b and k, which represent mass, damping constant, and spring constant variables respectively, as shown below:

Figure 2: Mass-Spring-Damper diagram \[ \frac{X(s)}{F(s)} = \frac{1}{ms^2+bs+k} \]

where m = 1; k = 1; b = 0.2

In this Google Summer of Code (GSoC) 2017 project, we have tried to include such capability within the control library, making it possible to create a transfer function model from the evaluation of a rational expression (the transfer function). A function named TF() was created to accept transfer function expressions, and return a transfer function model after evaluation. Therefore, to create this mass-spring-damper model within R, use the control library as shown in the snippet below:

 m <- 1;  k <- 1;  b <- 0.2
TF("1/(m*s^2 + b *s + k)")

## 
## y1:
##                 1 
##   - - - - - - - - - - -
##      s^2 + 0.2 s + 1 
## 
## 
## Transfer Function: Continuous time model

Electric Motor Example

Consider the following example from a University of Michigan course page on Control Systems using MATLAB – an example of an electric motor [2]

Figure 3: showing electric motor physical model [2]

Wikipedia: A DC motor is any class of rotary electric machines that converts direct current electrical energy into mechanical energy. It can provide translational motion when coupled with wheel, drums and cables. DC motors are used in propulsion of electric vehicles, elevator and hoists, or in drives for steel rolling mills. Several applications are: air compressors, vacuum cleaner, hair drier, sewing machine, and even toys or any system that requires variable speed.

The physical parameters for this example are [3]:

Moment of inertia of the rotor, J = 3.2284E-6 kg.m^2

Motor viscous friction constant, b = 3.5077E-6 N.m.s

Electromotive force constant, Ke = 0.0274 V/rad/sec

Motor torque constant, Kt = 0.0274 N.m/Amp

Electric resistance, R = 4 Ohm

Electric inductance, L = 2.75E-6 H

where, Kt = Ke = K

The input of the system is the voltage source (V) applied to the motor’s armature, while the output is the rotational speed of the shaft.

Open-Loop Response of the electric motor

The open-loop transfer function for the DC motor described above in the s-domain is:

\[ P(s) = \frac{K}{(s(Js + b)(Ls + R) + K^2)} \]

The step function is usually used to analyze the system’s step response, so it is applied here to analyze the open-loop response of the system. Using the control library, this model in the s-domain could be created and analyzed in R as follows:

 J <- 3.2284E-6;
b <- 3.5077E-6;
K <- 0.0274;
R <- 4;
L <- 2.75E-6;
P_motor <- TF("K/(s*((J*s + b)*(L*s + R) + K^2))")
time <- seq(0,0.2,0.001)
stepplot(P_motor, t = time)

Figure 1: Figure 4: Electric Motor Open-loop Step Response

From the above plot showing the system’s step response, it is clear that when a step input (of 1 volt) is applied to the system, the electric motor’s position grows unbounded - system instability.

The stability of the system can also be determined from the poles of the transfer function using the pole function:

pole(P_motor)

##               [,1]
## [1,]  0.000000e+00
## [2,] -1.454487e+06
## [3,] -5.922604e+01

As shown above, one of the poles of the open-loop transfer function is on the imaginary axis while the other two poles are in the left half of the complex s-plane. A pole on the imaginary axis indicates that the free response of the system will not only grow unbounded, but also will not decay to zero [3].

Closed-Loop Response of the electric motor

We can add feedback the motor model with a controller set to 1, and then investigate the system’s closed-loop step response for stability. This can be obtained using the following R commands:

P_motor2 <- feedback(P_motor, 1)
print(P_motor2)

## 
## y1:
##                                                     0.0274 
##   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
##     8.8781e-12 s^3 + 1.291361e-05 s^2 + 0.0007647908 s + 0.0274 
## 
## 
## Transfer Function: Continuous time model

# OR

P_motor2 <- cloop(P_motor, -1)
print(P_motor2)

## 
## y1:
##                                                     0.0274 
##   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
##     8.8781e-12 s^3 + 1.291361e-05 s^2 + 0.0007647908 s + 0.0274 
## 
## 
## Transfer Function: Continuous time model

stepplot(P_motor2, t = time)

Figure 2: Figure 5: Electric Motor Closed-loop Step Response

The above closed-loop step response shows that the addition of feedback has stabilized the system.

The damping and natural frequencies associated with the poles of this closed-loop system could be obtained using the following damp function:

damp(P_motor2)

## 
##      Eigenvalue            Damping     Freq. (rad/s)    Freq. (Hz). 
##      ----------            -------     -------------    -----------
## -29.61     +     35.28j   0.64           46.06           0.46      
## -29.61     -     35.28j   0.64           46.06           0.46      
## -1454487.32 +      0.00j   1.00           1454487.32      0.00

## $omega
##              [,1]
## [1,] 4.606386e+01
## [2,] 4.606386e+01
## [3,] 1.454487e+06
## 
## $zeta
## [1] 0.642853 0.642853 1.000000

Furthermore, using the TF() function, transfer function models could be summed, subtracted, multiplied, and divided. This is exemplified in the following section.

System Interconnections

In many cases, systems or subsystems have to be interconnected to obtain a particular controlled or regulated activity. Interconnecting models of components allows you to easily construct large control system models.

Analyzing systems that are modeled in a block diagram should be easily done using a scientific computing tool like R. The time response or frequency response of a system (such as shown in Figure 1), can be obtained after the block diagram is simplified. Simplification means that the several components (blocks described by different transfer functions) that make up the complete system have to be algebraically reduced to one transfer function to represent the system. This mathematical reduction is done based on the rules of control theory.

Figure 6: Block Diagram Interconnection

For an example, Figure 6 shows the interconnection of dynamic system models of a plant P(s), a controller C(s), sensor dynamics S(s), and a filter F(s). To analyze the system requires constructing a single (equivalent) model that represents the entire closed-loop control system.

In the second coding phase of the GSoC 2017, the following model interconnection functions have been produced as shown in Figure 7 below:

Figure 7: System Model Interconnections

Append Connection using the `append` function

A number of systems, whether in transfer-function, state-space or zero-pole model, could be appended into one state-space model using the append function. The block diagram arrangement is found in Figure 7 (e).

The syntax is: append(sys1, sys2, …, sysN). The resulting model is in state-space form.

For example:

 sys1 <- ss(1,2,3,4)
 sys2 <- ss(2,3,4,5)
 sys3 <- ss(6,7,8,9)
 sys4 <- tf(1, c(1,2,5))
 append(sys1, sys2, sys3)
 append(sys1, sys2, sys4, sys3)

Series Connection using the `series` function

This function connects two systems in series, in transfer-function, state-space or zero-pole model, into one model. The block diagram arrangement is found in Figure 7 (a).

Syntax: series(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1*sys2”)

For now, TF(“sys1*sys2”) is only possible when sys1 and sys2 are transfer-function models

Parallel Connection using the `parallel` function

The parallel function connects two systems in parallel, in transfer-function, state-space or zero-pole model, into one model. The block diagram arrangement is found in Figure 7 (c).

Syntax: parallel(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1 + sys2”)

For now, TF(“sys1+ sys2”) is only possible when sys1 and sys2 are transfer-function models

Feedback Connection using the `feedback` function

This function sets up a closed-loop transfer function from output to input based on the standard configuration shown in Figure 7 (b).

Syntax: feedback(sys1, sys2)

Arithmetically, this could be achieved using the TF() function in the following manner:

TF(“sys1 / (1 + sys2*sys1)”), but this is not recommended in terms of accuracy. feedback(sys1, sys2) is preferred.

General block diagram building using `connect` function

Using the connect function, more complicated (especially multiple-input multiple-output (MIMO) ) block diagrams could be set up. For this case, all systems that are to be connected are first appended, then connected using their inputs and outputs.

Syntax: connect(append(sys1, sys2, sys3), connections, inputs, outputs)

Case Study – Unmanned Free Swimming Submersible (UFSS) Vehicle

The Unmanned Free-Swimming Submersible (UFSS) vehicle was one of the first autonomous underwater vehicles (AUVs). It was developed in the late 1970s as a test vehicle to demonstrate the potential of autonomous vehicles over long distances, and to study laminar flow [5]. A picture showing an Unmanned Free-Swimming Submersible (UFSS) vehicle is shown in Figure 8 below.

Figure 8: Unmanned Free Swimming Submersible Vehicle (UFSS). Source: [5]

Nise [4] relates the following concerning the UFSS:

The depth of the vehicle is controlled as follows. During forward motion, an elevator surface on the vehicle is deflected by a selected amount. This deflection causes the vehicle to rotate about the pitch axis. The pitch of the vehicle creates a vertical force that causes the vehicle to submerge or rise.

A detailed block diagram for the pitch control system is shown in Figure 9 below.

Figure 9: Block diagram for the pitch control system of UFSS

Rationale:

The subsystems of the UFSS vehicle are developed independently, and studied using computer simulation before integration into the vehicle [5]. Thus, for this reason, we are examining the pitch control system independently.

For this case study, we shall use the developing control library in R to:

Obtain the closed-loop UFSS pitch control system by block diagram reduction of the system in Figure 9 above, using series, parallel and feedback.
Obtain the step response of the pitch control system simplified by series, parallel and feedback
Obtain the closed-loop UFSS pitch control system by block diagram reduction of the system in Figure 9 above, using the algebraic operations provided by TF().
Obtain the step response of the pitch control system from the method of algebraic operations.

Solution using `series`, `parallel` and `feedback`

K1 <- 1
K2  <- 1
G1 <- tf(-K1,1)
G2 <- tf(c(0,2), c(1,2))

numg3 <- -0.125*c(1, 0.435)
deng3 <- pracma::polymul(c(1, 1.23), c(1, 0.226, 0.0169))
G3 <- tf(numg3, deng3)

H1 <- tf(c(-K2, 0), c(0, 1))

G4 <- series(G2, G3)
G5 <- feedback(G4, H1)
Ge <- series(G1, G5)
T1 <- feedback(Ge, 1)
stepplot(T1, t = seq(0,40,0.1))

Figure 3: Figure 10: R plot showing UFSS vehicle system response - Block diagram reduction by series, parallel and feedback

Solution via algebraic operations

G4 <- TF("G3*G2")
G5 <- TF("G4/(1+ G4*H1)")
Ge <- TF("G5*G1")
T2 <- TF("Ge/(1+Ge)")
stepplot(T2, t = seq(0,40,0.1))

Figure 4: Figure 11: R plot showing UFSS vehicle system response - Block diagram reduction by Algebraic Operations

Clearly, the step responses (Figure 10 and Figure 11) of the pitch control system using the two methods are identical to each other - a justification to the framework of the control systems toolbox.

Feedback

Interested users are welcome to install the development version of this package using devtools::install_github("benubah/control"), and feel free to get in touch for any related issues.

Conclusion

In this article, we have discussed on our progress with the Control Systems Toolbox in R as a GSoC 2017 project. We have highlighted the use of this toolbox to carry out the following tasks:

Create Transfer function models by evaluating transfer function expressions in the s-domain
Obtain the poles and their associated damping and natural frequencies of a system
Open-loop and closed-loop responses
System Model Interconnections using series, parallel and feedback and algebraic functions in the UFSS vehicle case study.

Other functions that have been developed in this GSoC phase are: givens_rot, ordschur, care, ctrb, acker, place.

You may want to read about what was achieved in the first GSoC coding phase here.

References

http://ctms.engin.umich.edu/CTMS/index.php?example=Introduction&section=SystemModeling
http://ctms.engin.umich.edu/CTMS/index.php?example=MotorSpeed&section=SystemModeling
http://ctms.engin.umich.edu/CTMS/index.php?example=MotorPosition&section=SystemAnalysis
Norman, S. Nise, Control Systems Engineering, 6th. Ed, 2011.
http://www.navalunderseamuseum.org/ufss/

End-to-end visualization using ggplot2

Mon, 14 Aug 2017 00:00:00 +0000

ggplot2 is kind of a household word for R users. I’ve ended up using it for complex data munging and wrangling work, where I needed to get clarity on different aspects of the data, especially being able to get different views, slices and dices of it, but in a nice visualization. At some point along the line, I slowly stopped using more traditional plotting functions like plot(), matplot(), barplot(), etc.

This article is an end-to-end data visualization exercise, using only ggplot2(). It has been helpful for me to see such pieces online on the endless possibilities of ggplot2(), so I wanted to give back to the community by doing one of my own.

1. Pima Indian Diabetes data

Consider the Pima Indian Diabetes dataset available in R. It looks at the population of women who were at least 21 years of age, of Pima Indian heritage and living near Phoenix, Arizona, and were tested for diabetes according to WHO criteria. In this exercise, I will use the 332 test data subjects. There are no missing values in this data. It is a very simple dataset, but my goal is to use it to demonstrate the tools available in ggplot2 to visually investigate a dataset we know very little about. This is part of the important data exploration phase of a data science project, to help prepare for the modeling phase.

library(MASS)
d <- Pima.te
summary(d)

##      npreg             glu              bp              skin      
##  Min.   : 0.000   Min.   : 65.0   Min.   : 24.00   Min.   : 7.00  
##  1st Qu.: 1.000   1st Qu.: 96.0   1st Qu.: 64.00   1st Qu.:22.00  
##  Median : 2.000   Median :112.0   Median : 72.00   Median :29.00  
##  Mean   : 3.485   Mean   :119.3   Mean   : 71.65   Mean   :29.16  
##  3rd Qu.: 5.000   3rd Qu.:136.2   3rd Qu.: 80.00   3rd Qu.:36.00  
##  Max.   :17.000   Max.   :197.0   Max.   :110.00   Max.   :63.00  
##       bmi             ped              age         type    
##  Min.   :19.40   Min.   :0.0850   Min.   :21.00   No :223  
##  1st Qu.:28.18   1st Qu.:0.2660   1st Qu.:23.00   Yes:109  
##  Median :32.90   Median :0.4400   Median :27.00            
##  Mean   :33.24   Mean   :0.5284   Mean   :31.32            
##  3rd Qu.:37.20   3rd Qu.:0.6793   3rd Qu.:37.00            
##  Max.   :67.10   Max.   :2.4200   Max.   :81.00

head(d)

##   npreg glu bp skin  bmi   ped age type
## 1     6 148 72   35 33.6 0.627  50  Yes
## 2     1  85 66   29 26.6 0.351  31   No
## 3     1  89 66   23 28.1 0.167  21   No
## 4     3  78 50   32 31.0 0.248  26  Yes
## 5     2 197 70   45 30.5 0.158  53  Yes
## 6     5 166 72   19 25.8 0.587  51  Yes

The target variable, type, tells us whether a patient is diabetic or not.

2. Distributions across categories

When the target is categorical, as in this case, type, I like to start by examining distributions for the continuous input columns. This gives us an overall sense of which input is likely to be useful. To do this, I like to do both boxplots and a density plot, since each has a different goal.

2.1. Boxplots

First, I’ll use boxplots, but ggplot2-style. I really like the look of a ggplot2() boxplot. It also allows me to seamlessly have multiple plots in a grid, as well as tinker around with the plotting parameters more flexibly than in a classical boxplot() approach, and end up with a nice-looking plot. We can see below how some inputs clearly vary across the 2 target categories, and others don’t.

df <- subset(d, select=c(glu, bp, skin, bmi, ped, age, type))  

library(gridExtra)
library(ggplot2)
p <- list()

for (j in colnames(df)[1:6]) {
  p[[j]] <- ggplot(data=df, aes_string(x="type", y=j)) + # Specify dataset, input or grouping col name and Y
            geom_boxplot(aes(fill=factor(type))) + guides(fill=FALSE) + # Boxplot by which factor + color guide
            theme(axis.title.y = element_text(face="bold", size=14))  # Make the Y-axis labels bigger/bolder
}

do.call(grid.arrange, c(p, ncol=3))

2.2. Density plots

I have used various overlay-density packages in the past, sm.density.compare() for example. I find the overlay-density rendering in ggplot2() to be more visually pleasing, with little plotting parameter tuning. E.g., it’s clear in the plot below that diabetic patients are associated with more number of pregnancies. I really like the alpha parameter.

df$npreg <- d$npreg
g <- ggplot(df, aes(npreg))
g + geom_density(aes(fill=factor(type)), alpha=0.8) + 
    labs(title="Density plot", 
         subtitle="# Pregnancies Grouped by Diabetes Type",
         x="# Pregnancies",
         fill="Diabetes Type")

3. Grid views

Next, I want to mix things up a little, so that I can have multi-dimensional views. By this, I mean that I want to know how the target is distributed across a few important inputs, but I want to link those inputs up as well. Sort of like a 3-way table, but visualized nicely instead of numbers. I came across this problem recently in one of the projects, and while it seems like a basic must-have output to dig deeper, I really needed something like ggplot2 to implement it. Using facet_grid() was amazing, even more so on account of the smooth control one has on the plotting parameters within a ggplot2 setup.

3.1. Data preparation

Facet-wrapping and gridding is a must-have tool for deeper data views, but the process is a multi-step one. Not too complicated though - very intuitive under ggplot2. We start with creating some new categorical columns using the continuous ones. Note that this can be done in different ways: appending new columns directly to the data frame, or using the more sleeker dplyr() in combination with magrittr(), which I absolutely love. This integrates a number of operations into a single chunk, making it quite seamless. I am also loading up plyr(), since I will be using it later.

library(magrittr)
library(plyr)
library(dplyr)
df_grid <- d %>% 
          mutate(Skin = ifelse(d$skin <= 29, "low skin fold", "high skin fold"),
                  BMI = ifelse(d$bmi <= 33, "low BMI", "high BMI"),
                  Ped = ifelse(d$ped <= 0.31, "low pedigree",
                ifelse(d$ped > 0.3134 & d$ped <= 0.5844, "medium pedigree", "high pedigree"))) %>% 
  
            mutate(Ped = factor(Ped, levels = c("low pedigree", "medium pedigree", "high pedigree")))

3.2. Reshaping the data

Next, we need to prepare the data a little more before throwing it into the facet_grid() mix. Most importantly, we need to “reshape” it, i.e., while our data is a “wide”-form data frame, we need to convert this to a “long”-form to enable facet_grid() to easily pick up what it needs to “facet” the plot by. We will also add a “size” column - this will allow us to make more granular adjustments in our plot. I will also rename columns in order to enable easier axis labeling when plotting. Again, notice that instead of using reshape2(), which I have used for many years, we’re using gather() from tidyr(), all sewn together with the pipe in magrittr().

library(tidyr)
DF <- df_grid %>% 
    subset(select=c(type, Skin, BMI, Ped)) %>% 
    gather(variable, value, -c(Skin, Ped, BMI))

colnames(DF)[5] <- "Diabetes_Value"
DF$size <- rep(1.5, nrow(DF))
s <- 1.5

3.3. Facet Grid

We’ll try the basic facet_grid() plot, after which we’ll go in and make some adjustments. For now, our goal is the following: to see a “matrix” or “grid” of the BMI distribution across diabetes type, as a 2x2 table of pedigree/skin fold combinations. In other words, for low pedigree/low skin fold, how does BMI distribute across diabetes type? You can see the amount of information you can pack into just one plot. I have found this to be useful when presenting to an end-user or customer. It becomes all the more useful since its a very clear representation of this slice/dice, with little room for ambiguity.

# Simple
library(ggplot2)
ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  facet_grid(Skin ~ Ped)   # wrap up everything to showcase by multiple cols

This looks nice, but I would like to add more of a “pop”. I am going to outline each box, and bolden the fonts. Note that you can also color the “grid strips”, but I won’t do that right now.

# More color
p <- ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  geom_rect(aes(fill=NA, size=size),xmin =-Inf,xmax=Inf,ymin=-Inf,ymax=Inf,alpha = 0.0002, colour="black",show.legend = F) +   # use box drawn around each location to cleanly separate facets + suppress guide
  scale_size(range=c(s,s), guide=FALSE) + # use line width/size feature for cleaner plotting
  facet_grid(Skin ~ Ped) +   # wrap up everything to showcase by multiple cols
  theme(strip.text.x = element_text(face="bold", size=12)) +
  theme(strip.text.y = element_text(face="bold", size=12)) 
  # optional changes in strip
#+ theme(strip.text.x = element_text(face="bold", size=12, colour="white")) +
#  theme(strip.text.y = element_text(face="bold", size=12, color="white")) +
#  theme(strip.background = element_rect(fill="black"))
plot(p)

Much better. Look how nicely this granular plot adjustment in ggplot2 allows each “block” in the matrix to pop out. Its very clear how BMI is distributed across diabetes type, and how that in turn is distributed across both pedigree function and skin fold. We see that (as expected): 1. A higher triceps skin fold thickness is associated with a higher BMI, as well as a higher count of diabetic people. 2. The above is more true for a higher diabetes pedigree function.

This kind of a grid plot presents a very powerful tool for such multi-dimensional data views.

4. Heatmaps

I like heatmaps - there’s a sense of drama in the way you can see where “something is happening”. I’ve used heatmap.2() to implement hierarchical clustering and translating that to a heatmap. But I wanted to use ggplot2() to simply look at a dataset as a heatmap, without any underlying analysis, to detect patterns before any analysis begins.

In this case, I want ggplot2() to show me patterns across different input columns, for the two diabetes types, i.e., what inputs seem to differ across diabetic/non-diabetic patients. This will be clear once we render our dataset into a nice ggplot2() heatmap.

4.1. Data preparation

As usual, we need to prep our data before pushing it into the ggplot2() function. We’ll reshape and scale the data first, all within the plyr(), dplyr(), and magrittr() framework. I’ll also specify some plotting parameters that I will call into my ggplot2() function. I’m going to rely on RColorBrewer() for these.

df_heat <- d[order(d$type),1:8]
DF_Heat <- df_heat %>%
          mutate(id = 1:nrow(df_heat)) %>%
          select(c(npreg:age, id))  %>%
          gather(variable, value, -id)  %>%
          ddply(.(variable), transform,
                    rescale = scale(value))  # Notice that this reorders by "variables"
          
# Color scale for heatmap
library(RColorBrewer)
colors <- brewer.pal(9, 'Reds')

# Lines to split patients into diabetic/non-diabetic
my.lines <- data.frame(x1 = 0.5, x2 = 7.5, y1 = 223.5, y2 = 223.5)

4.2. Rendering the heatmap

# Basic plot
p <- ggplot(DF_Heat, aes(as.factor(variable), as.factor(id), group=id)) + 
  geom_tile(aes(fill = rescale),colour = "white") +
  scale_fill_gradient(low="green", high="red")

# Make adjustments
base_size <- 9
p_adj <- p + theme_grey(base_size = base_size) + labs(x = "",y = "") + scale_x_discrete(expand = c(0, 0)) +
  scale_y_discrete(expand = c(0, 0)) +
  geom_segment(data=my.lines, aes(x = x1, y = y1, xend=x2, yend=y2), size=1, inherit.aes=F) +
  theme(axis.text.y = element_blank(), axis.ticks.y = element_blank()) 
plot(p_adj)

Note that I have suppressed the ticks on the Y-axis. We can clearly see regions of interest on the heatmap. It would be better for these to easily pop out at the viewer, to enable which, I am going to invoke geom_rect().

# Borders of rectangles to indicate areas of interest on heatmap
my.lines.rect.1 <- data.frame(xmin = 1.5, xmax = 2.5, ymin = 223.5, ymax = 255.5)
my.lines.rect.2 <- data.frame(xmin = 3.5, xmax = 4.5, ymin = 223.5, ymax = 332)
my.lines.rect.3 <- data.frame(xmin = 5.5, xmax = 6.5, ymin = 223.5, ymax = 280.5)

p_adj + geom_rect(data=my.lines.rect.1, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=2, inherit.aes = F) +
  geom_rect(data=my.lines.rect.2, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=5, inherit.aes = F) +
  geom_rect(data=my.lines.rect.3, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=4, inherit.aes = F)

Much better.

5. Segmentation in a scatterplot

Finally, I want to try to implement some “basic-level clustering”. This is not model-based clustering; rather, it is simply using a scatterplot and a few nice plotting parameters in ggplot2() to make some things pop right out at the viewer - again, with little room for ambiguity. What I like most here is the boxes that we can draw nicely to showcase the “clusters” a little better, along-with the multi-layered information, e.g., age, BMI, glucose, etc.

The conclusions are logical and obvious from the following plot, but quite nicely illustrate the use of ggplot2() for such a specific purpose.

d$Age <- ifelse(d$age < 30, "<30 yrs", ">= 30 yrs")

ggplot(d, aes(x = glu, y = bmi)) +
  geom_rect(aes(linetype = "High BMI - Diabetic"), xmin = 160, ymax = 40, fill = NA, xmax = 200, 
            ymin = 25, col = "black") + 
  geom_rect(aes(linetype = "Low BMI - Not Diabetic"), xmin = 0, ymax = 25, fill = NA, xmax = 120, 
            ymin = 10, col = "black") + 
  geom_point(aes(col = factor(type), shape = factor(Age)), size = 3) +
  scale_color_brewer(name = "Type", palette = "Set1") +
  scale_shape(name = "Age") +
  scale_linetype_manual(values = c("High BMI - Diabetic" = "dotted", "Low BMI - Not Diabetic" = "dashed"),
                        name = "Segment")

Hopefully, this little exercise will be helpful for someone wanting to use ggplot2() for an innovative slice/dice of a complex dataset, and to visualize it nicely.

June 2017 New Package Picks

Wed, 26 Jul 2017 00:00:00 +0000

Two hundred and thirty-eight new packages were added to CRAN in June. Below are my picks for the “Top 40”, organized into six categories: Biostatistics, Data, Machine Learning, Miscellaneous, Statistics and Utilities. Some packages, including geofacet and secret, already seem to be gaining traction.

Biostatistics

BIGL v1.0.1: Implements response surface methods for drug synergy analysis, including generalized and classical Loewe formulations and the Highest Single Agent methodology. There are vignettes on Methodology and Synergy Analysis.

colorpatch v0.1.2: Provides functions to show color patches for encoding fold changes (e.g., log ratios) and confidence values within a diagram; especially useful for rendering gene expression data and other types of differential experiments. See the vignette.

eesim v0.1.0: Provides functions to create simulated time series of environmental exposures (e.g., temperature, air pollution) and health outcomes for use in power analysis and simulation studies in environmental epidemiology. The vignette gives an overview of the package.

personalized v0.0.2: Provides functions for fitting and validating subgroup identification and personalized medicine models under the general subgroup identification framework of Chen et al. The vignette provides a brief tutorial.

tidygenomics v0.1.0: Provides method to deal with genomic intervals the “tidy way”. The vignette explains how they work.

Data

alfred v0.1.1: Provides direct access to the ALFRED and FRED databases. The vignette gives a brief example.

CityWaterBalance v0.1.0: Provides functions to retrieve data and estimate unmeasured flows of water through an urban network. Data for US cities can be gathered via web services using this package and dependencies. See the vignette for an introduction to the package.

censusapi v0.2.0: Provides a wrapper for the U.S. Census Bureau APIs that returns data frames of census data and metadata. Available data sets include the Decennial Census, American Community Survey, Small Area Health Insurance Estimates, Small Area Income and Poverty Estimates, and Population Estimates and Projections. There is a brief vignette.

dataverse v0.2.0: Provides access to Dataverse version 4 APIs, enabling data search, retrieval, and deposit. There are four vignettes: Introduction, Search and Discovery, Retrieval and Data Archiving.

data.world v1.1.1: Provides high-level tools for working with data.world data sets. There is a Quickstart Guide and a vignette for writing Queries.

SimMultiCorrData v0.1.0: Provides functions to generate continuous, binary, ordinal, and count variables with a specified correlation matrix that can be used to simulate data sets that mimic real-world situations (e.g., clinical data sets, plasmodes). There are several vignettes including an Overall Workflow for Data Simulation and a Comparison to Other Packages.

tidycensus v0.1.2: Provides an integrated R interface to the decennial US Census and American Community Survey APIs, and the US Census Bureau’s geographic boundary files.

ukbtools v0.9.0: Provides tools to work with UK Biobank datasets. The vignette shows how to get started.

wpp2017 v1.0-1: Provides and interface to data sets from the United Nation’s World Population Prospects 2017.

Machine Learning

cld3 v1.0: Provides an interface to Google’s experimental Compact Language Detector 3 algorithm, a neural network model for language identification that is the successor of cld2.

datafsm v0.2.0: Implements a method that automatically generates models of dynamic decision-making that both have strong predictive power and are interpretable in human terms. The vignette provides an example.

diceR v0.1.0: Provides functions for cluster analysis using an ensemble clustering framework. The vignette shows some examples.

glmertree v0.1-1: Implements recursive partitioning based on (generalized) linear mixed models (GLMMs) combining lmer() and glmer() from lme4 and lmtree() and glmtree() from partykit. The vignette shows an example.

greta v0.2.0: Lets users write statistical models in R and fit them by MCMC on CPUs and GPUs, using Google TensorFlow. There is a website, a Getting Started Guide, and vignettes providing Examples andTechnical Details.

penaltyLearning v2017.07.11: Implements algorithms from Learning Sparse Penalties for Change-point Detection using Max Margin Interval Regression. There is a vignette.

SentimentAnalysis v1.2-0: Implements functions to perform sentiment analysis of textual data using various existing dictionaries, such as Harvard IV, or finance-specific dictionaries, and create customized dictionaries. The vignette provides an introduction.

Miscellaneous

convexjlr v0.5.1: Provides a high-level wrapper for Julia package Convex.jl, which makes it easy to describe and solve convex optimization problems. There is a very nice vignette that shows how to optimize the parameters for several machine learning models.

interp v1.0-29: Implements bivariate data interpolation on both regular and irregular grids using either linear methods or splines.

pkggraph v0.2.0: Allows users to interactively explore and plot package dependencies for CRAN.

parallelDist v0.1.1: Provides a parallelized alternative to R’s native dist function to calculate distance matrices for continuous, binary, and multi-dimensional input matrices with support for a broad variety of distance functions from the stats, prox and dtw R packages. The vignette offers some results on performance.

Stats

anchoredDistR v1.0.3: Supplements the MAD# software that implements the Method of Anchored Distributions for inferring geostatistical parameters. There is a vignette.

bssm v01.1-1: Efficient methods for Bayesian inference of state space models via particle Markov chain Monte Carlo and importance sampling type corrected Markov chain Monte Carlo. There is a vignette on Bayesian Inference of State Space Models and an example of a Logistic Growth Model.

factorMerger v0.3.1 Provides a set of tools to support results of post-hoc testing and enable to extract hierarchical structure of factors. There is an Introduction and vignettes on Cox Regression Factor Merging and Multidimensional Gaussian Merging.

MittagLeffleR v0.1.0: Provides density, distribution, and quantile functions as well as random variate generation for the Mittag-Leffler distribution based on the algorithm by Garrappa. There are short vignettes for the math, distribution functions and random variate generation.

walker v0.2.0: Provides functions for building dynamic Bayesian regression models where the regression coefficients can vary over time as random walks. The vignette shows some examples.

Utilities

charlatan v0.1.0: Provides functions to make fake data, including addresses, person names, dates, times, colors, coordinates, currencies, DOIs, jobs, phone numbers, ‘DNA’ sequences, doubles and integers from distributions and within a range. The Introduction will get you started.

colordistances v0.8.0: Provides functions to load and display images, selectively mask specified background colors, bin pixels by color, quantitatively measure color similarity among images,and cluster images by object color similarity. There is an Introduction and vignettes on Pixel Binning Methods and Color Distance Metrics.

dbplyr v1.1.0: Implements a dplyr back end for databases that allows working with remote database tables as if they are in-memory data frames. There is an Introduction, a vignette for Adding a new DBI backend and one for SQL translation.

geofacet v0.1.5: Provides geofaciting functionality (the ability to arrange a sequence of plots for different geographical entities into a grid that preserves some geographical orientation) for ggplot2. There is a Package Reference vignette and an Introduction. The package is already getting some traction. This is a user submission:

ggformula v0.4.0: Provides a formula interface to ggplot2. There is a vignette explaining how it works.

gqlr v0.0.1: Provides an implementation of the GraphQL query language created by Facebook for describing data requirements on complex application data models. gqlr should be useful for integrating R computations into production applications that use GraphQL.

later v0.3: Allows users to execute arbitrary R or C functions some time after the current time, after the R execution stack has emptied. The vignette shows how to use later from C++.

secret v1.0.0: Allows sharing sensitive information like passwords, API keys, etc., in R packages, using public key cryptography. There is a vignette.

sessioninfo v1.0.0: Provides functions to query and print information about the current R session. It is similar to utils::sessionInfo(), but includes more information.

webglobe v1.0.2: Provides functions to display geospatial data on an interactive 3D globe. There is a vignette

Some Ideas for your Internal R Package

Wed, 19 Jul 2017 00:00:00 +0000

At RStudio, I have the pleasure of interacting with data science teams around the world. Many of these teams are led by R users stepping into the role of analytic admins. These users are responsible for supporting and growing the R user base in their organization and often lead internal R user groups.

One of the most successful strategies to support a corporate R user group is the creation of an internal R package. This article outlines some common features and functions shared in internal packages. Creating an R package is easier than you might expect. A good place to start is this webinar on package creation.

Logos and Custom CSS

Interestingly, one powerful way to increase the adoption of data science outputs - plots, reports, and even slides - is to stick to consistent branding. Having a common look and feel makes it easier for management to recognize the work of the data science team, especially as the team grows. Consistent branding also saves the R user time that would normally be spent picking fonts and color schemes.

It is easy to include logos and custom CSS inside of an R package, and to write wrapper functions that copy the assets from the package to a user’s local working directory. For example, this wrapper function adds a logo from the RStudioInternal package to the working directory:

getLogo <- function(copy_to = getwd()){
      copy_to <- normalizePath(copy_to)
      file.copy(system.file(“logo.png”, package = “RStudioInternal”) , copy_to)
}

Once available, logos and CSS can be added to Shiny apps and R Markdown documents.

ggplot2 Themes

Similar to logos and custom CSS, many internal R packages include a custom ggplot2 theme. These themes ensure consistency across plots in an organization, making data science outputs easier to recognize and read.

ggplot2 themes are shared as functions. To get started writing a ggplot2 theme, see resource 1. For inspiration, take a look at the ggthemes package.

Data Connections

Internal R packages are also an effective way to share functions that make it easy for analysts to connect to internal data sources. Nothing is more frustrating for a first time R user than trying to navigate the world of ODBC connections and complex database schemas before they can get started with data relevant to their day-to-day job.

If you’re not sure where to begin, look through your own scripts for common database connection strings or configurations. db.rstudio.com can provide more information on how to handle credentials, drivers, and config files.

learnr Tutorials

RStudio recently released a new package for creating interactive tutorials in R Markdown called learnr. There are many great resources online for getting started with R, but it can be useful to create tutorials specific to your internal data and domain. learnr tutorials can serve as training wheels for the other components of the internal R package or teach broader concepts and standards accepted across the organization. For example, you might provide a primer that teaches new users your organization’s R style guide.

Wrap Up

If you are leading an R group, an internal R package is a powerful way to support your users and the adoption of R. Imagine how easy it would be to introduce R to co-workers if they could connect to real, internal data and create a useful, beautiful plot in under 10 minutes. Investing in an internal R packages makes that on boarding experience possible.

May New Package Picks

Fri, 23 Jun 2017 00:00:00 +0000

Two hundred and twenty-nine new packages were submitted to CRAN in May. Here are my picks for the “Top 40”, organized into five categories: Data, Data Science and Machine Learning, Education, Miscellaneous, Statistics and Utilities.

Data

angstroms v0.0.1: Provides helper functions for working with Regional Ocean Modeling System (ROMS) output.

bikedata v0.0.1: Download and aggregate data from public bicycle systems from around the world. There is a vignette.

datasauRus https://CRAN.R-project.org/package=datasauRus v0.1.2: The Datasaurus Dozen is a set of datasets that have the same summary statistics, despite having radically different distributions. As well as being an engaging variant on the Anscombe’s Quartet, the data is generated in a novel way through a simulated annealing process. Look here for details, and in the vignette for examples.

dwapi v0.1.1: Provides a set of wrapper functions for data.world’s REST API. There is a quickstart guide.

HURDAT v0.1.0: Provides datasets from the Hurricane Research Division’s Hurricane Re-Analysis Project, giving details for most known hurricanes and tropical storms for the Atlantic and northeastern Pacific ocean (northwestern hemisphere). The vignette describes the datasets.

neurohcp v0.6: Implements an interface to the Human Connectome Project. The vignette shows how it works.

osmdata v0.0.3: Provides functions to download and import of OpenStreetMap data as ‘sf’ or ‘sp’ objects. There is an Introduction and a vignette describing Translation to Simple Features.

parlitools v0.0.4: Provides various tools for analyzing UK political data, including creating political cartograms and retrieving data. There is an Introduction, and vignettes on the British Election Study, Mapping Local Suthorities, and Using Cartograms.

rerddap v0.4.2: Implements an R client to NOAA’s ERDDDAP data servers. There is an Introduction.

soilcarbon v1.0.0: Provides tools for analyzing the Soil Carbon Database created by Powell Center Working Group. The vignette launches a local Shiny App.

suncalc v0.1: Implements an R interface to the ‘suncalc.js’ library, part of the SunCalc.net’s project for calculating sun position, sunlight phases, moon position and lunar phase for the given location and time.

Data Science and Machine Learning

EventStudy v0.3.1: Provides an interface to the EventStudy API. There is an Introduction, and vignettes on Preparing EventStudy, parameters, and the RStudio Addin.

kmcudaR v1.0.0: Provides a fast, drop-in replacement for the classic K-means algorithm based on Yingyang K-Means. Look here for details.

openEBGM v0.1.0: Provides an implementation of DuMouchel’s Bayesian data mining method for the market basket problem. There is an Introduction, and vignettes for Processing Raw Data, Hyperparameter Estimation, Empirical Bayes Metrics, and Objects and Class Functions.

spacyr v0.9.0: Provides a wrapper for the Python spaCy Natural Language Processing library. Look here for help with installation and use.

Education

learnr v0.9: Provides functions to create interactive tutorials for learning about R and R packages using R Markdown, using a combination of narrative, figures, videos, exercises, and quizzes. Look here to get started.

olsrr v0.2.0: Provides tools for teaching and learning ordinary least squares regression. There is an Introduction and vignettes on Heteroscedascitity, Measures of Influence, Collinearity Diagnostics, Residual Diagnostics and Variable Selection Methods.

rODE v0.99.4: Contains functions to show students how an ODE solver is made and how classes can be effective for constructing equations that describe natural phenomena. Have a look at the free book Computer Simulations in Physics. There are several vignettes providing brief examples, including one on the Pendulum and another on Planets.

Miscelaneous

atlantistools v0.4.2: Provides access to the Atlantis framework for end-to-end marine ecosystem modelling. There is a package demo and vignettes for model preprocessing, model calibration, species calibration, and model comparison.

phylodyn v0.9.0: Provides statistical tools for reconstructing population size from genetic sequence data. There are several vignettes including a Coalescent simulation of genealogies and a case study using New York Influenza data.

Statistics

adaptiveGPCA v0.1: Implements the adaptive gPCA algorithm described in Fukuyama. The vignette shows an example using data stored in a phyloseq object.

BayesNetBP v1.2.1: Implements belief propagation methods for Bayesian Networks based on the paper by Cowell. There is a function to invoke a Shiny App.

RPEXE.RPEXT v0.0.1: Implements the likelihood ration test and backward elimination procedure for the reduced piecewise exponential survival analysis technique described in described in Han et al. 2012 and 2016. The vignette provides examples.

sfdct v0.0.3: Provides functions to construct a constrained ‘Delaunay’ triangulation from simple features objects. There is a vignette.

simglm v0.5.0: Provides functions to simulate linear and generalized linear models with up to three levels of nesting. There is an Introduction and vignettes for simulating GLMs and Missing Data performing Power Analysis and dealing with Unbalanced Data.

Utilities

checkarg v0.1.0: Provides utility functions that allow checking the basic validity of a function argument or any other value, including generating an error and assigning a default in a single line of code.

CodeDepends v0.5-3: Provides tools for analyzing R expressions or blocks of code and determining the dependencies between them. The vignette shows how to use them.

desctable v0.1.0: Provides functions to create descriptive and comparative tables that are ready to be saved as csv, or piped to DT::datatable() or pander::pander() to integrate into reports. There is a vignette to get you started.

lifelogr v0.1.0: Provides a framework for combining self-data from multiple sources, including fitbit and Apple Health. There is a general introduction as well as an introduction for visualization functions.

processx v2.0.0: Portable tools to run system processes in the background.

printr v0.1: Extends knitr generic function knit_print() to automatically print objects using an appropriate format such as Markdown or LaTeX. The vignette provides an introduction.

RHPCBenchmark v0.1.0: Provides microbenchmarks for determining the run-time performance of aspects of the R programming environment, and packages that are relevant to high-performance computation. There is an Introduction.

rlang v0.1.1: Provides a toolbox of functions for working with base types, core R features like the condition system, and core ‘Tidyverse’ features like tidy evaluation. The vignette explains R’s capabilities for creating Domain Specific Languages.

readtext v0.50: Provides functions for importing and handling text files and formatted text files with additional meta-data, including ‘.csv’, ‘.tab’, ‘.json’, ‘.xml’, ‘.pdf’, ‘.doc’, ‘.docx’, ‘.xls’, ‘.xlsx’ and other file types. There is a vignette

tangram v0.2.6: Provides an extensible formula system to implements a grammar of tables for creating production-quality tables using a three-step process that involves a formula parser, statistical content generation from data, and rendering. There is a vignette introducing the Grammar, a Global Style for Rmd, and duplicating SAS PROC Tabulate.

tatoo v1.0.6: Provides functions to combine data.frames and to add metadata that can be used for printing and xlsx export. The vignette shows some examples.

Visualizations

ContourFunctions v0.1.0: Provides functions for making contour plots. A vignette introduces the package.

mbgraphic v1.0.0: Implements a two-step process for describing univariate and bivariate behavior similar to the cognostics measures proposed by Paul and John Tuke. First, measures describing variables are computed and then plots are selected. The vignette describes the details.

polypoly v0.0.2: Provides tools for reshaping, plotting, and manipulating matrices of orthogonal polynomials. The vignette provides an overview.

RJSplot v2.1: Provides functions to create interactive graphs with ‘R’. It joins the data analysis power of R and the visualization libraries of JavaScript in one package There is a tutorial.

April New Package Picks

Tue, 30 May 2017 00:00:00 +0000

Here are my picks for the “Top 40” new packages submitted to CRAN in April 2017. These selections, which were culled from 208 submissions, are organized into four categories: Data, Finance, Statistics and Utilities. The number of entries in the Data and Utilities categories reflect the initiatives of R developers to connect to external resources.

Data

comtradr v0.0.1: Provides functions to extract country-level shipping data for a variety of commodities data from the United Nations Comtrade API.

countyfloods v0.0.2: Provides access to county-level United States flood data using United States Geological Service (USGS) API. The vignette provides examples.

EdSurvey v1.0.6: Provides functions to access and analyze survey and assessment data from the National Center for Education Statistics (NCES), including the National Assessment of Educational Progress (NAEP. There are is an Introduction, and vignettes for Advanced Data Manipilation, NAEP Accommodations, Statistical Methods, and Achievement Levels.

RNRCS v0.1.1: Provides functions to download data from the Natural Resources Conservation Service (NRCS) for sites in the Soil Climate Analysis Network (SCAN), and Snow Telemetry (SNOTEL and SNOLITE) networks. Metadata can be returned for all sites in the NRCS’ Air and Water Data Base (AWDB).

roadoi v0.2: Implements a web client interface to oaDOI, a service providing free full-text access to academic papers through various sources including Crossref, Bielefeld Academic Search Engine (BASE), and the Directory of Open Access Journals (DOAJ). The vignette shows how to use it.

rsoi v0.2.1: Provides functions to download Southern Oscillation Index data and Oceanic Nino Index data from NOAA.

rsolr v0.0.4: Implements an API for querying Apache Solr databases. There is a vignette.

tubern v0.1.0: Implements an R client for the YouTube Analytics and Reporting API. The vignette shows how to use the API.

wikitaxa v0.1.4: Provides access to taxonomic information from Wikipedia, Wikicommons, Wikispecies, and Wikidata. There is a vignette.

Finance

partialCI v1.1.0: Provides functions for estimating, testing, and simulating partially cointegrated time series. Clegg et al. provide an in-depth discussion of the package functionality, as well as illustrating examples in the fields of finance and macroeconomics. The vignette provides a guide to the package.

PortfolioOptim v1.0.3: Provides two functions for financial portfolio optimization by linear programming.

Statistics

bigKRLS v1.5.2: Provides functions for Kernel-Regularized Least Squares that are optimized for speed and memory usage. The author’s website has sample code and more. A vignette shows the basics.

brglm2 v0.1.4: Provides functions for achieving bias reduction in Generalized Linear Models via solving the mean bias-reducing adjusted score equations in Firth and Kosmidis and Firth, or the median bias-reduction adjusted score equations in Kenne et al., or through the direct subtraction of an estimate of the bias of the maximum likelihood estimator from the maximum likelihood estimates as in Cordeiro and McCullagh. There are vignettes on bias reduction and on Detecting Separation.

casebase v0.1.0: Implements the case-base sampling approach of Hanley and Miettinen, Saarela and Arjas, and Saarela, for fitting flexible hazard regression models to survival data with single event type or multiple competing causes via logistic and multinomial regression. There are vignettes on Competing Risk Analysis, Population Time Plots, and Casebase Sampling.

CausalImpact v1.2.0: Implements a Bayesian approach to causal impact estimation in time series, as described in Brodersen et al. Look here to get started, and also at the vignette.

To estimate a causal effect, specify which period causal should be used for training the model (pre-intervention period), and which period for computing a counterfactual prediction (post-intervention period).

ggeffects v0.1.0: Provides functions to compute marginal effects at the mean or average marginal effects from statistical models, and returns the result as tidy data frames. The vignette shows how to use it.

glmnetUtils v1.0.2: Provides a formula interface for the glmnet package. The vignette shows how to use it.

lspline v1.0-0: Implements linear splines with convenient parameterizations such that (1) coefficients are slopes of consecutive segments, or (2) coefficients are slope changes at consecutive knots. The vignette gives examples.

marcher v0.0-2: Provides tools for likelihood-based estimation, model selection, and testing of two- and three-range shift and migration models for animal movement data as described in Gurarie et al.. The vignette shows how to use the model and create visualizations.

Simulation of Range Shift Processes

Meiosis v1.0.2: Provides tools for simulation of meiosis in plant breeding research. There is a vignette.

MultiVarSel v1.0: Implements a novel variable selection approach in the multivariate framework of the general linear model, taking into account the dependence that may exist between the columns of the observations matrix. For details, see the paper Perrot-Dockes et al.. The vignette shows an example.

mrbsizeR v1.0.1: Implements a method for the multiresolution analysis of spatial fields and images to capture scale-dependent features based on scale space smoothing. The vignette provides an example.

networktools v1.0.0: Includes assorted tools for network analysis, including functions for calculating impact statistics (global strength impact, network structure impact, edge impact), and for calculating and visualizing expected influence. The vignette describes the impact statistics.

philentropy v0.3.3: Provides functions to compute forty-six optimized distance and similarity measures for comparing probability functions that are useful in a broad range of scientific disciplines. There is an Introduction and vignettes for Distances Measures and Information Theory.

plink v1.5-1: Provides functions to facilitate the linking of mixed-format tests for multiple groups under a common item design using unidimensional and multidimensional IRT-based methods. The very thorough vignette presents the theory.

RLT v3.1.0: Random forest with a variety of additional features for regression, classification, and survival analysis.

spup v0.1-0: Implements for analyzing uncertainty propagation in spatial environmental modeling using the methodologies of Heuvelink et al. and Brown and Heuvelink. There are vignettes on spatially cross-correlated variables, spatially variable sd, uncertainty propagation, and categorical variables.

Utilities

ajv v1.0.0: Provides a thin wrapper around the ajv JSON validation package for JavaScript. There is a vignette to get you started.

aws.s3 v.3.1: Implements a simple client package for the Amazon Web Services (AWS) Simple Storage Service (S3) REST API.

cranlike v1.0.0: Provides a set of functions to manage ‘CRAN’-like repositories efficiently.

d3Tree v0.1.0: Provides functions to create and customize interactive collapsible ‘D3’ trees using the D3 JavaScript library and the htmlwidgets package, which can be used directly from the R console, from RStudio, in Shiny apps, and in R Markdown documents. Look here for examples.

library(d3Tree)

  d3tree(list(root = df2tree(rootname = 'book',
                             struct = stan.models),
                             layout = 'collapse'))

dataMeta v0.1.0: Provides functions to create a basic data dictionary and append it to the original dataset’s attributes list. The vignette shows how it works.

dparser v0.1.3: Implements a scannerless GLR parser based on the Tomita algorithm.

glue v1.0.0: Provides an implementation of interpreted string literals, inspired by Python’s Literal String Interpolation, Docstrings, and Julia’s Triple-Quoted String Literals. The README shows how they work.

icosa v0.98.1: Employs triangular tessellation to refine icosahedra defined in 3D space. The vignette provides the details.

leaflet.minicharts v0.2.0: Allows users to add and modify small charts on interactive maps created with the leaflet package

RcppQuantuccia v0.0.1: Provides QuantLib bindings for R using ‘Rcpp’ and the header-only Quantuccia variant.

shinymaterial v0.2.1: Allows shiny developers to incorporate UI elements based on Google’s Material Design. See the package website more information.

unitizer v1.4-2: Provides functions to simplify regression tests by comparing objects produced by test code with earlier versions of those same objects. There is an Introduction, and vignettes for the interactive environment, Reproductble Tests, unitizeR details, and Miscellanea.

unix v1.3: Provides bindings to various Unix system utilities.

Mapping Quandl Macroeconomic Data

Wed, 10 May 2017 00:00:00 +0000

In previous posts, we built a map to access global ETFs and a simple Shiny app to import and forecast commodities data from Quandl. Today, we will begin a project that combines those previous apps. Our end goal is to build an interactive map to access macroeconomic data via Quandl, allowing the user to choose an economic indicator and click on a country to access that indicator’s time series. For example, if a user wants to import and graph GDP-per-capita data from China, that user will be able to select GDP per Capita from a drop-down menu, and then click on China to display the time series. As usual, we’ll start with a Notebook for data import, test visualizations, and map building.

Careful readers will note that we are not simply reusing code from those previous projects; that is because we are going to introduce some new packages to our toolkit. Specifically:

Instead of downloading a spatial data frame from Natural Earth, we will use the new R package rnaturalearth and it’s function ne_countries() to import our geographic data.
Relatedly, we are not going work with a “spatial” data frame, but rather with a simple features data frame of class sf. Learn more here.
Finally, when we test our data import of several macroeconomic data sets, we will use the map() function from the purrr package instead of relying on lapply(), as we did previously when passing ticker symbols to getSymbols(). If you’re like me, you won’t be sad to be done with lapply().

Let’s get to it!

We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators (WDI). The Quandl code for WDI is WWDI, and thus we’ll prepend WWDI/ to each data set call.

One difference from our earlier work with Quandl is that previously we imported and visualized global time series, such as the price of oil, gold or copper. By global, I mean these time series were not associated with a particular country. That simplified our job a bit because we did not need to supply a country code to Quandl. Today, we do need to include a country code.

Let’s start with a simple example and import GDP-per-capita data for China, whose country code is CHN. We will pass the string WWDI/CHN_NY_GDP_PCAP_KN to Quandl, which consists of the WDI code WWDI/, appended by the China country code CHN, appended by the GDP-per-capita code _NY_GDP_PCAP_KN.

library(Quandl)
# Pass the code string to Quandl. 
China_GDPPC <- Quandl("WWDI/CHN_NY_GDP_PCAP_KN", type = 'xts') %>% 
  # Add a nice column name
  `colnames<-`("GDP Per Capita")

# Take a look at the result.
tail(China_GDPPC, n = 6)

##            GDP Per Capita
## 2010-12-31       30876.04
## 2011-12-31       33658.85
## 2012-12-31       36126.73
## 2013-12-31       38737.58
## 2014-12-31       41354.61
## 2015-12-31       43991.55

That looks pretty good, but note that the data only goes back to December of 2015 - not ideal, but fine for our illustrative purposes.

Our eventual Shiny app, though, will give the user a choice among many economic time series, and we want to test the import and visualization of all those choices. In addition to GDP-per-capita, let’s test GDP-per-capita growth, real interest rate, exchange rate, CPI, and labor force participation rate. Those offer a nice snapshot of a country’s economy.

To run our test in the code chunk below, we will first build a vector of data set codes. Note that each of them start with WWDI/, then CHN, and then the relevant code. Next, we will pipe that vector to Quandl using the map() function. map() takes a function, in this case the Quandl() function, and applies it to the elements in a vector, in this case our vector of data set codes. We also want to specify type = "xts".

If we stopped our pipe there, the output would be a list of six time series and that would suit our testing purposes, but I don’t love having to look at a list of six. Thus, we’ll keep on piping and use the reduce() function from the purrr package and call merge() to combine the six lists into one xts object. That is much easier to deal with, and easier to pass to dygraphs for our test visualization. The last piped function will invoke colnames<- to clean up the column names.

library(purrr)
# Create a vector of economic indicators that can be passed to Quandl via map().
# Include names and values for easy naming of xts columns.
econIndicators <- c("GDP Per Capita" = "WWDI/CHN_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "WWDI/CHN_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "WWDI/CHN_FR_INR_RINR",
                  "Exchange Rate" = "WWDI/CHN_PX_REX_REER",
                  "CPI" = "WWDI/CHN_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "WWDI/CHN_SL_TLF_ACTI_ZS")

# You might want to supply your Quandl api key. It's free to get one.
Quandl.api_key("d9EidiiDWoFESfdk5nPy")

China_all_indicators <- 
  # Start with the vector of Quandl codes
  econIndicators %>% 
  # Pass them to Quandl via map(). 
  map(Quandl, type = "xts") %>% 
  # Use the reduce() function to combine them into one xts objects.
  reduce(merge) %>% 
  # Use the names from the original vector to set nicer column names.
  `colnames<-`(names(econIndicators))
# Have a look.
tail(China_all_indicators, n = 6)

##            GDP Per Capita GDP Per Capita Growth Real Interest Rate
## 2011-12-31       33658.85              9.012854          -1.472149
## 2012-12-31       36126.73              7.332031           3.523236
## 2013-12-31       38737.58              7.226936           3.692594
## 2014-12-31       41354.61              6.755778           4.732429
## 2015-12-31       43991.55              6.376423           4.270386
## 2016-12-31             NA                    NA                 NA
##            Exchange Rate      CPI Labor Force Part. Rate
## 2011-12-31      102.6904 5.410850                   76.7
## 2012-12-31      108.4435 2.624921                   77.0
## 2013-12-31      115.2980 2.627119                   77.3
## 2014-12-31      118.9863 1.996847                   77.6
## 2015-12-31      131.6298 1.442555                     NA
## 2016-12-31      124.1864 2.007556                     NA

That seems to have done the job, and note that both CPI and Exchange have data up to December of 2016. Much better, and a sign that there will be date variation among countries and indicators. That won’t be crucial today, but it could be if we needed consistent dates for all of our data sets.

As a final step, let’s use the dygraph() function to visualize these time series to make sure things work properly before moving to Shiny.

library(dygraphs)

## Warning: package 'dygraphs' was built under R version 3.3.2

dygraph(China_all_indicators$`GDP Per Capita`, main = "GDP Per Capita")

dygraph(China_all_indicators$`GDP Per Capita Growth`, main = "GDP Per Capita Growth")

dygraph(China_all_indicators$`Real Interest Rate`, main = "Real Interest Rate")

dygraph(China_all_indicators$`Exchange Rate`, main = "Exchange Rate")

dygraph(China_all_indicators$`CPI`, main = "CPI")

dygraph(China_all_indicators$`Labor Force Part. Rate`, main = "Labor Force Part. Rate")

GDP-per-capita has been on a steady march higher, labor force participation plummeted in the 1990’s and 2000’s, probably as the labor market became more market-oriented, and real interest rates look quite choppy. Plenty to learn, and it signals that our Shiny app might be a nice exploratory tool.

For now, it’s on to map-building!

In previous posts, we have started map-building by grabbing a spatial object from the internet. Today, we will use the new rnaturalearth package and its fantastic function ne_countries(). We will specify type = "countries" and returnclass = 'sf'. We could have set returnclass = 'sp' if we wanted to work with a spatial data frame, but I have been migrating over to the sf package and simple features data frames for mapping projects. Lots to learn here, but in general, it seems to use more tidy concepts than the sp package.

library(rnaturalearth)
library(sp)
world <- ne_countries(type = "countries",  returnclass = 'sf')

# Take a peek at the name, gdp_md_est column and economy columns. 
# The same way we would peek at any data frame.
head(world[c('name', 'gdp_md_est', 'economy')], n = 6)

## Simple feature collection with 6 features and 3 fields
## geometry type:  MULTIPOLYGON
## dimension:      XY
## bbox:           xmin: -73.41544 ymin: -55.25 xmax: 75.15803 ymax: 42.68825
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
##                   name gdp_md_est                   economy
## 0          Afghanistan      22270 7. Least developed region
## 1               Angola     110300 7. Least developed region
## 2              Albania      21810      6. Developing region
## 3 United Arab Emirates     184300      6. Developing region
## 4            Argentina     573900   5. Emerging region: G20
## 5              Armenia      18770      6. Developing region
##                         geometry
## 0 MULTIPOLYGON(((61.210817091...
## 1 MULTIPOLYGON(((16.326528354...
## 2 MULTIPOLYGON(((20.590247430...
## 3 MULTIPOLYGON(((51.579518670...
## 4 MULTIPOLYGON(((-65.5 -55.2,...
## 5 MULTIPOLYGON(((43.582745802...

This output looks similar to an sp object, but it’s not identical. In particular, note the geometry column, which contains the multipolygon latitude and longitude coordinates. It’s a matter of personal preference, but I find this sf object more intuitive in terms of how the geometry corresponds with the rest of the data.

Now we’ll create our shading and popups. Note that this code is identical to when we were working with spatial data frames.

library(leaflet)
# Create shading by GDP. Let's go with purples.
gdpPal <- colorQuantile("Purples", world$gdp_md_est, n = 20)

# Create popup country name and income group so that something happens
# when a user clicks the map.
popup <- paste0("<strong>Country: </strong>", 
                world$name, 
                "<br><strong>Market Stage: </strong>", 
                world$income_grp)

Now, it’s time to build our leaflet map. This code is nearly identical to our code from previous projects, but with one major change. As we saw above, we need to pass in a country code to Quandl, not a country name and not a ticker symbol. Thus, we don’t want to set layerId = name, but we also don’t need to add anything special to our world object. It already contains a column called iso_a3, and that column contains the country codes used by Quandl. Fantastic!

We just need to set layerId = ~iso_a3 and we will be able to access country codes in our Shiny app when a user clicks the map. This is quite fortunate, but it also highlights a good reason to spend time studying the world object. It might contain other data that will be useful in future projects.

leaf_world <- leaflet(world) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7, 
      
      # Note the layer ID. Not a country name! It's a country code! 
      
      color = ~gdpPal(gdp_md_est), layerId = ~iso_a3, popup = popup)

leaf_world

Because Quandl uses the ISO 3-letter code to identify a country, and because the rnaturalearth sf object already contains a column with iso_a3 country codes, building our map wasn’t very hard. It’s worth pointing out that in various projects, we have now built maps with three different layerIDs. In the Global ETF Map project, we built one map with layerID = name, using the built-in name column in the rnaturalearth object, and we built another map with layerID = tickers, using a column of tickers that we added ourselves with the merge() function. Now, today, we have a map with layerID = iso_a3, using the built-in column of country codes. We are building up a nice portfolio of maps with different layerIDs should we ever want or need them in the future.

That’s all for today. We’ll save that leaflet_world object for use in our Shiny app! See you next time.

Missing Values, Data Science and R

Wed, 30 Nov 2016 00:00:00 +0000

One great advantages of working in R is the quantity and sophistication of the statistical functions and techniques available. For example, R’s quantile() function allows you to select one of the nine different methods for computing quantiles. Who would have thought there could be so many ways to do something that seems to be so simple? The issue here is not unnecessary complication, but rather an appreciation of the nuances associated with inference problems gained over the last hundred years of modern statistical practice. Much of this knowledge is reflected in R. As I imagine it, if R’s functions were laid out like city streets there would be many broad avenues displaying commonly used algorithms, but these would flow into many thousands of small, rarely-visited alleys, containing treasures for those who seek them out.

My impression is that most data science is done on the avenues, but that data scientists could benefit from occasionally venturing deeper into the alleys of statistical practice. Statistics might be the least important part of data science but there are times when coming to grips with sound statistical inference is essential. Consider the problem of dealing with missing values for example. It is difficult to imagine any large, real-world data set that wouldn’t require a strategy for imputing missing values. The obvious first step in developing a strategy would be to form some ideas about why the data are missing. Gelman and Hill identify four different “missingness mechanisms”: (1) Missingness completely at random, (2) Missingness at random, (3) Missingness that depends on unobserved predictors, (4) Missingness that depends on the missing value itself; and they provide some advice on how to cope with each of them. In a large data set, there is no reason to believe that there is just one mechanism in play! Evaluating your data with respect to these categories and their combinations could require a frightening amount of exploratory work. So tools for looking at patterns in missing values are likely to be very helpful, even if using them requires sampling your data set.

The next step of deciding how to proceed in a statistically sound manner will likely pose considerable technical challenges, and opting for simple solutions may not be advisable, or even possible. Even with big data, ignoring observations with missing values could result in a catastrophic loss of information, and simple approaches to imputation such as replacing missing the missing values of each variable with the variable mean, or a common value, will produce a “completed” data set reflecting an artificial amount of certainty that will likely underestimate the variability of the data and bias results. R can’t eliminate the hard work, but it does provide a formidable array of missing value imputation tools that can get you in and out of the statistical alleys and help you to decide what techniques might be useful.

Before evaluating the various methods, it is helpful to make a couple of distinctions about imputation methods for multivariate data. The first is between single and multiple imputation. In single imputation, a particular algorithm or technique is used to produce a single estimate for each missing value. Multiple imputation methods, on the other hand, develop distributions for missing values and estimate individual missing values by drawing from this distribution. In general, these algorithms proceed in three steps. In the first imputation step, multiple draws are made from the distribution of missing values for each variable. This process results in several “completed” data sets where each missing value has been estimated by a plausible value. In the second step, the intended analysis is performed on each completed data set. In the last step, a “pooling”algorithm estimates the final values for the statistics of interest.

The second distinction is between Joint Modeling (JM) and Fully Conditional Specification (FCS) imputing. In JM, the data are assumed to follow a multivariate parametric distribution of some sort. This is theoretically sound but may not be flexible enough to adequately model the data. In FCS, the multivariate data model is specified by developing a separate conditional model for each variable with missing values.

Here is a short list of R packages for missing value imputation. I have selected these to give some idea of the variety of tools available.

Amelia implements the Amelia II algorithm which assumes that the complete data set (missing and observed data) are multivariate normal. Imputations are done via the EMB (expectation-maximization with bootstrapping) algorithm. The JSS paper describes a strategy for combining the models resulting from each imputed data set. The Amelia vignette contains examples.

BaBoon provides two variants of the the Bayesian Bootstrap predictive mean matching to impute multiple missing values. Originally developed for survey data, the imputation algorithms are described as being robust with respect to imputation model misspecification. The best description and rationale for the algorithms seems to be the PhD thesis of one of the package authors.

Hmisc contains several functions that are helpful for missing value imputation including agreImpute(), impute() and transcan(). Documentation on Hmisc can be found here.

mi takes a Bayesian approach to imputing missing values. The imputation algorithm runs multiple MCMC chains to iteratively draw imputed values from conditional distributions of observed and imputed data. In addition to imputation algorithm, the package contains functions for visualizing the pattern of missing values in a data set and assessing the convergence of the MCMC chains. A vignette shows a worked example and the associated JSS paper delves deeper into the theory and the mechanics of using the method.

mice which is an acronym for multivariate imputation of chained equations, formalizes the multiple implementation process outline above and is probably the gold standard for FCS multiple imputation. Package features include:

Columnwise specification of the imputation model
Support for arbitrary patterns of missing data
Passive imputation techniques that maintain consistency among data transformations
Subset selection of predictors
Support of arbitrary complete-data methods
Support pooling various types of statistics
Diagnostics for imputations
Callable user-written imputation functions

The JSS paper describes how the desire to provide a separate imputation model for each variable led to the development of the chained equation technique where a Gibbs sampler fills out the missing data. miceAdds provides additional functions to be used with mice including plausible value imputation, multilevel imputation functions, imputation using partial least squares (PLS) for high dimensional predictors, nested multiple imputation, and two-way imputation.

missingDataGUI implements a nice graphical interface for exploring missing data patterns with numeric and graphical summaries for numeric and categorical missing values and implements a number of imputation methods. The figure below shows the missing value map for the HouseVotes84 data set in the mlbench package.

missMDA performs principal component methods on incomplete data sets obtaining scores, loadings and graphical representations despite missing values. The package also includes functions to perform single and multiple imputation. The JSS paper provides the details.

VIM provides tools for visualizing missing or imputed values. Before imputation they can be used to study the pattern of missing values, afterwards these same tools can be used as diagnostics. VIMGUI puts a front end on the VIM functions and helps with handling the plot and imputation functions. The vignette is thorough and provides some nice examples of how one might look at missing value distributions.

vtreat provides tools to assist in the statistically sound preparation of data sets. It is not a package explicitly devoted to missing value imputation, but it can produce “cleaned” data sets that have no “Infinite/NA/NaN in the effective variable columns”. I include it here to emphasize that proper data preparation can simplify the missing value problem. The package has several vignettes.

yaImpute takes what might be thought of as a machine learning approach to imputing missing values by using the k-nearest neighbor (kNN) algorithm to impute missing values. The JSS paper covers the theory and explains the package using a forestry application.

For additional R packages see Stef van Buuren’s webpage cataloging software for imputation (Editor: the list is no longer available.) which lists twelve R packages that implement some method of single imputation and eighteen R packages concerned with multiple imputation and provides a brief explanation of each of these. Also, have a look on the post on analyticsvidha.com that provides informative short write ups on amelia, Hmisc, mi, mice and missForest that include some sample code.

I also recommend looking at Stef van Buuren presentation on the history and future of Fully Conditional Specification from the 2015 Rennes missing value conference and Julie Josse’s tutorial at useR! 2016.

Even with the best of tools there is no doubt that dealing with missing values in a statistically sound manner is difficult. However, this is the that kind of work that helps to put the “science” in data science, and R is the most helpful environment you are likely to find.

Packages on R Views

reticulate, virtualenv, and Python in Linux

Setting up Python, virtualenv, and RStudio

Using Python from within RStudio via reticulate

Test that it works

Reproducible Environments

Safely Upgrading Packages

Team Collaboration

Adaptable Strategies

Alternative Design for Shiny

Shiny’s Design

Bootstrap

Alternatives

Material Design

Standards vs. Framework

Material Design in Shiny

Conclusion

Recent R Data Packages

Control Systems Toolbox – System Interconnection

Introduction

Laplace Transform

Transfer Function Expressions

Electric Motor Example

Open-Loop Response of the electric motor

Closed-Loop Response of the electric motor

System Interconnections

Append Connection using the append function

Series Connection using the series function

Parallel Connection using the parallel function

Feedback Connection using the feedback function

General block diagram building using connect function

Case Study – Unmanned Free Swimming Submersible (UFSS) Vehicle

Rationale:

Solution using series, parallel and feedback

Solution via algebraic operations

Feedback

Conclusion

References

End-to-end visualization using ggplot2

1. Pima Indian Diabetes data

2. Distributions across categories

2.1. Boxplots

2.2. Density plots

3. Grid views

3.1. Data preparation

3.2. Reshaping the data

3.3. Facet Grid

4. Heatmaps

4.1. Data preparation

4.2. Rendering the heatmap

5. Segmentation in a scatterplot

June 2017 New Package Picks

Biostatistics

Data

Machine Learning

Miscellaneous

Stats

Utilities

Some Ideas for your Internal R Package

Logos and Custom CSS

ggplot2 Themes

Data Connections

learnr Tutorials

Sharing an Internal R Package

Wrap Up

May New Package Picks

Data

Data Science and Machine Learning

Education

Miscelaneous

Statistics

Utilities

Visualizations

April New Package Picks

Data

Finance

Statistics

Utilities

Mapping Quandl Macroeconomic Data

Missing Values, Data Science and R

Append Connection using the `append` function

Series Connection using the `series` function

Parallel Connection using the `parallel` function

Feedback Connection using the `feedback` function

General block diagram building using `connect` function

Solution using `series`, `parallel` and `feedback`