Tidymodels on R Views

Modern Rule-Based Models

Thu, 21 May 2020 00:00:00 +0000

Modern Rule-Based Models

Machine learning models come in many shapes and sizes. While deep learning models currently have the lion’s share of coverage, there are many other classes of models that are effective across many different problem domains. This post gives a short summary of several rule-based models that are closely related to tree-based models (but are less widely known).

While this post is focused on explaining on how these models work, it coincides with the release of the rules package, a tidymodels package that provides a user interface to these models. A companion post at the tidyverse blog describes the usage of the package.

To start, let’s discuss the concept of rules more generally.

What is a rule?

Rules in machine learning have been around for a long time (Quinlan, 1979). The focus of this article is using rules for traditional supervised learning (as opposed to association rule mining). In the context of feature engineering, a rule is a conditional logical statement. It can be attached to some sort of predicted value too, such as

if (chance_of_rain > 0.75) {
  umbrella <- "yes"
} else {
  umbrella <- "no"
}

There are various ways to create rules from data. The most popular method is to create a tree-based model and then “flatten” the model structure into a set of rules. This is called a “separate and conquer” approach.

To demonstrate this, let’s use a data set with housing prices from Sacramento CA. Two predictors with a large number of levels are removed to make the rule output more readable:

for (pkg in c('dplyr', 'modeldata', 'rpart')) {
  if (!requireNamespace(pkg)) {
    install.packages(pkg)
  }
}

library(dplyr)

data(Sacramento, package = "modeldata")

Sacramento <-
  Sacramento %>% 
  mutate(price = log10(price)) %>% 
  select(-zip, -city)

str(Sacramento)

## tibble [932 × 7] (S3: tbl_df/tbl/data.frame)
##  $ beds     : int [1:932] 2 3 2 2 2 3 3 3 2 3 ...
##  $ baths    : num [1:932] 1 1 1 1 1 1 2 1 2 2 ...
##  $ sqft     : int [1:932] 836 1167 796 852 797 1122 1104 1177 941 1146 ...
##  $ type     : Factor w/ 3 levels "Condo","Multi_Family",..: 3 3 3 3 3 1 3 3 1 3 ...
##  $ price    : num [1:932] 4.77 4.83 4.84 4.84 4.91 ...
##  $ latitude : num [1:932] 38.6 38.5 38.6 38.6 38.5 ...
##  $ longitude: num [1:932] -121 -121 -121 -121 -121 ...

Consider a basic CART model created using rpart:

library(rpart)

rpart(price ~ ., data = Sacramento)

## n= 932 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
##  1) root 932 48.5000 5.334  
##    2) sqft< 1594 535 18.7200 5.211  
##      4) sqft< 1170 235  8.0440 5.107  
##        8) sqft< 934.5 71  2.2760 5.002 *
##        9) sqft>=934.5 164  4.6500 5.152 *
##      5) sqft>=1170 300  6.1390 5.292  
##       10) longitude< -121.3 243  4.8850 5.270 *
##       11) longitude>=-121.3 57  0.6081 5.388 *
##    3) sqft>=1594 397 10.6000 5.501  
##      6) sqft< 2317 245  4.7420 5.432  
##       12) longitude< -121.3 205  3.3190 5.407 *
##       13) longitude>=-121.3 40  0.6179 5.562 *
##      7) sqft>=2317 152  2.8450 5.611  
##       14) longitude< -121.2 110  1.6790 5.570 *
##       15) longitude>=-121.2 42  0.4864 5.720 *

The splits in this particular tree involve the same two predictors. These paths to the terminal nodes are comprised of a set of if-then rules. Consider the path to the eighth terminal node:

if (sqft < 1594 & sqft < 1169.5 & sqft < 934.5) then pred = 5.001820

It is easy to prune this rule down to just sqft < 934.5. In all, the rules generated from this model would be:

##  price                                              
##    5.0 when sqft <   935                            
##    5.2 when sqft is  935 to 1170                    
##    5.3 when sqft is 1170 to 1594 & longitude <  -121
##    5.4 when sqft is 1170 to 1594 & longitude >= -121
##    5.4 when sqft is 1594 to 2317 & longitude <  -121
##    5.6 when sqft is 1594 to 2317 & longitude >= -121
##    5.6 when sqft >=         2317 & longitude <  -121
##    5.7 when sqft >=         2317 & longitude >= -121

What are some more modern models that can use or generate rules? We’ll walk through three (which are all included in the rules package).

C5.0 rules

The C4.5 algorithm (Quinlan, 1993b) was an early tree-based model that was released not long after the more well known CART model. One cool aspect of this model is that it could generate a classification tree or a set of rules. These rules are derived from the original tree much in the same way that was shown above for rpart.

Over the years, the author (Ross Quinlan) kept evolving the next generation of the model called C5.0. About 10 years ago, he open-sourced that model and the C50 R package was born. Like its predecessor, C5.0 could be used for trees or for rules. There are a variety of advances to this model (detailed in Kuhn and Johnson (2013)), but the most significant was the inclusion of boosting. In effect, you could create an ensemble of classification rules. Rather than approaching the problem via the more modern stochastic gradient boosting paradigm, it is more similar to the classical AdaBoost methodology.

Since C5.0 is classification only, an example of a single rule set for the iris data is:

Rule 1: (50, lift 2.9)
    Petal.Length <= 1.9
    ->  class setosa  [0.981]

Rule 2: (48/1, lift 2.9)
    Petal.Length > 1.9
    Petal.Length <= 4.9
    Petal.Width <= 1.7
    ->  class versicolor  [0.960]

Rule 3: (46/1, lift 2.9)
    Petal.Width > 1.7
    ->  class virginica  [0.958]

Rule 4: (46/2, lift 2.8)
    Petal.Length > 4.9
    ->  class virginica  [0.938]

Default class: setosa

If used on the iris data, a boosting ensemble with 50 constituent models used a total of 276 rules and each iteration averaged 5.52 rules per rule set.

The primary method of controlling the complexity of each rule set is to adjust the minimum number of data points required to make additional splits within a node. The default for this parameter is two data points. If this value was increased to 20, the mean number of rules was reduced from 5.52 rules to 3.2 rules.

When tuning, the other main tuning parameter is the number of boosting iterations. The C50 R package enables sub-model predictions across boosting iterations; tuning this parameter over many values is not very computation expensive.

Cubist

The aforementioned Ross Quinlan also developed model trees (Quinlan, 1992). These are regression tree-based models that contain linear regression models in the terminal nodes. This model was called M5 and, much like C5.0, there was a rule-based analog. Most tree-based models, especially ensembles of trees, tend to produce models that underfit in the tails (much like regression to the mean). Model trees do not suffer from this issue since their terminal regression models could make predictions across the whole range of the outcome data.

After an initial set of papers in the 1990’s, Quinlan didn’t publish much on the methodology as he evolved it. The modern version of model rules was called Cubist. There were a number of small technical differences between Cubist and M5 rules (enumerated in Kuhn and Johnson (2013)) but the main improvements were:

An ensemble method for predictions called committees.
A nearest-neighbor adjustment that occurs after the model predictions.

We’ll summarize these approaches in-turn.

Committee ensembles are similar to boosting. In boosting, a set of models are created sequentially. For the current iteration of boosting, the model is created using case weights that are defined by the results of previous models. In committees, case weights are not changed. Instead the outcome values are modified for each iteration of committees. For example, if a sample is under-predicted previously, its outcome value is changed to be larger so that the model will be pulled upward in an effort to stop under-predicting.

In committees, the first model uses the original outcome value $y$. On further iterations, a modified value $y^*$ is used instead. For iteration $m$, the model uses this adjustment formula:

\[ y^*_{(m)} = y - (\widehat{y}_{(m-1)} - y) \]

As a demonstration, the plot below shows how the pseudo-outcome changes over iterations. The observed price of the house in question is 5.3. On the first iteration, the model under-predicts (show as the solid black dot). The vertical line represents the residual from this model. On the next iteration, the open blue circle shows the value of $y^*_{(2)}$. On the second iteration, the prediction for this data point becomes worse but, as iterations proceed, the residuals generally become smaller.

Ensemble predictions are made by averaging over the committee model predictions. This concludes in the model-based prediction made by Cubist.

After the model prediction, there is an option to conduct a post-model nearest-neighbor adjustment (Quinlan, 1993a). When predicting a new data point, the K-nearest neighbors are found in the training set (along with their original predictions). If the training set predictions for the neighbors are denoted as $t$, the adjustment is:

\[ \widehat{y}_{adj} = \frac{1}{K}\sum_{\ell=1}^K w_\ell \left[t_\ell + \left(\widehat{y} - \widehat{t}_\ell \right)\right] \]

where the weights $w_\ell$ are based on the inverse distances (so that far points contribute less to the adjustment). The adjustment is large when the difference between the original and new predictions is large. As the difference between the predictions of the new sample and its closest neighbor increases, the adjustment becomes larger.

Suppose our model used only square footage and longitude as predictors. The training data are shown below with a new prediction point represented as a large red square. The 6-closest neighbors in the training set are shown as red circles. The size of the circle represents the magnitude of the weight. This shows that the nearby points influence the adjustment more than distant points.

This adjustment generally improves performance of the model. Interestingly, there is often a pattern when tuning where the 1-nearest neighbor model does much worse than using no adjustment but two or more neighbors do a much better job. One idea is that the use of a single neighbor is likely overfitting to the training set (as would occur with a more traditional K-NN model).

Using both of these techniques, Cubist tends to produce very competitive in terms of performance. The two primary parameters are the number of committee members and the number of nearest neighbors to use in the adjustment. The Cubist package can use the same fitted model to make predictions across the number of neighbors, so there is little computational cost when tuning this parameter.

For the Sacramento data, a single model (e.g. one committee) consisted of 6 rules, each with its own linear regression model. For example, the first four rules are:

##   Rule 1: [198 cases, mean 5.187264, range 4.681241 to 5.759668, est err 0.131632]
## 
##     if
##  sqft <= 1712
##  latitude > 38.46639
##  latitude <= 38.61377
##  longitude > -121.5035
##     then
##  outcome = -48.838393 + 0.000376 sqft + 1.39 latitude
## 
##   Rule 2: [254 cases, mean 5.220439, range 4.477121 to 5.923762, est err 0.105572]
## 
##     if
##  sqft <= 1712
##  latitude > 38.61377
##  longitude > -121.5035
##  longitude <= -121.0504
##     then
##  outcome = 93.155414 + 0.000431 sqft + 0.78 longitude + 0.16 latitude
## 
##   Rule 3: [90 cases, mean 5.273133, range 4.851258 to 5.580444, est err 0.078920]
## 
##     if
##  sqft <= 1712
##  latitude <= 38.46639
##     then
##  outcome = 15.750124 + 0.000344 sqft + 0.09 longitude - 0.005 beds
##            + 0.005 baths
## 
##   Rule 4: [35 cases, mean 5.340909, range 5.018076 to 5.616476, est err 0.086056]
## 
##     if
##  sqft <= 1712
##  longitude <= -121.5035
##     then
##  outcome = 4.865655 + 0.000357 sqft

New samples being predicted will fall into one or more rule conditions and the final prediction is the average of all of the corresponding linear model predictions.

If the model is run for 50 committees, a total of 271 rules were used across the committees with the average of 5.42 rules per committee member.

RuleFit

RuleFit models (Friedman and Popescu, 2008) are fairly simple in concept: use a tree ensemble to create a large set of rules, use the rules as binary predictors, then fit a regularized model that only includes the most important rule features.

For example, if a boosted tree were used to generate rules, each path through each tree would generate a conditional statement that can be used to define a model predictor (as was shown above for rpart). If an xgboost model with 100 boosting iterations with a limit of three splits were used on the Sacramento data, an initial set of 609 rules were generated. Some examples:

sqft >= 1594

longitude < -121.2 & latitude >= 38.73 & latitude >= 38.86

longitude >= -121.2 & longitude < -121 & baths < 2.75

longitude < -121.3 & sqft >= 1246 & type == "Multi_Family"

Clearly, the rules show some redundancy; there tends to be a significant amount of similarity in the rules.

These predictors are added to a regularized regression model (e.g. linear or logistic) that will conduct feature selection to remove unhelpful or redundant rules. Based on how much the model is penalized, the user can choose the number of rules that are contained in the final model. For example, depending on the penalty, the final rule set for the Sacramento data can be as large as hundreds or as small as a handful.

We can generate glmnet variable importance scores and then parameterize the importance in terms of the original predictors (instead of the rules). For example: for a penalty value of $\lambda$ = 0.005:

RuleFit has many tuning parameters; it inherits them from the boosting model as well as one from glmnet (the amount of lasso regularization). Fortunately, multiple predictions can be made across the lasso penalty using the same model object.

All-in-all, RuleFit is a neat and powerful method for using rules as features. It is interesting to contrast this model and Cubist:

Cubist creates rules as data subsets then estimates a linear regression models within each.
RuleFit creates rules as predictors then fits one (generalized) linear model.

Acknowledgments

Ross Quinlan has been supportive of our efforts to publish the inner workings of C5.0 and Cubist. I’d like to thank him for his help and all of the excellent work he has done over the years.

References

Friedman JH; Popescu BE (2008) “Predictive learning via rule ensembles.” _Annals of Applied Statistic_s, pp. 916-954.

Kuhn M; Johnson K (2013) Applied Predictive Modeling, Springer. New York.

Quinlan R (1979). “Discovering rules by induction from large collections of examples.” Expert Systems in the Micro Electronics Age.

Quinlan R (1992). “Learning with continuous classes.” Proceedings of the 5th Australian Joint Conference On Artificial Intelligence, pp. 343-348.

Quinlan R (1993a). “Combining instance-based and model-based learning.” Proceedings of the Tenth International Conference on Machine Learning, pp. 236-243.

Quinlan R (1993b). C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers.

The Case for tidymodels

Tue, 21 Apr 2020 00:00:00 +0000

If you are a data scientist with a built-out set of modeling tools that you know well, and which are almost always adequate for getting your work done, it is probably difficult for you to imagine what would induce you to give them up. Changing out what works is a task that rarely generates much enthusiasm. Nevertheless, in this post, I would like to point out a few features of tidymodels that could help even experienced data scientists make the case to give tidymodels a try.

So what are we talking about? tidymodels are an integrated, modular, extensible set of packages that implement a framework that facilitates creating predicative stochastic models. tidymodels are first class members of the tidyverse. They adhere to tidyverse syntax and design principles that promote consistency and well-designed human interfaces over speed of code execution. Nevertheless, they automatically build in parallel execution for tasks such as resampling, cross validation and parameter tuning. Moreover, they don’t just work through the steps of the basic modeling workflow, they implement conceptual structures that make complex iterative workflows possible and reproducible.

If you are an R user and you have building predictive models then there is a good chance that you are familiar with the caret package. One straightforward path to investigate tidymodels is to follow the thread that leads form caret to parsnip. caret, the result of a monumental fifteen year plus effort, incorporates two hundred thirty-eight predictive models into a common framework. For example, any one of the included models can be substituted for lm in the following expression.

lmFit <- train(Y ~ X1 + X2, data = training, 
                 method = "lm", 
                 trControl = fitControl)

By itself this is a pretty big deal. parsnip refines this idea by creating a specification structure that identifies a class of models that allows users to easily change algorithms and also permits the models to run on different “engines”.

spec_lin_reg <- linear_reg() %>%   # a linear model specification
                set_engine( "lm")  # set the model to use lm
# fit the model
lm_fit <- fit(spec_lin_reg, Y ~ X1 + X2, data = my_data)

This same specification can be modified to run a Bayesian model using Stan, or any number of other linear model backends such as glmnet, keras or spark.

spec_stan <- 
  spec_lin_reg %>%
  set_engine("stan", chains = 4, iter = 1000) # set engine specific arguments
fit_stan <- fit(spec_stan, Y ~ X1 + X2, data = my_data)

On its own, parnsnip provides a time saving framework for exploring multiple models. It is really nice not to have to worry about the idiosyncratic syntax developed for different model algorithms. But, the real power of tidymodels is baked into the recipes package. Recipes are structures that bind a sequence of preprocessing steps to a training data set. They define the roles that the variables are to play in the design matrix, specify what data cleaning needs to take place, and what feature engineering needs to happen.

To see how all of this comes together, lets look at recipe used in the tidymodels recipes tutorial that uses the New York City flights data set, nycflights13. We assume that all of the data wrangling code in the tutorial has been executed, and we pick up with the code to define the recipe:

flights_rec <- 
  recipe(arr_delay ~ ., data = train_data) %>% 
  update_role(flight, time_hour, new_role = "ID") %>% 
  step_date(date, features = c("dow", "month")) %>% 
  step_rm(date) %>% 
  step_dummy(all_nominal(), -all_outcomes())

The first line identifies the variable arr_delay as the variable to be predicted and the other variables in the data set train_data to be predictors. The second line amends that by updating the roles of the variables flight and time_hour to be identifiers and not predictors. The third and fourth lines continue with the feature engineering by creating a new date variable and removing the old one. The last line explicitly converts all categorical or factor variables into binary dummy variables.

The recipe is ready to be evaluated, but if a modeler thought that she might want to keep track of this workflow for the future, she might bind the recipe and model together in a workflow() that saves everything as a reproducible unit with a command something like this.

lr_mod <- logistic_reg() %>% set_engine("glm")

flights_wflow <- 
  workflow() %>% 
  add_model(lr_mod) %>% 
  add_recipe(flights_rec)

Then, fitting the model is just a matter calling fit with the workflow as a parameter.

flights_fit <- fit(flights_wflow, data = train_data)

At this point, everything is in place to complete a statistical analysis. A modeler can extract coefficients, p-values etc., calculate performance statistics, make statistical inferences and easily save the workflow in a reproducible markdown document. However the real gains from tidymodels become apparent when the modeler goes on to build predictive models.

The following diagram from Kuhn and Johnson (2019) illustrates a typical predictive modeling workflow.

It indicates that before going on to predict model performance on new data (the test set), a modeler will want to make use of cross validation or some other resampling technique to first evaluate the performance of multiple candidate models, and then tune the selected model. This is where the great power of the recipe() and workflow() constructs becomes apparent. In addition, to encouraging experiments with multiple models by rationalizing algorithm syntax, providing interchangeable model constructs, and enabling modelers to grow chains of recipe steps with the pipe operator; recipies helps to enforce good statistical practice.

For example, although it is common practice to split the available data between training and test sets before preprocessing the training data set, it is also very common to see pipelines where data preparation is applied to the entire training set at one go. It is not common to see data cleaning and preparation processes individually applied to each fold of a ten-fold cross validation effort. But, that is exactly the right thing to do to mitigate the deleterious effects of data imputation, centering and scaling and numerous other preparation steps that contribute to bias and limit the predictive value of a model. This is the whole point of resampling, but it is not easy to do in a way that saves necessary intermediate artifacts, and provides a reproducible set of instructions for others on the modeling team.

Because, recipes are not evaluated until the model is fit tidymodel workflows make an otherwise laborious and error prone process very straightforward. This is a game changer!

The next two lines of code set up and execute ten-fold cross-validation for our example.

set.seed(123)
folds <- vfold_cv(train_data, v = 10)
flights_fit_rs <- fit_resamples(flights_wflow, folds)

And then, another line of code collects the metrics over the folds and prints out the statistics for accuracy and area under the ROC curve.

collect_metrics(flights_fit_rs)

So, here we are with a mediocre model, and I’ll stop now having shown you only a small portion of what tidymodels can do, but enough, I hope to motivate you to take a closer look. tidymodels.org is a superbly crafted website with multiple layers of documentation. There are sections on packages, getting started guides, detailed tutorials, help pages and a section on making contributions.

Happy modeling!

A Gentle Introduction to tidymodels

Wed, 19 Jun 2019 00:00:00 +0000

Recently, I had the opportunity to showcase tidymodels in workshops and talks. Because of my vantage point as a user, I figured it would be valuable to share what I have learned so far. Let’s begin by framing where tidymodels fits in our analysis projects.

The diagram above is based on the R for Data Science book, by Wickham and Grolemund. The version in this article illustrates what step each package covers. Even though it is a single step, developing models can benefit from having a tidyverse-friendly interface. That is where tidymodels comes in.

It is important to clarify that the group of packages that make up tidymodels do not implement statistical models themselves. Instead, they focus on making all the tasks around fitting the model much easier. Those tasks are data pre-processing and results validation.

In a way, the Model step itself has sub-steps. For these sub-steps, tidymodels provides one or several packages. This article will showcase functions from four tidymodels packages:

rsample - Different types of re-samples
recipes - Transformations for model data pre-processing
parnip - A common interface for model creation
yardstick - Measure model performance

The following diagram illustrates each modeling step, and lines up the tidymodels packages that we will use in this article:

In a given analysis, a tidyverse package may or may not be used. Not all projects need to work with time variables, so there is no need to use functions from the hms package. The same idea applies to tidymodels. Depending on what type of modeling is going to be done, only functions from some its packages will be used.

An Example

We will use the iris data set for an example. Its data is already imported, and sufficiently tidy to move directly to modeling.

Load only the `tidymodels` library

This may be the first article I have written where only one package is called via library(). Apart from loading its core modeling packages, tidymodels also conveniently loads some tidyverse packages, including dplyr and ggplot2. Throughout this exercise, we will use some functions out of those packages, but we don’t have to explicitly load them into our R session.

library(tidymodels)

Pre-Process

This step focuses on making data suitable for modeling by using data transformations. All transformations can be accomplished with dplyr, or other tidyverse packages Consider using tidymodels packages when model development is more heavy and complex.

Data Sampling

The initial_split() function is specially built to separate the data set into a training and testing set. By default, it holds 3/4 of the data for training and the rest for testing. That can be changed by passing the prop argument. This function generates an rplit object, not a data frame. The printed output shows the row count for testing, training, and total.

iris_split <- initial_split(iris, prop = 0.6)
iris_split

## <90/60/150>

To access the observations reserved for training, use the training() function. Similarly, use testing() to access the testing data.

iris_split %>%
  training() %>%
  glimpse()

## Observations: 90
## Variables: 5
## $ Sepal.Length <dbl> 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.9, 5.4, 4…
## $ Sepal.Width  <dbl> 3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 3.1, 3.7, 3…
## $ Petal.Length <dbl> 1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.5, 1.5, 1…
## $ Petal.Width  <dbl> 0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.1, 0.2, 0…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…

These sampling functions are courtesy of the rsample package, which is part of tidymodels.

Pre-process interface

In tidymodels, the recipes package provides an interface that specializes in data pre-processing. Within the package, the functions that start, or execute, the data transformations are named after cooking actions. That makes the interface more user-friendly. For example:

recipe() - Starts a new set of transformations to be applied, similar to the ggplot() command. Its main argument is the model’s formula.
prep() - Executes the transformations on top of the data that is supplied (typically, the training data).

Each data transformation is a step. Functions correspond to specific types of steps, each of which has a prefix of step_. There are several step_ functions; in this example, we will use three of them:

step_corr() - Removes variables that have large absolute correlations with other variables
step_center() - Normalizes numeric data to have a mean of zero
step_scale() - Normalizes numeric data to have a standard deviation of one

Another nice feature is that the step can be applied to a specific variable, groups of variables, or all variables. The all_outocomes() and all_predictors() functions provide a very convenient way to specify groups of variables. For example, if we want the step_corr() to only analyze the predictor variables, we use step_corr(all_predictors()). This capability saves us from having to enumerate each variable.

In the following example, we will put together the recipe(), prep(), and step functions to create a recipe object. The training() function is used to extract that data set from the previously created split sample data set.

iris_recipe <- training(iris_split) %>%
  recipe(Species ~.) %>%
  step_corr(all_predictors()) %>%
  step_center(all_predictors(), -all_outcomes()) %>%
  step_scale(all_predictors(), -all_outcomes()) %>%
  prep()

If we call the iris_recipe object, it will print details about the recipe. The Operations section describes what was done to the data. One of the operations entries in the example explains that the correlation step removed the Petal.Length variable.

iris_recipe

## Data Recipe
## 
## Inputs:
## 
##       role #variables
##    outcome          1
##  predictor          4
## 
## Training data contained 90 data points and no missing data.
## 
## Operations:
## 
## Correlation filter removed Petal.Length [trained]
## Centering for Sepal.Length, Sepal.Width, Petal.Width [trained]
## Scaling for Sepal.Length, Sepal.Width, Petal.Width [trained]

Execute the pre-processing

The testing data can now be transformed using the exact same steps, weights, and categorization used to pre-process the training data. To do this, another function with a cooking term is used: bake(). Notice that the testing() function is used in order to extract the appropriate data set.

iris_testing <- iris_recipe %>%
  bake(testing(iris_split)) 

glimpse(iris_testing)

## Observations: 60
## Variables: 4
## $ Sepal.Length <dbl> -1.597601746, -1.138960096, 0.007644027, -0.7949788…
## $ Sepal.Width  <dbl> -0.41010139, 0.71517681, 2.06551064, 1.61539936, 0.…
## $ Petal.Width  <dbl> -1.2085003, -1.2085003, -1.2085003, -1.0796318, -1.…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…

Performing the same operation over the training data is redundant, because that data has already been prepped. To load the prepared training data into a variable, we use juice(). It will extract the data from the iris_recipe object.

iris_training <- juice(iris_recipe)

glimpse(iris_training)

## Observations: 90
## Variables: 4
## $ Sepal.Length <dbl> -0.7949789, -1.0242997, -1.2536205, -1.3682809, -0.…
## $ Sepal.Width  <dbl> 0.94023245, -0.18504575, 0.26506553, 0.04000989, 1.…
## $ Petal.Width  <dbl> -1.2085003, -1.2085003, -1.2085003, -1.2085003, -1.…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…

Model Training

In R, there are multiple packages that fit the same type of model. It is common for each package to provide a unique interface. In other words, things such as an argument for the same model attribute is defined differently for each package. For example, the ranger and randomForest packages fit Random Forest models. In the ranger() function, to define the number of trees we use num.trees. In randomForest, that argument is named ntree. It is not easy to switch between packages to run the same model.

Instead of replacing the modeling package, tidymodels replaces the interface. Better said, tidymodels provides a single set of functions and arguments to define a model. It then fits the model against the requested modeling package.

In the example below, the rand_forest() function is used to initialize a Random Forest model. To define the number of trees, the trees argument is used. To use the ranger version of Random Forest, the set_engine() function is used. Finally, to execute the model, the fit() function is used. The expected arguments are the formula and data. Notice that the model runs on top of the juiced trained data.

iris_ranger <- rand_forest(trees = 100, mode = "classification") %>%
  set_engine("ranger") %>%
  fit(Species ~ ., data = iris_training)

The payoff is that if we now want to run the same model against randomForest, we simply change the value in set_engine() to “randomForest”.

iris_rf <-  rand_forest(trees = 100, mode = "classification") %>%
  set_engine("randomForest") %>%
  fit(Species ~ ., data = iris_training)

It is also worth mentioning that the model is not defined in a single, large function with a lot of arguments. The model definition is separated into smaller functions such as fit() and set_engine(). This allows for a more flexible - and easier to learn - interface.

Predictions

Instead of a vector, the predict() function ran against a parsnip model returns a tibble. By default, the prediction variable is called .pred_class. In the example, notice that the baked testing data is used.

predict(iris_ranger, iris_testing)

## # A tibble: 60 x 1
##    .pred_class
##    <fct>      
##  1 setosa     
##  2 setosa     
##  3 setosa     
##  4 setosa     
##  5 setosa     
##  6 setosa     
##  7 setosa     
##  8 setosa     
##  9 setosa     
## 10 setosa     
## # … with 50 more rows

It is very easy to add the predictions to the baked testing data by using dplyr’s bind_cols() function.

iris_ranger %>%
  predict(iris_testing) %>%
  bind_cols(iris_testing) %>%
  glimpse()

## Observations: 60
## Variables: 5
## $ .pred_class  <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…
## $ Sepal.Length <dbl> -1.597601746, -1.138960096, 0.007644027, -0.7949788…
## $ Sepal.Width  <dbl> -0.41010139, 0.71517681, 2.06551064, 1.61539936, 0.…
## $ Petal.Width  <dbl> -1.2085003, -1.2085003, -1.2085003, -1.0796318, -1.…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…

Model Validation

Use the metrics() function to measure the performance of the model. It will automatically choose metrics appropriate for a given type of model. The function expects a tibble that contains the actual results (truth) and what the model predicted (estimate).

iris_ranger %>%
  predict(iris_testing) %>%
  bind_cols(iris_testing) %>%
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 x 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass     0.917
## 2 kap      multiclass     0.874

Because of the consistency of the new interface, measuring the same metrics against the randomForest model is as easy as replacing the model variable at the top of the code.

iris_rf %>%
  predict(iris_testing) %>%
  bind_cols(iris_testing) %>%
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 x 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass     0.883
## 2 kap      multiclass     0.824

Per classifier metrics

It is easy to obtain the probability for each possible predicted value by setting the type argument to prob. That will return a tibble with as many variables as there are possible predicted values. Their name will default to the original value name, prefixed with .pred_.

iris_ranger %>%
  predict(iris_testing, type = "prob") %>%
  glimpse()

## Observations: 60
## Variables: 3
## $ .pred_setosa     <dbl> 0.677480159, 0.978293651, 0.783250000, 0.983972…
## $ .pred_versicolor <dbl> 0.295507937, 0.011706349, 0.150833333, 0.001111…
## $ .pred_virginica  <dbl> 0.02701190, 0.01000000, 0.06591667, 0.01491667,…

Again, use bind_cols() to append the predictions to the baked testing data set.

iris_probs <- iris_ranger %>%
  predict(iris_testing, type = "prob") %>%
  bind_cols(iris_testing)

glimpse(iris_probs)

## Observations: 60
## Variables: 7
## $ .pred_setosa     <dbl> 0.677480159, 0.978293651, 0.783250000, 0.983972…
## $ .pred_versicolor <dbl> 0.295507937, 0.011706349, 0.150833333, 0.001111…
## $ .pred_virginica  <dbl> 0.02701190, 0.01000000, 0.06591667, 0.01491667,…
## $ Sepal.Length     <dbl> -1.597601746, -1.138960096, 0.007644027, -0.794…
## $ Sepal.Width      <dbl> -0.41010139, 0.71517681, 2.06551064, 1.61539936…
## $ Petal.Width      <dbl> -1.2085003, -1.2085003, -1.2085003, -1.0796318,…
## $ Species          <fct> setosa, setosa, setosa, setosa, setosa, setosa,…

Now that everything is in one tibble, it is easy to calculate curve methods. In this case we are using gain_curve().

iris_probs%>%
  gain_curve(Species, .pred_setosa:.pred_virginica) %>%
  glimpse()

## Observations: 141
## Variables: 5
## $ .level          <chr> "setosa", "setosa", "setosa", "setosa", "setosa"…
## $ .n              <dbl> 0, 1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, …
## $ .n_events       <dbl> 0, 1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, …
## $ .percent_tested <dbl> 0.000000, 1.666667, 5.000000, 6.666667, 8.333333…
## $ .percent_found  <dbl> 0.000000, 5.882353, 17.647059, 23.529412, 29.411…

The curve methods include an autoplot() function that easily creates a ggplot2 visualization.

iris_probs%>%
  gain_curve(Species, .pred_setosa:.pred_virginica) %>%
  autoplot()

This is an example of a roc_curve(). Again, because of the consistency of the interface, only the function name needs to be modified; even the argument values remain the same.

iris_probs%>%
  roc_curve(Species, .pred_setosa:.pred_virginica) %>%
  autoplot()

To measured the combined single predicted value and the probability of each possible value, combine the two prediction modes (with and without prob type). In this example, using dplyr’s select() makes the resulting tibble easier to read.

predict(iris_ranger, iris_testing, type = "prob") %>%
  bind_cols(predict(iris_ranger, iris_testing)) %>%
  bind_cols(select(iris_testing, Species)) %>%
  glimpse()

## Observations: 60
## Variables: 5
## $ .pred_setosa     <dbl> 0.677480159, 0.978293651, 0.783250000, 0.983972…
## $ .pred_versicolor <dbl> 0.295507937, 0.011706349, 0.150833333, 0.001111…
## $ .pred_virginica  <dbl> 0.02701190, 0.01000000, 0.06591667, 0.01491667,…
## $ .pred_class      <fct> setosa, setosa, setosa, setosa, setosa, setosa,…
## $ Species          <fct> setosa, setosa, setosa, setosa, setosa, setosa,…

Pipe the resulting table into metrics(). In this case, specify .pred_class as the estimate.

predict(iris_ranger, iris_testing, type = "prob") %>%
  bind_cols(predict(iris_ranger, iris_testing)) %>%
  bind_cols(select(iris_testing, Species)) %>%
  metrics(Species, .pred_setosa:.pred_virginica, estimate = .pred_class)

## # A tibble: 4 x 3
##   .metric     .estimator .estimate
##   <chr>       <chr>          <dbl>
## 1 accuracy    multiclass     0.917
## 2 kap         multiclass     0.874
## 3 mn_log_loss multiclass     0.274
## 4 roc_auc     hand_till      0.980

Closing remarks

This end-to-end example is intended to be a gentle introduction to tidymodels. The number of functions, and options of such functions, were kept at a minimum for the purposes of this demonstration, but there is much more that can be done with this wonderful group of packages. Hopefully, this article will help you get started, and maybe even encourage you to expand your knowledge further.

Thank you!

I would like to thank Max Kuhn and Davis Vaughan, the primary developers of tidymodels. They have been very gracious in providing instruction, feedback, and guidance throughout my journey of learning tidymodels.

Parsnipping Fama French

Thu, 14 Mar 2019 00:00:00 +0000

Today, we will continue our exploration of developments in the world of tidy models, and we will stick with our usual Fama French modeling flow to do so. For new readers who want get familiar with Fama French before diving into this post, see here where we covered importing and wrangling the data, here where we covered rolling models and visualization, and here where we covered managing many models. If you’re into Shiny, this flexdashboard might be of interest, as well.

Let’s get to it.

First, we need our data and, as usual, we’ll import data for daily prices of five ETFs, convert them to returns (have a look here for a refresher on that code flow), then import the five Fama French factor data and join it to our five ETF returns data. Here’s the code to make that happen (this code was covered in detail in this post:

symbols <- c("SPY", "EFA", "IJS", "EEM", "AGG")


# The prices object will hold our daily price data.
prices <- 
  getSymbols(symbols, 
             src = 'yahoo', 
             from = "2012-12-31",
             to = "2017-12-31",
             auto.assign = TRUE, 
             warnings = FALSE) %>% 
  map(~Ad(get(.))) %>% 
  reduce(merge) %>%
  `colnames<-`(symbols)


asset_returns_long <-  
  prices %>% 
  tk_tbl(preserve_index = TRUE, rename_index = "date") %>%
  gather(asset, prices, -date) %>% 
  group_by(asset) %>%  
  mutate(daily_returns = (log(prices) - log(lag(prices)))) %>% 
  na.omit()

factors_data_address <- 
"http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/Global_5_Factors_Daily_CSV.zip"

factors_csv_name <- "Global_5_Factors_Daily.csv"

temp <- tempfile()

download.file(
  # location of file to be downloaded
  factors_data_address,
  # where we want R to store that file
  temp, 
  quiet = TRUE)


Global_5_Factors <- 
  read_csv(unz(temp, factors_csv_name), skip = 6 ) %>%
  rename(date = X1, MKT = `Mkt-RF`) %>%
  mutate(date = ymd(parse_date_time(date, "%Y%m%d")))%>%
  mutate_if(is.numeric, funs(. / 100)) %>% 
  select(-RF)

data_joined_tidy <- 
  asset_returns_long %>%
  left_join(Global_5_Factors, by = "date") %>% 
  na.omit()

For today, let’s work with just the SPY data by filtering our data set by asset.

spy_2013_2017 <- data_joined_tidy %>% 
  filter(asset == "SPY")

Next, we re-sample this five years’ worth of data into smaller subsets of training and testing sets. This is frequently done by k-fold cross validation (see here for an example), where random samples are taken from the data, but since we are working with time series, we will use a time-aware technique. The rsample package has a function for exactly this purpose, the rolling_origin() function. We covered this process extensively in this previous post. Here’s the code to make it happen.

rolling_origin_spy_2013_2017 <- 
 rolling_origin(
  data       = spy_2013_2017,
  initial    = 100,
  assess     = 1,
  cumulative = FALSE
)

rolling_origin_spy_2013_2017 %>% 
  dim()

[1] 1159    2

We now have a data object called rolling_origin_spy_2013_2017 that holds 1159 splits of data. Each split consists of an analysis data set with 100 days of return and factor data, and an assessment data set with one day of return and factor data.

Now, we can start using that collection of data splits to fit a model on the assessment data, and then test our model on the assessment data. That means it’s time to introduce a relatively new addition to the R tool chain, the parsnip package.

parsnip is a unified model interface that allows us to create a model specification, set an analytic engine, and then fit a model. It’s a ‘unified’ interface in the sense that we can use the same scaffolding but insert different models, or different engines, or different modes. Let’s see how that works with linear regression.

Recall that in the previous post, we piped our data into a linear model like so:

analysis(rolling_origin_spy_2013_2017$splits[[1]]) %>% 
do(model = lm(daily_returns ~ MKT + SMB + HML + RMW + CMA, 
      data = .)) %>% 
tidy(model)

# A tibble: 6 x 6
# Groups:   asset [1]
  asset term         estimate std.error statistic  p.value
  <chr> <chr>           <dbl>     <dbl>     <dbl>    <dbl>
1 SPY   (Intercept)  0.000579  0.000338      1.71 8.98e- 2
2 SPY   MKT          0.909     0.0739       12.3  2.79e-21
3 SPY   SMB         -0.495     0.112        -4.43 2.52e- 5
4 SPY   HML         -0.609     0.208        -2.92 4.38e- 3
5 SPY   RMW         -0.591     0.259        -2.28 2.47e- 2
6 SPY   CMA         -0.395     0.206        -1.92 5.81e- 2

Now, we will pipe into the parsnip scaffolding, which will allow us to quickly change to a different model and specification further down in the code.

Since we are running a linear regression, we first create a specification with linear_reg(), then set the engine with set_engine("lm"), and finally fit the model with fit(five_factor_model, data = one of our splits)

lm_model <-
  linear_reg() %>%
  set_engine("lm") %>%
  fit(daily_returns ~ MKT + SMB + HML + RMW + CMA, 
      data = analysis(rolling_origin_spy_2013_2017$splits[[1]]))

lm_model

parsnip model object


Call:
stats::lm(formula = formula, data = data)

Coefficients:
(Intercept)          MKT          SMB          HML          RMW  
  0.0005794    0.9086303   -0.4951297   -0.6085088   -0.5910375  
        CMA  
 -0.3954515

Now that we’ve fit the model on our test set, let’s see how well it predicted the test set. We can use the predict() function and pass it the results of our parnsip code flow, along with the assessment split.

assessment(rolling_origin_spy_2013_2017$splits[[1]]) %>% 
  select(returns) %>% 
  bind_cols(predict(lm_model, 
        new_data = assessment(rolling_origin_spy_2013_2017$splits[[1]])))

# A tibble: 1 x 3
# Groups:   asset [1]
  asset returns   .pred
  <chr>   <dbl>   <dbl>
1 SPY      148. 0.00737

That worked well, but now let’s head to a more complex model and use the ranger package as an engine for a random forest analysis.

To set up the ranger random forest model in parsnip, we first use rand_forest(mode = "regression", mtry = 3, trees = 100) to create the specification, set_engine("ranger") to set the engine as the ranger package, and fit(daily_returns ~ MKT + SMB + HML + RMW + CMA ~ , data = analysis(rolling_origin_spy_2013_2017$splits[[1]]) to fit the five-factor Fama French model to the 100-day sample in our first split.

# Need to load the packages to be used as the random forest engine
library(ranger)

rand_forest(mode = "regression", mtry = 3, trees = 100) %>%
  set_engine("ranger") %>%
  fit(daily_returns ~ MKT + SMB + HML + RMW + CMA, 
      data = analysis(rolling_origin_spy_2013_2017$splits[[1]]))

parsnip model object

Ranger result

Call:
 ranger::ranger(formula = formula, data = data, mtry = ~3, num.trees = ~100,      num.threads = 1, verbose = FALSE, seed = sample.int(10^5,          1)) 

Type:                             Regression 
Number of trees:                  100 
Sample size:                      100 
Number of independent variables:  5 
Mtry:                             3 
Target node size:                 5 
Variable importance mode:         none 
Splitrule:                        variance 
OOB prediction error (MSE):       1.514654e-05 
R squared (OOB):                  0.6880896

Notice that ranger gives us an OOB prediction error (MSE) value as part of its return. parsnip returns to us what the underlying engine returns.

Now, let’s apply that random forest regression to all 1159 of our splits (recall that each split consists of 100 days of training data and one day of test data), so we can get an average RMSE. Warning: this will consume some resources on your machine and some time in your day.

To apply that model to our entire data set, we create a function that takes one split, passes it to our parsnip enabled model, and then uses the predict function to attempt to predict our assessment split. The function also allows us to specify the number of trees and the number of variables randomly sampled at each tree split, which is set with the mtry argument.

ranger_rf_regress <- function(mtry = 3, trees = 5, split){
    
    analysis_set_rf <- analysis(split)
     
    model <- 
      rand_forest(mtry = mtry, trees = trees) %>%
        set_engine("ranger") %>%
        fit(daily_returns ~ MKT + SMB + HML + RMW + CMA, data = analysis_set_rf)

    
    assessment_set_rf <- assessment(split)

    assessment_set_rf %>%
      select(date, daily_returns) %>%
      mutate(.pred = unlist(predict(model, new_data = assessment_set_rf))) %>% 
      select(date, daily_returns, .pred)
   
}

Now we want to pass it our object of 1159 splits, rolling_origin_spy_2013_2017$splits, and we want the function to iterate over each split. For that we turn to map_df() from the purrr package, which allows us to iterate over the data object and return a data frame. map_df() takes the data as an argument and our function as an argument.

ranger_results <- 
  map_df(.x = rolling_origin_spy_2013_2017$splits,
         ~ranger_rf_regress(mtry = 3, trees = 100, split = .x))

Here are the results. We now have 1159 predictions.

ranger_results %>% 
  head()

# A tibble: 6 x 4
# Groups:   asset [1]
  asset date       daily_returns    .pred
  <chr> <date>             <dbl>    <dbl>
1 SPY   2013-05-28       0.00597  0.00583
2 SPY   2013-05-29      -0.00652 -0.00403
3 SPY   2013-05-30       0.00369  0.00658
4 SPY   2013-05-31      -0.0145  -0.0114 
5 SPY   2013-06-03       0.00549  0.00119
6 SPY   2013-06-04      -0.00482  0.00202

Notice how the date of each prediction is included since we included it in the select() call in our function. That will come in handy for charting later.

Now, we can use the rmse() function from yardstick to calculate the root mean-squared error each of our predictions (our test sets had only one observation in them because we were testing on one month, so the RMSE is not a complex calculation here, but it would be the same code pattern if we had a larger test set). We can then find the average RMSE by calling summarise(avg_rmse = mean(.estimate)).

library(yardstick)

ranger_results %>%
  group_by(date) %>% 
  rmse(daily_returns, .pred) %>% 
  summarise(avg_rmse = mean(.estimate))

# A tibble: 1 x 1
  avg_rmse
     <dbl>
1  0.00253

We have the average RMSE; let’s see if the RMSE were stable over time, first with ggplot.

ranger_results %>%
  group_by(date) %>% 
  rmse(daily_returns, .pred) %>% 
  ggplot(aes(x = date, y = .estimate)) +
  geom_point(color = "cornflowerblue") +
  labs(y = "rmse", x = "", title = "RMSE over time via Ranger RF")

And with highcharter.

ranger_results %>%
  group_by(date) %>% 
  rmse(daily_returns, .pred) %>% 
  hchart(., hcaes(x = date, y = .estimate),
         type = "point") %>% 
  hc_title(text = "RMSE over time via Ranger RF") %>% 
  hc_yAxis(title = list(text = "RMSE"))

It looks like our RMSE is relatively stable, except for a period in mid to late 2015.

The amazing power of parsnip is how efficiently we can toggle to another random forest engine. Let’s suppose we wished to use the randomForest package instead of ranger. Here’s how we could reconfigure our previous work to use a different engine.

First, we’ll load up the randomForest package, because we need to load the package in order to use it as our engine. Then, we make one tweak to the original ranger_rf_regress function, by changing set_engine("ranger") to set_engine("randomForest"). That’s all, and we’re now running a random forest model using a different package.

library(randomForest)

randomForest_rf_regress <- function(mtry = 3, trees = 5, split){
    
    analysis_set_rf <- analysis(split)
     
    model <- 
      rand_forest(mtry = mtry, trees = trees) %>%
        set_engine("randomForest") %>%
        fit(daily_returns ~ MKT + SMB + HML + RMW + CMA, data = analysis_set_rf)

    
    assessment_set_rf <- assessment(split)

    assessment_set_rf %>%
      select(date, daily_returns) %>%
      mutate(.pred = unlist(predict(model, new_data = assessment_set_rf))) %>% 
      select(date, daily_returns, .pred)
   
}

We now have a new function called randomForest_rf_regress() that uses randomForest as the engine for our model and can use the same code scaffolding to run that model on our 1159 splits.

randomForest_results <- 
  map_df(.x = rolling_origin_spy_2013_2017$splits,
         ~randomForest_rf_regress(mtry = 3, trees = 100, split = .x))

randomForest_results %>% 
  head()

# A tibble: 6 x 4
# Groups:   asset [1]
  asset date       daily_returns    .pred
  <chr> <date>             <dbl>    <dbl>
1 SPY   2013-05-28       0.00597  0.00609
2 SPY   2013-05-29      -0.00652 -0.00438
3 SPY   2013-05-30       0.00369  0.00597
4 SPY   2013-05-31      -0.0145  -0.00987
5 SPY   2013-06-03       0.00549  0.00134
6 SPY   2013-06-04      -0.00482  0.00118

And we can use the same yardstick code to extract the RMSE.

randomForest_results %>%
  group_by(date) %>% 
  rmse(daily_returns, .pred) %>% 
  summarise(avg_rmse = mean(.estimate))

# A tibble: 1 x 1
  avg_rmse
     <dbl>
1  0.00252

There’s a lot more to explore in the parsnip package and the tidymodels collection. See you next time when we’ll get into some classification!

Wait: shameless book plug for those who read to the end: if you like this sort of thing, check out my new book Reproducible Finance with R!

Tidymodels on R Views

Modern Rule-Based Models

Modern Rule-Based Models

What is a rule?

C5.0 rules

Cubist

RuleFit

Acknowledgments

References

The Case for tidymodels

A Gentle Introduction to tidymodels

An Example

Load only the tidymodels library

Pre-Process

Data Sampling

Pre-process interface

Execute the pre-processing

Model Training

Predictions

Model Validation

Per classifier metrics

Closing remarks

Thank you!

Parsnipping Fama French

Load only the `tidymodels` library