ggplot2 on R Views

Accelerate your plots with ggforce

Thu, 19 Sep 2019 00:00:00 +0000

In this post, I will walk you through some examples that show off the major features of the ggforce package. The main goal is to share a few ideas about customizing visualizations that you may find useful in your everyday work.

The ggforce package is an extension to ggplot2 developed by Thomas Pedersen. Thanks to ggforce, you can enhance almost any ggplot by highlighting data groupings, and focusing attention on interesting features of the plot. The package contains geoms, stats, facets, and other ggplot functions. Among such functions, there are some for marking the convex hull of a set of points, jittering data, and creating Voronoi plots.

Base `ggplot`

The examples in this article will use data from the nycflights13 package. Most of the examples will build on the same basic ggplot that visualizes airports by geographical location. I am using this data set because it makes it easy to plot x/y coordinates without having to remember what they “mean”. This basic plot will be saved to a variable, and then that variable will be used as the base of the examples of enhancing the visualization using ggforce

library(tidyverse)
library(ggforce)
library(nycflights13)

p <- airports %>%
  filter(lon < 0, tzone != "\\N") %>%
  ggplot(aes(lon, lat, color = tzone)) + 
  geom_point(show.legend = FALSE)  

p

Make your `mark` with `ggforce`

I have long been waiting for an easy way to draw an outline around groups of data. The geom_mark_...() family of functions does exactly that. There are four mark functions in ggforce, all different based on the shape they draw around the group:

geom_mark_circle()
geom_mark_ellipse()
geom_mark_hull()
geom_mark_rect()

Let’s start with geom_mark_rect(); it will draw a rounded rectangle around each time zone group.

p +
  geom_mark_rect()

Like magic! The rectangles look amazing, even without modifying any arguments. Of course, more customization is possible via setting arguments. In this post, I will review some of the many great arguments available in ggforce functions, but I don’t want to rob you of the fun of trying it yourself and discovering all of the different options.

Label, and an arrow!

This next addition to our plot deserves its own subheading. Adding a label and an arrow pointing to a group would typically be a major undertaking. Without ggforce, this would require manually adding both the text and the arrow to the ggplot. But, with geom_mark it is a simple as setting the label argument. So, without further ado, here is the label argument in action:

p + 
  geom_mark_rect(aes(label = tzone))

The labels and arrows are not only drawn, but they are also placed in an optimized location. In addition, the position will recalculate if the plot is re-sized! There are too many little details about this label argument to mention. The backdrop is automatically white, the indicator is not really an arrow, it is a simple line that also underlines the text, so it is easy for the eye to know which group belongs to which label.

It is now easy to finalize the plot by resetting the theme, and again suppressing the legend using show.legend.

p + 
  geom_mark_rect(aes(label = tzone), show.legend = FALSE) +
  theme_void()

Hull-k, enhance!

There are many cases where drawing a rectangle or circle around the groups is not ideal, or even preferable. The geom_mark_hull() essentially traces a more complex polygon around the shape of the outline of the group.

p + 
  geom_mark_hull(aes(label = tzone)) +
  theme_void()

Again, without adding any arguments to the function, the traced outline already looks wonderful. Another option to add now is fill. And since the legend table is now redundant, it can be suppressed by setting show.legend to FALSE.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE) +
  theme_void()

Notice that the fill color is not totally opaque; by default, ggforce has set the translucency lower to make sure that the dots are visible. This is something that I would have done anyway, usually by adding the alpha argument. In this case, it saves having to remember to add that argument.

Another adjustment that I thought was important for this plot was to modify the size of the hull, to change the padding around the outline of the group. The expand argument controls this aesthetic; it is possible to change it using the units() command.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_void()

Axe `theme_void()`

To finalize plots such as this one, it is necessary to remove most components from the default theme. Usually, theme_void() does the trick. For printed or online articles with white backgrounds, which is essentially all of them, it is often hard to determine the margins of the plot. theme_no_axes() provides a great compromise by removing all but the one element.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_no_axes()

Another facet of `ggforce`, and it’s magnify-cent

It is common to produce two plots, one to show the full picture, and another to magnify or focus on a specific area. With facet_zoom(), it is incredibly easy to show “macro” and “micro” in one plot by using the same xlim and ylim arguments to focus on an area of a plot.

p +
  facet_zoom(xlim = c(-155, -160.5), ylim = c(19, 22.3))

Skip the coordinates

Another cool feature of facet_zoom() is the ability to set the zoom region based on a row selection. To do this, simply pass an expression that you would use in a function such as filter() to the facet. So instead of using coordinates, I just tell the facet to zoom on anything that has a Pacific/Honolulu time zone.

p +
  facet_zoom(xy = tzone == "Pacific/Honolulu")

Putting it all together, with three lines of code

Using what has been covered so far, it is easy go from a very simple point plot to a sophisticated and nice-looking visualization with just three lines of code, thanks to ggforce.

p +
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_no_axes() +
  facet_zoom(x = tzone == "America/Los_Angeles")

“What is a Voronoi?”

This section title is based on my first reaction when I heard the word “Voronoi”. I have since learned about it, and can see why the Voronoi Diagram can be useful for very specific use cases. The good news is that if you encounter one of those use cases, you know that it is easy to draw it up in ggplot using geom_voronoi_segment().

The idea behind a Voronoi diagram is to split the area of the plot into as many sections as there are points. Unlike a grid or heat map, Voronoi draws custom shapes for each point based on the proximity to other points. It returns a plot that looks like stained glass. This can be good to determine the closest point inside each area. For example, a retailer can use it to see the area their store locations cover, and can help them make decisions to optimize their location based on the size of each Voronoi shape.

The following example will focus on airports in Alaska. The ggplot will zoom into that state’s general location, and then trace a hull shape. The hull will provide a quasi-map overlay. The final step is to add the Voronoi diagram layer by calling the function: geom_voronoi_segment()

p +
  geom_mark_hull(aes(fill = tzone), expand = unit(3, "mm")) +
  coord_cartesian(xlim = c(-130, -180), ylim = c(50, 75))  +
  geom_voronoi_segment()

Parallel to alluvial

The geom_parallel... functions allow visualizing interactions between categorical variables. The implementation is generic enough to create Sankey or alluvial charts.

For this, I will use the Manufacturer and Engine data from the planes table inside nycflights13. In this case, some simple data preparation is needed first.

prep_planes <- planes %>%
  filter(year > 1998, year < 2005) %>%
  filter(engine != "Turbo-shaft") %>%
  select(manufacturer, engine) %>%
  head(500)

prep_planes

## # A tibble: 500 x 2
##    manufacturer     engine   
##    <chr>            <chr>    
##  1 EMBRAER          Turbo-fan
##  2 AIRBUS INDUSTRIE Turbo-fan
##  3 AIRBUS INDUSTRIE Turbo-fan
##  4 EMBRAER          Turbo-fan
##  5 AIRBUS INDUSTRIE Turbo-fan
##  6 AIRBUS INDUSTRIE Turbo-fan
##  7 AIRBUS INDUSTRIE Turbo-fan
##  8 AIRBUS INDUSTRIE Turbo-fan
##  9 AIRBUS INDUSTRIE Turbo-fan
## 10 EMBRAER          Turbo-fan
## # … with 490 more rows

Prep for plotting with one line

The gather_set_data() is a convenience function that, just like gather(), creates a single line for each combination of categorical variables. The table contains three new columns - id, x, and y - which contain the combinations that each new row represents, and the row ID number from the original table.

prep_planes %>%
  gather_set_data(1:2)

## # A tibble: 1,000 x 5
##    manufacturer     engine       id x            y               
##    <chr>            <chr>     <int> <chr>        <chr>           
##  1 EMBRAER          Turbo-fan     1 manufacturer EMBRAER         
##  2 AIRBUS INDUSTRIE Turbo-fan     2 manufacturer AIRBUS INDUSTRIE
##  3 AIRBUS INDUSTRIE Turbo-fan     3 manufacturer AIRBUS INDUSTRIE
##  4 EMBRAER          Turbo-fan     4 manufacturer EMBRAER         
##  5 AIRBUS INDUSTRIE Turbo-fan     5 manufacturer AIRBUS INDUSTRIE
##  6 AIRBUS INDUSTRIE Turbo-fan     6 manufacturer AIRBUS INDUSTRIE
##  7 AIRBUS INDUSTRIE Turbo-fan     7 manufacturer AIRBUS INDUSTRIE
##  8 AIRBUS INDUSTRIE Turbo-fan     8 manufacturer AIRBUS INDUSTRIE
##  9 AIRBUS INDUSTRIE Turbo-fan     9 manufacturer AIRBUS INDUSTRIE
## 10 EMBRAER          Turbo-fan    10 manufacturer EMBRAER         
## # … with 990 more rows

The ggplot is primed with x for x, and then new aesthetics: id, split and value. For id, we pass the id column, split takes y, and finally, value is fixed to 1. The value is used to express the amount of “thickness” to add to that particular relationship; using 1 means that all combinations are weighted the same. At this point, the only argument to pass geom_parallel_sets() will be the color fill; in this case we will use engine.

Plotting with parallel

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine))

The plot shows how a specific plane’s engine relates to each of the manufacturers. Next geom_parallel_sets_axes() provides a terminal box; the axis.width argument is the only one necessary to use at this stage, and we will set it to 0.1. The labels are added by using geom_parallel_sets_labels(), and they are automatically rotated.

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine)) +
  geom_parallel_sets_axes(axis.width = 0.1) +
  geom_parallel_sets_labels()

The following is done to finalize the plot:

geom_parallel_sets() - Hide the legend and lower the alpha
geom_parallel_sets_axes() - Change the fill color and font color
geom_parallel_sets_labels() - Remove the rotation of the label

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine), show.legend = FALSE, alpha = 0.3) +
  geom_parallel_sets_axes(axis.width = 0.1, color = "lightgrey", fill = "white") +
  geom_parallel_sets_labels(angle = 0) +
  theme_no_axes()

Danger zone!

When visualizing the combination of a continuous and a categorical variable, it is common practice to resort to a bar or column plot. Cases that require representing this in a single circle shape usually involve modifying a polar bar in ggplot. But, this is much easier now with ggforce. I start with the total number of planes by engine planes:

planes %>%
  count(engine)

## # A tibble: 6 x 2
##   engine            n
##   <chr>         <int>
## 1 4 Cycle           2
## 2 Reciprocating    28
## 3 Turbo-fan      2750
## 4 Turbo-jet       535
## 5 Turbo-prop        2
## 6 Turbo-shaft       5

and then pipe those results into ggplot using geom_arc_bar() to create the circle-shaped plot. The new aesthetics employed here are: x0, y0, r0, r, amount, and explode. The x, y, and r aesthetics refer to the position and the radius of the circle. Since only one plot is needed, I fix x and y to 0. For radius, the r0 refers to the inside of the circle, and r to the outside. Setting r0 to 0.7 and r to 1 will create a sort of doughnut with a 0.3 thickness. Finally, I use “pie” as the stat.

planes %>%
  count(engine) %>%
  ggplot() +
  geom_arc_bar(aes(x0 = 0, y0 = 0, r0 = 0.7, r = 1, amount = n, fill = engine), alpha = 0.3, stat = "pie")

Another cool thing this geom does is to make it east to “break-away” one or several segments of the plot. The explode aesthetic controls that. To break away the “Turbo-jet” results, I create a new column called focus, setting it to 0.2 if it is part of that engine group, and to 0 if it is not, then finish up with theme_no_axes().

planes %>%
  count(engine) %>%
  mutate(focus = ifelse(engine == "Turbo-jet", 0.2, 0)) %>%
  ggplot() +
  geom_arc_bar(aes(x0 = 0, y0 = 0, r0 = 0.7, r = 1, amount = n, fill = engine, explode = focus), alpha = 0.3, stat = "pie") +
  theme_no_axes()

This section is titled “Danger Zone”, because hanging the r0 in geom_arc_bar() may change the look of the plot to one that has fallen out of favor. That plot type happens to be the same name of the stat that we are using.

Closing remarks

ggforce is a great package that does a lot more than what I covered today. My hope is to have shared one or two things that will encourage you to try ggforce in your everyday work.

Special thanks to Thomas Pedersen, the author of the package and a co-worker of mine. His contributions to the R community also include the tidygraph and ggraph packages, which I wrote about in this blog post a few months back.

Graph analysis using the tidyverse

Wed, 06 Mar 2019 00:00:00 +0000

It is because I am not a graph analysis expert that I thought it important to write this article. For someone who thinks in terms of single rectangular data sets, it is a bit of a mental leap to understand how to apply tidy principles to a more robust object, such as a graph table. Thankfully, there are two packages that make this work much easier:

tidygraph - Provides a way for dplyr to interact with graphs
ggraph - Extension to ggplot2 for graph analysis

Quick intro

Simply put, graph theory studies relationships between objects in a group. Visually, we can think of a graph as a series of interconnected circles, each representing a member of a group, such as people in a Social Network. Lines drawn between the circles represent a relationship between the members, such as friendships in a Social Network. Graph analysis helps with figuring out things such as the influence of a certain member, or how many friends are in between two members. A more formal definition and detailed explanation of Graph Theory can be found in Wikipedia here.

Example

Using an example, this article will introduce concepts of graph analysis work, and how tidyverse and tidyverse-adjacent tools can be used for such analysis.

Data source

The tidytuesday weekly project encourages new and experienced users to use the tidyverse tools to analyze data sets that change every week. I have been using that opportunity to lean new tools and techniques. One of the most recent data sets relates to French trains; it contains aggregate daily total trips per connecting stations.

library(readr)

url <- "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-02-26/small_trains.csv"
small_trains <- read_csv(url)

head(small_trains)

## # A tibble: 6 x 13
##    year month service departure_stati… arrival_station journey_time_avg
##   <int> <int> <chr>   <chr>            <chr>                      <dbl>
## 1  2017     9 Nation… PARIS EST        METZ                        85.1
## 2  2017     9 Nation… REIMS            PARIS EST                   47.1
## 3  2017     9 Nation… PARIS EST        STRASBOURG                 116. 
## 4  2017     9 Nation… PARIS LYON       AVIGNON TGV                161. 
## 5  2017     9 Nation… PARIS LYON       BELLEGARDE (AI…            164. 
## 6  2017     9 Nation… PARIS LYON       BESANCON FRANC…            129. 
## # … with 7 more variables: total_num_trips <int>,
## #   avg_delay_all_departing <dbl>, avg_delay_all_arriving <dbl>,
## #   num_late_at_departure <int>, num_arriving_late <int>,
## #   delay_cause <chr>, delayed_number <dbl>

Data Preparation

Even though it was meant to analyze delays, I thought it would be interesting to use the data to understand how stations connect with each other. A new summarized data set is created, called routes, which contains a single entry for each connected station. It also includes the average journey time it takes to go between stations.

library(dplyr)

routes <- small_trains %>%
  group_by(departure_station, arrival_station) %>%
  summarise(journey_time = mean(journey_time_avg)) %>%
  ungroup() %>%
  mutate(from = departure_station, 
         to = arrival_station) %>%
  select(from, to, journey_time)

routes

## # A tibble: 130 x 3
##    from                       to                 journey_time
##    <chr>                      <chr>                     <dbl>
##  1 AIX EN PROVENCE TGV        PARIS LYON                186. 
##  2 ANGERS SAINT LAUD          PARIS MONTPARNASSE         97.5
##  3 ANGOULEME                  PARIS MONTPARNASSE        146. 
##  4 ANNECY                     PARIS LYON                225. 
##  5 ARRAS                      PARIS NORD                 52.8
##  6 AVIGNON TGV                PARIS LYON                161. 
##  7 BARCELONA                  PARIS LYON                358. 
##  8 BELLEGARDE (AIN)           PARIS LYON                163. 
##  9 BESANCON FRANCHE COMTE TGV PARIS LYON                131. 
## 10 BORDEAUX ST JEAN           PARIS MONTPARNASSE        186. 
## # … with 120 more rows

The next step is to transform the tidy data set, into a graph table. In order to prepare routes for this transformation, it has to contain two variables specifically named: from and to, which are the names that tidygraph expects to see. Those variables should contain the name of each member (e.g., “AIX EN PROVENCE TGV”), and the relationship (“AIX EN PROVENCE TGV” -> “PARIS LYON”) .

In graph terminology, a member of the group is called a node (or vertex) in the graph, and a relationship between nodes is called an edge.

library(tidygraph)

graph_routes <- as_tbl_graph(routes)

graph_routes

## # A tbl_graph: 59 nodes and 130 edges
## #
## # A directed simple graph with 1 component
## #
## # Node Data: 59 x 1 (active)
##   name               
##   <chr>              
## 1 AIX EN PROVENCE TGV
## 2 ANGERS SAINT LAUD  
## 3 ANGOULEME          
## 4 ANNECY             
## 5 ARRAS              
## 6 AVIGNON TGV        
## # … with 53 more rows
## #
## # Edge Data: 130 x 3
##    from    to journey_time
##   <int> <int>        <dbl>
## 1     1    39        186. 
## 2     2    40         97.5
## 3     3    40        146. 
## # … with 127 more rows

The as_tbl_graph() function splits the routes table into two:

Node Data - Contains all of the unique values found in the from and to variables. In this case, it is a table with a single column containing the names of all of the stations.
Edge Data - Is a table of all relationships between from and to. A peculiarity of tidygraph is that it uses the row position of the node as the identifier for from and to, instead of its original name.

Another interesting thing about tidygraph is that it allows us to attach more information about the node or edge in an additional column. In this case, journey_time is not really needed to create the graph table, but it may be needed for the analysis we plan to perform. The as_tbl_graph() function automatically created the column for us.

Thinking about graph_routes as two tibbles inside a larger table graph, was one of the two major mental breakthroughs I had during this exercise. At that point, it became evident that dplyr needs a way to know which of the two tables (nodes or edges) to perform the transformations on. In tidygraph, this is done using the activate() function. To showcase this, the nodes table will be “activated” in order to add two new string variables derived from name.

library(stringr)

graph_routes <- graph_routes %>%
  activate(nodes) %>%
  mutate(
    title = str_to_title(name),
    label = str_replace_all(title, " ", "\n")
    )

graph_routes

## # A tbl_graph: 59 nodes and 130 edges
## #
## # A directed simple graph with 1 component
## #
## # Node Data: 59 x 3 (active)
##   name                title               label                   
##   <chr>               <chr>               <chr>                   
## 1 AIX EN PROVENCE TGV Aix En Provence Tgv "Aix\nEn\nProvence\nTgv"
## 2 ANGERS SAINT LAUD   Angers Saint Laud   "Angers\nSaint\nLaud"   
## 3 ANGOULEME           Angouleme           Angouleme               
## 4 ANNECY              Annecy              Annecy                  
## 5 ARRAS               Arras               Arras                   
## 6 AVIGNON TGV         Avignon Tgv         "Avignon\nTgv"          
## # … with 53 more rows
## #
## # Edge Data: 130 x 3
##    from    to journey_time
##   <int> <int>        <dbl>
## 1     1    39        186. 
## 2     2    40         97.5
## 3     3    40        146. 
## # … with 127 more rows

It was really impressive how easy it was to manipulate the graph table, because once one of the two tables are activated, all of the changes can be made using tidyverse tools. The same approach can be used to extract data from the graph table. In this case, a list of all the stations is pulled into a single character vector.

stations <- graph_routes %>%
  activate(nodes) %>%
  pull(title)

stations

##  [1] "Aix En Provence Tgv"            "Angers Saint Laud"             
##  [3] "Angouleme"                      "Annecy"                        
##  [5] "Arras"                          "Avignon Tgv"                   
##  [7] "Barcelona"                      "Bellegarde (Ain)"              
##  [9] "Besancon Franche Comte Tgv"     "Bordeaux St Jean"              
## [11] "Brest"                          "Chambery Challes Les Eaux"     
## [13] "Dijon Ville"                    "Douai"                         
## [15] "Dunkerque"                      "Francfort"                     
## [17] "Geneve"                         "Grenoble"                      
## [19] "Italie"                         "La Rochelle Ville"             
## [21] "Lausanne"                       "Laval"                         
## [23] "Le Creusot Montceau Montchanin" "Le Mans"                       
## [25] "Lille"                          "Lyon Part Dieu"                
## [27] "Macon Loche"                    "Madrid"                        
## [29] "Marne La Vallee"                "Marseille St Charles"          
## [31] "Metz"                           "Montpellier"                   
## [33] "Mulhouse Ville"                 "Nancy"                         
## [35] "Nantes"                         "Nice Ville"                    
## [37] "Nimes"                          "Paris Est"                     
## [39] "Paris Lyon"                     "Paris Montparnasse"            
## [41] "Paris Nord"                     "Paris Vaugirard"               
## [43] "Perpignan"                      "Poitiers"                      
## [45] "Quimper"                        "Reims"                         
## [47] "Rennes"                         "Saint Etienne Chateaucreux"    
## [49] "St Malo"                        "St Pierre Des Corps"           
## [51] "Strasbourg"                     "Stuttgart"                     
## [53] "Toulon"                         "Toulouse Matabiau"             
## [55] "Tourcoing"                      "Tours"                         
## [57] "Valence Alixan Tgv"             "Vannes"                        
## [59] "Zurich"

Visualizing

In graphs, the absolute position of the each node is not as relevant as it is with other kinds of visualizations. A very minimal ggplot2 theme is set to make it easier to view the plotted graph.

library(ggplot2)

thm <- theme_minimal() +
  theme(
    legend.position = "none",
     axis.title = element_blank(),
     axis.text = element_blank(),
     panel.grid = element_blank(),
     panel.grid.major = element_blank(),
  ) 

theme_set(thm)

To create the plot, start with ggraph() instead of ggplot2(). The ggraph package contains geoms that are unique to graph analysis. The package contains geoms to specifically plot nodes, and other geoms for edges.

As a first basic test, the point geom will be used, but instead of callinggeom_point(), we call geom_node_point(). The edges are plotted using geom_edge_diagonal().

library(ggraph) 

graph_routes %>%
  ggraph(layout = "kk") +
    geom_node_point() +
    geom_edge_diagonal()

To make it easier to see where each station is placed in this plot, the geom_node_text() is used. Just as with regular geoms in ggplot2, other attributes such as size, color, and alpha can be modified.

graph_routes %>%
  ggraph(layout = "kk") +
    geom_node_text(aes(label = label, color = name), size = 3) +
    geom_edge_diagonal(color = "gray", alpha = 0.4)

Morphing time!

The second mental leap was understanding how a graph algorithm is applied. Typically, the output of a model function is a model object, not a data object. With tidygraph, the process begins and ends with a graph table. The steps are these:

Start with a graph table
Temporarily transform the graph to comply with the model that is requested (morph())
Add additional transformations to the morphed data using dplyr (optional)
Restore the original graph table, but modified to keep the changes made during the morph

The shortest path algorithm defines the “length” as the number of edges in between two nodes. There may be multiple routes to get from point A to point B, but the algorithm chooses the one with the fewest number of “hops”. The way to call the algorithm is inside the morph() function. Even though to_shortest_path() is a function in itself, and it is possible run it without morph(), it is not meant to be used that way. In the example, the journey_time is used as weights to help the algorithm find an optimal route between the Arras and the Nancy stations. The print output of the morphed graph will not be like the original graph table.

from <- which(stations == "Arras")
to <-  which(stations == "Nancy")

shortest <- graph_routes %>%
  morph(to_shortest_path, from, to, weights = journey_time)

shortest

## # A tbl_graph temporarily morphed to a shortest path representation
## # 
## # Original graph is a directed simple graph with 1 component
## # consisting of 59 nodes and 130 edges

It is possible to make more transformations with the use of activate() and dplyr functions. The results can be previewed, or committed back to the original R variable using unmorph(). By default, nodes are active in a morphed graph, so there is no need to set that explicitly.

shortest %>%
  mutate(selected_node = TRUE) %>%
  unmorph()

## # A tbl_graph: 59 nodes and 130 edges
## #
## # A directed simple graph with 1 component
## #
## # Node Data: 59 x 4 (active)
##   name               title              label                 selected_node
##   <chr>              <chr>              <chr>                 <lgl>        
## 1 AIX EN PROVENCE T… Aix En Provence T… "Aix\nEn\nProvence\n… NA           
## 2 ANGERS SAINT LAUD  Angers Saint Laud  "Angers\nSaint\nLaud" NA           
## 3 ANGOULEME          Angouleme          Angouleme             NA           
## 4 ANNECY             Annecy             Annecy                NA           
## 5 ARRAS              Arras              Arras                 TRUE         
## 6 AVIGNON TGV        Avignon Tgv        "Avignon\nTgv"        NA           
## # … with 53 more rows
## #
## # Edge Data: 130 x 3
##    from    to journey_time
##   <int> <int>        <dbl>
## 1     1    39        186. 
## 2     2    40         97.5
## 3     3    40        146. 
## # … with 127 more rows

While it was morphed, only the few nodes that make up the connections between the Arras and Nancy stations were selected. A simple mutate() adds a new variable called selected_node, which tags those nodes with TRUE. The new variable and value is retained once the rest of the nodes are restored via the unmorph() command.

To keep the change, the shortest variable is updated with the changes made to both edges and nodes.

shortest <- shortest %>%
  mutate(selected_node = TRUE) %>%
  activate(edges) %>%
  mutate(selected_edge = TRUE) %>%
  unmorph()

The next step is to coerce each NA into a 1, and the shortest route into a 2. This will allow us to easily re-arrange the order that the edges are drawn in the plot, ensuring that the route will be drawn at the top.

shortest <- shortest %>%
  activate(nodes) %>%
  mutate(selected_node = ifelse(is.na(selected_node), 1, 2)) %>%
  activate(edges) %>%
  mutate(selected_edge = ifelse(is.na(selected_edge), 1, 2)) %>%
  arrange(selected_edge)

shortest

## # A tbl_graph: 59 nodes and 130 edges
## #
## # A directed simple graph with 1 component
## #
## # Edge Data: 130 x 4 (active)
##    from    to journey_time selected_edge
##   <int> <int>        <dbl>         <dbl>
## 1     1    39        186.              1
## 2     2    40         97.5             1
## 3     3    40        146.              1
## 4     4    39        225.              1
## 5     6    39        161.              1
## 6     7    39        358.              1
## # … with 124 more rows
## #
## # Node Data: 59 x 4
##   name               title              label                 selected_node
##   <chr>              <chr>              <chr>                         <dbl>
## 1 AIX EN PROVENCE T… Aix En Provence T… "Aix\nEn\nProvence\n…             1
## 2 ANGERS SAINT LAUD  Angers Saint Laud  "Angers\nSaint\nLaud"             1
## 3 ANGOULEME          Angouleme          Angouleme                         1
## # … with 56 more rows

A simple way to plot the route is to use the selected_ variables to modify the alpha. This will highlight the shortest path, without completely removing the other stations. This is a personal design choice, so experimenting with different ways of highlighting the results is always recommended.

shortest %>%
  ggraph(layout = "kk") +
    geom_edge_diagonal(aes(alpha = selected_edge), color = "gray") +
    geom_node_text(aes(label = label, color =name, alpha = selected_node ), size = 3)

The selected_ fields can also be used in other dplyr functions to analyze the results. For example, to know the aggregate information about the trip, selected_edge is used to filter the edges, and then the totals can be calculated. There is no summarise() function for graph tables; this make sense because the graph table would become a summarized table with such a function. Since the end result we seek is a total rather than another graph table, a simple as_tibble() command will coerce the edges, which will then allows us to finish the calculation.

shortest %>%
  activate(edges) %>%
  filter(selected_edge == 2) %>%
  as_tibble() %>%
  summarise(
    total_stops = n() - 1,
    total_time = round(sum(journey_time) / 60)
    )

## # A tibble: 1 x 2
##   total_stops total_time
##         <dbl>      <dbl>
## 1           8         23

Re-using the code

To compile most of the code in a single chunk, here is an example of how to re-run the shortest path for a different set of stations: the Laval and Montpellier stations.

from <- which(stations == "Montpellier")
to <-  which(stations == "Laval")

shortest <- graph_routes %>%
  morph(to_shortest_path, from, to, weights = journey_time) %>%
  mutate(selected_node = TRUE) %>%
  activate(edges) %>%
  mutate(selected_edge = TRUE) %>%
  unmorph() %>%
  activate(nodes) %>%
  mutate(selected_node = ifelse(is.na(selected_node), 1, 2)) %>%
  activate(edges) %>%
  mutate(selected_edge = ifelse(is.na(selected_edge), 1, 2)) %>%
  arrange(selected_edge)

shortest %>%
  ggraph(layout = "kk") +
    geom_edge_diagonal(aes(alpha = selected_edge), color = "gray") +
    geom_node_text(aes(label = label, color =name, alpha = selected_node ), size = 3)

Additional, the same code can be recycled to obtain the trip summarized data.

shortest %>%
  activate(edges) %>%
  filter(selected_edge == 2) %>%
  as_tibble() %>%
  summarise(
    total_stops = n() - 1,
    total_time = round(sum(journey_time) / 60)
    )

## # A tibble: 1 x 2
##   total_stops total_time
##         <dbl>      <dbl>
## 1           3         10

Shiny app

To see how to use this kind of analysis inside Shiny, please refer to this application. It lets the user select two stations, and it returns the route, plus the summarized data. The source code is embedded in the app.

End-to-end visualization using ggplot2

Mon, 14 Aug 2017 00:00:00 +0000

ggplot2 is kind of a household word for R users. I’ve ended up using it for complex data munging and wrangling work, where I needed to get clarity on different aspects of the data, especially being able to get different views, slices and dices of it, but in a nice visualization. At some point along the line, I slowly stopped using more traditional plotting functions like plot(), matplot(), barplot(), etc.

This article is an end-to-end data visualization exercise, using only ggplot2(). It has been helpful for me to see such pieces online on the endless possibilities of ggplot2(), so I wanted to give back to the community by doing one of my own.

1. Pima Indian Diabetes data

Consider the Pima Indian Diabetes dataset available in R. It looks at the population of women who were at least 21 years of age, of Pima Indian heritage and living near Phoenix, Arizona, and were tested for diabetes according to WHO criteria. In this exercise, I will use the 332 test data subjects. There are no missing values in this data. It is a very simple dataset, but my goal is to use it to demonstrate the tools available in ggplot2 to visually investigate a dataset we know very little about. This is part of the important data exploration phase of a data science project, to help prepare for the modeling phase.

library(MASS)
d <- Pima.te
summary(d)

##      npreg             glu              bp              skin      
##  Min.   : 0.000   Min.   : 65.0   Min.   : 24.00   Min.   : 7.00  
##  1st Qu.: 1.000   1st Qu.: 96.0   1st Qu.: 64.00   1st Qu.:22.00  
##  Median : 2.000   Median :112.0   Median : 72.00   Median :29.00  
##  Mean   : 3.485   Mean   :119.3   Mean   : 71.65   Mean   :29.16  
##  3rd Qu.: 5.000   3rd Qu.:136.2   3rd Qu.: 80.00   3rd Qu.:36.00  
##  Max.   :17.000   Max.   :197.0   Max.   :110.00   Max.   :63.00  
##       bmi             ped              age         type    
##  Min.   :19.40   Min.   :0.0850   Min.   :21.00   No :223  
##  1st Qu.:28.18   1st Qu.:0.2660   1st Qu.:23.00   Yes:109  
##  Median :32.90   Median :0.4400   Median :27.00            
##  Mean   :33.24   Mean   :0.5284   Mean   :31.32            
##  3rd Qu.:37.20   3rd Qu.:0.6793   3rd Qu.:37.00            
##  Max.   :67.10   Max.   :2.4200   Max.   :81.00

head(d)

##   npreg glu bp skin  bmi   ped age type
## 1     6 148 72   35 33.6 0.627  50  Yes
## 2     1  85 66   29 26.6 0.351  31   No
## 3     1  89 66   23 28.1 0.167  21   No
## 4     3  78 50   32 31.0 0.248  26  Yes
## 5     2 197 70   45 30.5 0.158  53  Yes
## 6     5 166 72   19 25.8 0.587  51  Yes

The target variable, type, tells us whether a patient is diabetic or not.

2. Distributions across categories

When the target is categorical, as in this case, type, I like to start by examining distributions for the continuous input columns. This gives us an overall sense of which input is likely to be useful. To do this, I like to do both boxplots and a density plot, since each has a different goal.

2.1. Boxplots

First, I’ll use boxplots, but ggplot2-style. I really like the look of a ggplot2() boxplot. It also allows me to seamlessly have multiple plots in a grid, as well as tinker around with the plotting parameters more flexibly than in a classical boxplot() approach, and end up with a nice-looking plot. We can see below how some inputs clearly vary across the 2 target categories, and others don’t.

df <- subset(d, select=c(glu, bp, skin, bmi, ped, age, type))  

library(gridExtra)
library(ggplot2)
p <- list()

for (j in colnames(df)[1:6]) {
  p[[j]] <- ggplot(data=df, aes_string(x="type", y=j)) + # Specify dataset, input or grouping col name and Y
            geom_boxplot(aes(fill=factor(type))) + guides(fill=FALSE) + # Boxplot by which factor + color guide
            theme(axis.title.y = element_text(face="bold", size=14))  # Make the Y-axis labels bigger/bolder
}

do.call(grid.arrange, c(p, ncol=3))

2.2. Density plots

I have used various overlay-density packages in the past, sm.density.compare() for example. I find the overlay-density rendering in ggplot2() to be more visually pleasing, with little plotting parameter tuning. E.g., it’s clear in the plot below that diabetic patients are associated with more number of pregnancies. I really like the alpha parameter.

df$npreg <- d$npreg
g <- ggplot(df, aes(npreg))
g + geom_density(aes(fill=factor(type)), alpha=0.8) + 
    labs(title="Density plot", 
         subtitle="# Pregnancies Grouped by Diabetes Type",
         x="# Pregnancies",
         fill="Diabetes Type")

3. Grid views

Next, I want to mix things up a little, so that I can have multi-dimensional views. By this, I mean that I want to know how the target is distributed across a few important inputs, but I want to link those inputs up as well. Sort of like a 3-way table, but visualized nicely instead of numbers. I came across this problem recently in one of the projects, and while it seems like a basic must-have output to dig deeper, I really needed something like ggplot2 to implement it. Using facet_grid() was amazing, even more so on account of the smooth control one has on the plotting parameters within a ggplot2 setup.

3.1. Data preparation

Facet-wrapping and gridding is a must-have tool for deeper data views, but the process is a multi-step one. Not too complicated though - very intuitive under ggplot2. We start with creating some new categorical columns using the continuous ones. Note that this can be done in different ways: appending new columns directly to the data frame, or using the more sleeker dplyr() in combination with magrittr(), which I absolutely love. This integrates a number of operations into a single chunk, making it quite seamless. I am also loading up plyr(), since I will be using it later.

library(magrittr)
library(plyr)
library(dplyr)
df_grid <- d %>% 
          mutate(Skin = ifelse(d$skin <= 29, "low skin fold", "high skin fold"),
                  BMI = ifelse(d$bmi <= 33, "low BMI", "high BMI"),
                  Ped = ifelse(d$ped <= 0.31, "low pedigree",
                ifelse(d$ped > 0.3134 & d$ped <= 0.5844, "medium pedigree", "high pedigree"))) %>% 
  
            mutate(Ped = factor(Ped, levels = c("low pedigree", "medium pedigree", "high pedigree")))

3.2. Reshaping the data

Next, we need to prepare the data a little more before throwing it into the facet_grid() mix. Most importantly, we need to “reshape” it, i.e., while our data is a “wide”-form data frame, we need to convert this to a “long”-form to enable facet_grid() to easily pick up what it needs to “facet” the plot by. We will also add a “size” column - this will allow us to make more granular adjustments in our plot. I will also rename columns in order to enable easier axis labeling when plotting. Again, notice that instead of using reshape2(), which I have used for many years, we’re using gather() from tidyr(), all sewn together with the pipe in magrittr().

library(tidyr)
DF <- df_grid %>% 
    subset(select=c(type, Skin, BMI, Ped)) %>% 
    gather(variable, value, -c(Skin, Ped, BMI))

colnames(DF)[5] <- "Diabetes_Value"
DF$size <- rep(1.5, nrow(DF))
s <- 1.5

3.3. Facet Grid

We’ll try the basic facet_grid() plot, after which we’ll go in and make some adjustments. For now, our goal is the following: to see a “matrix” or “grid” of the BMI distribution across diabetes type, as a 2x2 table of pedigree/skin fold combinations. In other words, for low pedigree/low skin fold, how does BMI distribute across diabetes type? You can see the amount of information you can pack into just one plot. I have found this to be useful when presenting to an end-user or customer. It becomes all the more useful since its a very clear representation of this slice/dice, with little room for ambiguity.

# Simple
library(ggplot2)
ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  facet_grid(Skin ~ Ped)   # wrap up everything to showcase by multiple cols

This looks nice, but I would like to add more of a “pop”. I am going to outline each box, and bolden the fonts. Note that you can also color the “grid strips”, but I won’t do that right now.

# More color
p <- ggplot(data=DF, aes(x=Diabetes_Value, fill=BMI)) + geom_bar() +  # Barplot
  geom_rect(aes(fill=NA, size=size),xmin =-Inf,xmax=Inf,ymin=-Inf,ymax=Inf,alpha = 0.0002, colour="black",show.legend = F) +   # use box drawn around each location to cleanly separate facets + suppress guide
  scale_size(range=c(s,s), guide=FALSE) + # use line width/size feature for cleaner plotting
  facet_grid(Skin ~ Ped) +   # wrap up everything to showcase by multiple cols
  theme(strip.text.x = element_text(face="bold", size=12)) +
  theme(strip.text.y = element_text(face="bold", size=12)) 
  # optional changes in strip
#+ theme(strip.text.x = element_text(face="bold", size=12, colour="white")) +
#  theme(strip.text.y = element_text(face="bold", size=12, color="white")) +
#  theme(strip.background = element_rect(fill="black"))
plot(p)

Much better. Look how nicely this granular plot adjustment in ggplot2 allows each “block” in the matrix to pop out. Its very clear how BMI is distributed across diabetes type, and how that in turn is distributed across both pedigree function and skin fold. We see that (as expected): 1. A higher triceps skin fold thickness is associated with a higher BMI, as well as a higher count of diabetic people. 2. The above is more true for a higher diabetes pedigree function.

This kind of a grid plot presents a very powerful tool for such multi-dimensional data views.

4. Heatmaps

I like heatmaps - there’s a sense of drama in the way you can see where “something is happening”. I’ve used heatmap.2() to implement hierarchical clustering and translating that to a heatmap. But I wanted to use ggplot2() to simply look at a dataset as a heatmap, without any underlying analysis, to detect patterns before any analysis begins.

In this case, I want ggplot2() to show me patterns across different input columns, for the two diabetes types, i.e., what inputs seem to differ across diabetic/non-diabetic patients. This will be clear once we render our dataset into a nice ggplot2() heatmap.

4.1. Data preparation

As usual, we need to prep our data before pushing it into the ggplot2() function. We’ll reshape and scale the data first, all within the plyr(), dplyr(), and magrittr() framework. I’ll also specify some plotting parameters that I will call into my ggplot2() function. I’m going to rely on RColorBrewer() for these.

df_heat <- d[order(d$type),1:8]
DF_Heat <- df_heat %>%
          mutate(id = 1:nrow(df_heat)) %>%
          select(c(npreg:age, id))  %>%
          gather(variable, value, -id)  %>%
          ddply(.(variable), transform,
                    rescale = scale(value))  # Notice that this reorders by "variables"
          
# Color scale for heatmap
library(RColorBrewer)
colors <- brewer.pal(9, 'Reds')

# Lines to split patients into diabetic/non-diabetic
my.lines <- data.frame(x1 = 0.5, x2 = 7.5, y1 = 223.5, y2 = 223.5)

4.2. Rendering the heatmap

# Basic plot
p <- ggplot(DF_Heat, aes(as.factor(variable), as.factor(id), group=id)) + 
  geom_tile(aes(fill = rescale),colour = "white") +
  scale_fill_gradient(low="green", high="red")

# Make adjustments
base_size <- 9
p_adj <- p + theme_grey(base_size = base_size) + labs(x = "",y = "") + scale_x_discrete(expand = c(0, 0)) +
  scale_y_discrete(expand = c(0, 0)) +
  geom_segment(data=my.lines, aes(x = x1, y = y1, xend=x2, yend=y2), size=1, inherit.aes=F) +
  theme(axis.text.y = element_blank(), axis.ticks.y = element_blank()) 
plot(p_adj)

Note that I have suppressed the ticks on the Y-axis. We can clearly see regions of interest on the heatmap. It would be better for these to easily pop out at the viewer, to enable which, I am going to invoke geom_rect().

# Borders of rectangles to indicate areas of interest on heatmap
my.lines.rect.1 <- data.frame(xmin = 1.5, xmax = 2.5, ymin = 223.5, ymax = 255.5)
my.lines.rect.2 <- data.frame(xmin = 3.5, xmax = 4.5, ymin = 223.5, ymax = 332)
my.lines.rect.3 <- data.frame(xmin = 5.5, xmax = 6.5, ymin = 223.5, ymax = 280.5)

p_adj + geom_rect(data=my.lines.rect.1, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=2, inherit.aes = F) +
  geom_rect(data=my.lines.rect.2, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=5, inherit.aes = F) +
  geom_rect(data=my.lines.rect.3, aes(xmin = xmin, xmax = xmax, 
            ymin = ymin, ymax = ymax), fill = NA, col = "black", lty=4, inherit.aes = F)

Much better.

5. Segmentation in a scatterplot

Finally, I want to try to implement some “basic-level clustering”. This is not model-based clustering; rather, it is simply using a scatterplot and a few nice plotting parameters in ggplot2() to make some things pop right out at the viewer - again, with little room for ambiguity. What I like most here is the boxes that we can draw nicely to showcase the “clusters” a little better, along-with the multi-layered information, e.g., age, BMI, glucose, etc.

The conclusions are logical and obvious from the following plot, but quite nicely illustrate the use of ggplot2() for such a specific purpose.

d$Age <- ifelse(d$age < 30, "<30 yrs", ">= 30 yrs")

ggplot(d, aes(x = glu, y = bmi)) +
  geom_rect(aes(linetype = "High BMI - Diabetic"), xmin = 160, ymax = 40, fill = NA, xmax = 200, 
            ymin = 25, col = "black") + 
  geom_rect(aes(linetype = "Low BMI - Not Diabetic"), xmin = 0, ymax = 25, fill = NA, xmax = 120, 
            ymin = 10, col = "black") + 
  geom_point(aes(col = factor(type), shape = factor(Age)), size = 3) +
  scale_color_brewer(name = "Type", palette = "Set1") +
  scale_shape(name = "Age") +
  scale_linetype_manual(values = c("High BMI - Diabetic" = "dotted", "Low BMI - Not Diabetic" = "dashed"),
                        name = "Segment")

Hopefully, this little exercise will be helpful for someone wanting to use ggplot2() for an innovative slice/dice of a complex dataset, and to visualize it nicely.

What is the tidyverse?

Thu, 08 Jun 2017 00:00:00 +0000

Last week, I had the opportunity to talk to a group of Master’s level Statistics and Business Analytics students at Cal State East Bay about R and Data Science. Many in my audience were adult students coming back to school with job experience writing code in Java, Python and SAS. It was a pretty sophisticated crowd, but not surprisingly, their R skills were stitched together in a way that left some big gaps. Many for example, didn’t fully understand the importance of CRAN Task Views as curated source for the best packages to support their work in machine learning, time series and the other areas of Statistics they were studying. So, it made sense that even though ggplot2 and dplyr were mentioned in some of the student’s questions, a faculty member present asked: “What is the tidyverse?” in an attempt to cover an area that he knew was one of those gaps.

There is an incredible amount of good material available online about the tidyverse, and I will point to some of that below. But here, I’ll elaborate on the answer I gave during the Q&A.

The Basics

The tidyverse is a coherent system of packages for data manipulation, exploration and visualization that share a common design philosophy. These were mostly developed by Hadley Wickham himself, but they are now being expanded by several contributors. Tidyverse packages are intended to make statisticians and data scientists more productive by guiding them through workflows that facilitate communication, and result in reproducible work products. Fundamentally, the tidyverse is about the connections between the tools that make the workflow possible.

It is also the case that the tidyverse is work in progress. You can find the current state of development at tidyverse.org. Clicking on the icon for each package on this website will bring you to detailed documentation for each package. The following figure illustrates a canonical data science workflow, and shows how the individual packages fit in.

If you have some experience with R, you ought to be able to jump right into the online documentation and find your way around. If you are new to R, and maybe new to data science as well, you can’t do any better than work through the book R for Data Science by Hadley Wickham and Garrett Grolemund.

Advantages

The advantages of the tidyverse include consistent functions, workflow coverage, a path to data science education, a parsimonious approach to the development of data science tools, and the possibility of greater productivity.

Consistency

The tidyverse aspires to consistency on multiple levels. Examples of “micro”-level consistency include the convention of having variable names glide along in snake_case, and the signatures of tidyverse functions follow a regular pattern. (The first formal argument is always a data frame that provides the function’s input.) Higher-level consistency includes the idea of tidy data - a data frame where each row is an observation and each column contains the value of a single variable - and the way in which the pipe operator, %>%, channels the flow of tidy operations. Under the covers, there are even more levels of structure that aid the pursuit of consistency, including uniform standards for package organization, testing procedures, coding style, etc.

Coverage

The workflow shown above, with tidyverse packages associated with the various steps, or more usually rendered with the following iconic tidyverse diagram, preceded and motivated the development of the tidyverse.

It is an abstraction of the canonical data analysis workflow that has always guided statisticians, but now informs data science as a map to organize, streamline, automate and optimize the various processes involved. The fact that tidyverse packages are associate with all of the processes indicates that it comprises enough fundamental building blocks to support the entire end-to-end workflow for a variety of data sources and analysis goals. Moreover, the relatively recent addition of the purrr package extends the reach of the tidyverse to support the creation of new data science tools.

Critical Mass

A great strength of the R language is that with over ten thousand user contributed packages on CRAN alone, it has a lot to offer. This kind of organic growth makes it inevitable that packages will offer overlapping features. Users have to make decisions about which package, or suite of packages, they will make the effort to learn. For many users, the decision hinges on whether a collection of packages visibly supports important work. Does it have a large community of users and is it backed by committed developers and maintainers? All of the signals indicate that (at least, among R-using data scientists) the tidyverse has reached critical mass. For example, the tidyverse package has been downloaded 50,000 times in the last month. Moreover, it appears that tidyverse principles are propagating into other application areas. The tidyquant package, for example, is a serious attempt to bring tidy principles to Finance.

Education

A typical R user gets involved with R in the first place through a desire to compute in some quantitative field. The path to R competency frequently begins with mastering a small number of relevant functions. Statisticians, for example, may learn to read in data from a .csv file and build a linear regression model with lm(). Financial analysts may be introduced to R through a package like quantmod, which enables a new user to do quite a bit of real work. The tidyverse provides the path of least resistance, or “pit of success”, for data scientists interested in R. For example, the small number of compatible building blocks provided by dplyr enable even a relatively inexperienced user to tidy up a messy data set quickly and easily.

Parsimony

The packages and functions of the tidyverse are the result of trial-and-error experimentation carried out over several years, to find a minimum set of functions that are sufficient to enable the canonical data science workflow. Those of you who have been following Hadley’s work will remember cast() and melt() from the reshape and reshape2 packages, and ddply() from the plyr package, which were early attempts to find a vocabulary for wrangling data frames. After several attempts to identify and construct the most advantages set of primitive building blocks, the tidyverse has matured into its present form.

Productivity

Hadley has always been clear that a major goal for the tidyverse - and indeed much of his work over the years - has been to help anyone who needs to analyze data work productively, and he is fond of quoting Hal Abelson: “Programs must be written for people to read and only incidentally for machines to execute”. My take is that a major reason for the popularity of tidyverse packages is that they help people achieve and maintain flow in their daily data analysis work.

Some Limitations

The tidyverse, of course, is not without limitations. Some of these are due to factors that are beyond the designer’s control, and others may be by design. Limitations of the first kind may arise from a lack of agreement as to whether some data can be, or should be, forced into a “rectangular” data structure. For example, although there are scientists and data scientists working in genomics that are fans of dplyr and ggplot2, much of the work done in the Bioconductor Project remains outside of the tidyverse workflow.

The need for the close coordination of tidyverse packages produces some limitations of the second sort. There are many high-quality R packages that are of great use to data scientists, but based on design goals that differ from those of the tidyverse. There will always be more than the tidyverse.

A Bigger Picture

A powerful, but perhaps under-appreciated, capability of the R language is its ability to support the design and programming of Domain Specific Languages. Joe Cheng highlighted this feature in an interview he gave to R Views last year. He described R as being “shockingly close to LISP”, of which Joe says: “it’s almost like you change the language itself to be a DSL for whatever problem you’re trying to solve … the elegant, terse syntax of dplyr and the pipe operator are possible because of how malleable a language R is, and how great it is for writing DSLs in it.”

So, from a wider perspective, the tidyverse can be seen as sub-dialect of the R language that is evolving to express ideas and tasks inherent in Data Science workflows and software development. This dialect may not be for everyone, but it does seem to be helping many R fluent data scientists frame their conversations.

Some Resources

The following are some resources that you may find helpful in learning and mastering the tidyverse.

The video of Hadley Wickham’s Keynote address at rstudio::conf 2017
The slides corresponding to the above video
R for Data Science by Hadley Wickham and Garrett Grolemund
Text Mining with R
Getting Started with the Tidyverse in R