Visualizations on R Views

COVID-19 Data Forum: Data Journalism

Tue, 06 Apr 2021 00:00:00 +0000

The COVID-19 Data Forum, a joint project of the Stanford Data Science Institute and the R Consortium, is an ongoing series of multidisciplinary webinars where topic experts discuss data-related aspects of the scientific response to the pandemic. The most recent event, held on March 18, 2021, explored the role of data journalism in the pandemic. This was a bit of a departure from previous forum events¹ because it focused on issues relating to using and interpreting COVID-19 data, and not on the particular kinds of COVID-19 related data that are available.

I think you will find the webinar video worth watching. If you are a statistician or epidemiologist working on COVID-19, you may find the data journalists’ accounts of difficulties they faced working with COVID data and statistical models instructive. But, even if you are not directly working on COVID, you may find that listening to the journalists fills in some gaps between what you know about statistics and data visualizations and what you see in the news.

The data journalism event was moderated by Dr. Irena Hwang, a data reporter at ProPublica. Speakers included Dr. Mark Hansen, David and Helen Gurley Brown Professor of Journalism and Innovation at Columbia University; Ana Carolina Moreno, a senior data journalist at TV Globo in São Paulo, Brazil; and Meghan Hoyer, Director of Data Reporting at the Washington Post.

The video of the data journalism event is available here. The following short time map and the times referenced in my comments below should be helpful for browsing the ninety minute event.

2:37 Irena Hwang introduces Mark Hansen
3:50 Start of Mark’s talk
19:30 Irena introduces Ana Carolina Moreno (Carol)
21:10 Start of Carol’s talk
39:20 Irena introduces Meghan Hoyer
40:00 Start of Meghan’s talk
1:01:40 Start of discussion

Mark Hansen

In his talk, Mark offers an overview of the profession of data journalism that provides some historical context and emphasizes the hybrid nature of the practice which blends a hard nose detective’s drive to uncover facts with the empathy to tell stories “about who we are and how we live”.

7:00 Mark introduces Joseph Pulitzer’s 1904 paper The College of Journalism in which Pulitzer includes Statistics as a subject journalists should study. On page 673, Pulitzer writes:

You want statistics to tell you the truth. You can find truth there if you know how to get at it, and romance, human interest, humor and fascinating revelations as well.

10:19 Mark describes a piece, An ode to reporter’s notebooks, published by Philip Eil in the Columbia Journalism Review that offers a personal account of reporting: Eil writes:

To report is to be alert and alive at a particular time and place.

11:00 Mark remarks:

when we’re thinking about bringing computation to journalism we are taking that basic curiosity that we are cultivating in our students minds … and adding computational lines of inquiry to that habit of mind, that questioning why things look the way they do…

12:08 Mark calls attention to the report by Charles Berret and Cheryl Phillips Teaching Data And Computational Journalism and describes some recent activities of the Brown Institute at the Columbia School of Journalism.

Ana Carolina Moreno

22:32 Carol introduces Brazil’s universal healthcare system and shows a schematic of the available official and unofficial COVID-19 data sources.

26:00 Carol notes that a platform originally built to track SARS data was adapted to track COVID.

27:38 Carol explains that, in practice, there are many obstacles making it difficult to obtain the data necessary to understand how the pandemic is developing. Some of these are called out in the following slide:

30:37 Carol remarks that hospital data seems to be the most reliable.

31:06 Carol describes how the government changed its policy for reporting deaths. The new scheme of only reporting deaths that have been confirmed in the past twenty-four hours vastly undercounts the current death rate.

31:57 In an effort to obtain more reliable data, a consortium of competing journalists at local news organizations began cooperating by sharing information directly obtained from hospitals every day.

32:35 Carol provides a view of day-to-day journalism at the local news organizations and describes how the data journalist scrape data on a daily basis to populate dashboards showing rolling averages and daily indicators. By focusing on the more reliable hospitalization data journalists are doing their best to track the spread of the pandemic an expose inequities in the health care system.

Meghan Hoyer

40:07 Meghan begins her walk through of what last year was like for data journalists who were trying to tell the story of the pandemic in real time as it was happening.

41:22 Meghan recounts her experiences trying to make sense of COVID-19 models and expresses the frustration she and other data journalists felt with the multitude of contradictory predictive models.

44:03 In a memorable quote, Meghan remarks:

Models were inherently problematic and yet they were being forced upon us by society…

Consequently journalists at the AP agreed and decided that they were not going to base stories on models.

In absence of reliable case data, and wanting nothing to do with the models, Meghan explains that data journalists turned to whatever data they could get their hands on to quantify the story of the pandemic.

46:00 Meghan recounts how journalists used garbage pickup data as a proxy for population density to estimate where people were living in NYC and correlate it with case data.

47:30 Journalists struggled to find data to verify the anecdotal stories they were hearing about the the disparities in who was being affected by virus. Finding that one quarter to one third of the COVID case data was missing information on race, data journalists “hand collected” data by looking city by city to find the missing data.

50:30 Meghan recounts how they turned to age adjusted data to determine the impact of the virus on communities of color.

52:10 Data journalists find that excess deaths is a reliable metric for determining the impact of what is happening on the ground.

54:18 Journalists developed a survey which was returned by seven hundred schools to investigate how going back to school might be affecting students. Among their findings was that districts serving students of color were more likely to start online.

56:35 Meghan discusses the COVID-19 Tracking Project and the effort to sort out the impact of test positivity rates. She reports that because not all states measure the number of people who test in the same way, correctly comparing test positivity rates among states remains an unsolved problem.

58:33 Meghan shares the need to “flip the numbers” to help people understand the meaning of statistics stated in terms of very large numbers. For example, saying that “Since January of last year at least 1 in 15 people who live in Alexandria, Virginia have been infected by the virus” is easier for people to understand than something like: “On March 17th there were 14 cases per 100,000 in Alexandria”.

59:53 Vaccination tracking is another problematic data reporting area. Not only are vaccinations reported differently from state-to-state, but the data that is reported is changing from day-to-day. The CDC is apparently still adding new fields to the vaccination data sets.

The Q & A Discussion

1:02 The question and answer discussion begins.

1:02:56 Mark talks about how visualizations evolved over the course of the pandemic.

1:06:08 Carol and then Meghan talk how the lessons the pandemic taught data journalists about competition and collaboration.

1:10:04 Meghan describes how during the pandemic data journalists became advocates for public data.

1:11:21 Carol answers a question about the opportunities for data journalism in Brazil.

1:15:50 Answers a question of how academia is supporting data journalism during the pandemic and mentions an effort to have statistical and scientific experts collaborate with data journalists.

1:20:19 Meghan responds to a question about technical and social challenges for data journalists during the pandemic.

1:23:10 Carol talks about the difference between reporting online news and television news.

1:26:01 Mark answers a question about communicating emotional impact in COVID reporting and ends with emphasizing the importance of communicating honestly about what we do, and do not know.

¹The first forum on May 14, 2020 focused on the data needs and challenges of modeling and controlling the spread of COVID-19, The second forum on August 13, 2020 explored what was being done to make clinical data available and useful. The third forum on December 10, 2020 discussed the role of mobility data.

Wonderful Wednesdays

Mon, 11 Jan 2021 00:00:00 +0000

For almost a year now, the PSI Visualization Special Interest Group (VIS SIG) has been conducting a monthly graduate-level seminar on creating effective statistical visualizations that is open to everyone. Wonderful Wednesdays is a unique collegial event. Every month the SIG publishes a link to a new data set and issues a challenge to produce visualizations that effectively illustrate some specific aspects of the data. Anyone can submit an entry coded in the language of their choice. Submissions that are received by the deadline are then critiqued in free webinar that takes place roughly thirty days later. You don’t have to make a submission to attend the webinar.

The process is well organized and straightforward. The figure below illustrates the process and time liness for the for the webinar that will happen this week on Wednesday, January 13th.

Click here to register for the webinar.

Here is the link to the December webinar where the challenge was to visualize the heterogeneity among the data used for a meta-analysis of seven studies undertaken to show a reduction in hypertension.

The following image is a traditional forest plot showing a comparison of the odds ratios for the seven studies. (If you are not familiar with this plot type look here for some tips on how to interpret it.)

This image comes from a shiny app that is critiqued at the by the VIS-SIG statisticians towards the beginning of the webinar. The experts liked the clean look and labeling on the plot, but had mixed feelings about the colored bands which are meant to show regions where the studies overlap. (Darker color indicates more overlap.) Here are the links to the Shiny App and the code and also to the blog post that reviews all of the submissions for the December challenge.

Visit the VIS-SIG Blog to finds posts and code for the submissions for all of Wonderful Wednesday events so far.

Accelerate your plots with ggforce

Thu, 19 Sep 2019 00:00:00 +0000

In this post, I will walk you through some examples that show off the major features of the ggforce package. The main goal is to share a few ideas about customizing visualizations that you may find useful in your everyday work.

The ggforce package is an extension to ggplot2 developed by Thomas Pedersen. Thanks to ggforce, you can enhance almost any ggplot by highlighting data groupings, and focusing attention on interesting features of the plot. The package contains geoms, stats, facets, and other ggplot functions. Among such functions, there are some for marking the convex hull of a set of points, jittering data, and creating Voronoi plots.

Base `ggplot`

The examples in this article will use data from the nycflights13 package. Most of the examples will build on the same basic ggplot that visualizes airports by geographical location. I am using this data set because it makes it easy to plot x/y coordinates without having to remember what they “mean”. This basic plot will be saved to a variable, and then that variable will be used as the base of the examples of enhancing the visualization using ggforce

library(tidyverse)
library(ggforce)
library(nycflights13)

p <- airports %>%
  filter(lon < 0, tzone != "\\N") %>%
  ggplot(aes(lon, lat, color = tzone)) + 
  geom_point(show.legend = FALSE)  

p

Make your `mark` with `ggforce`

I have long been waiting for an easy way to draw an outline around groups of data. The geom_mark_...() family of functions does exactly that. There are four mark functions in ggforce, all different based on the shape they draw around the group:

geom_mark_circle()
geom_mark_ellipse()
geom_mark_hull()
geom_mark_rect()

Let’s start with geom_mark_rect(); it will draw a rounded rectangle around each time zone group.

p +
  geom_mark_rect()

Like magic! The rectangles look amazing, even without modifying any arguments. Of course, more customization is possible via setting arguments. In this post, I will review some of the many great arguments available in ggforce functions, but I don’t want to rob you of the fun of trying it yourself and discovering all of the different options.

Label, and an arrow!

This next addition to our plot deserves its own subheading. Adding a label and an arrow pointing to a group would typically be a major undertaking. Without ggforce, this would require manually adding both the text and the arrow to the ggplot. But, with geom_mark it is a simple as setting the label argument. So, without further ado, here is the label argument in action:

p + 
  geom_mark_rect(aes(label = tzone))

The labels and arrows are not only drawn, but they are also placed in an optimized location. In addition, the position will recalculate if the plot is re-sized! There are too many little details about this label argument to mention. The backdrop is automatically white, the indicator is not really an arrow, it is a simple line that also underlines the text, so it is easy for the eye to know which group belongs to which label.

It is now easy to finalize the plot by resetting the theme, and again suppressing the legend using show.legend.

p + 
  geom_mark_rect(aes(label = tzone), show.legend = FALSE) +
  theme_void()

Hull-k, enhance!

There are many cases where drawing a rectangle or circle around the groups is not ideal, or even preferable. The geom_mark_hull() essentially traces a more complex polygon around the shape of the outline of the group.

p + 
  geom_mark_hull(aes(label = tzone)) +
  theme_void()

Again, without adding any arguments to the function, the traced outline already looks wonderful. Another option to add now is fill. And since the legend table is now redundant, it can be suppressed by setting show.legend to FALSE.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE) +
  theme_void()

Notice that the fill color is not totally opaque; by default, ggforce has set the translucency lower to make sure that the dots are visible. This is something that I would have done anyway, usually by adding the alpha argument. In this case, it saves having to remember to add that argument.

Another adjustment that I thought was important for this plot was to modify the size of the hull, to change the padding around the outline of the group. The expand argument controls this aesthetic; it is possible to change it using the units() command.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_void()

Axe `theme_void()`

To finalize plots such as this one, it is necessary to remove most components from the default theme. Usually, theme_void() does the trick. For printed or online articles with white backgrounds, which is essentially all of them, it is often hard to determine the margins of the plot. theme_no_axes() provides a great compromise by removing all but the one element.

p + 
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_no_axes()

Another facet of `ggforce`, and it’s magnify-cent

It is common to produce two plots, one to show the full picture, and another to magnify or focus on a specific area. With facet_zoom(), it is incredibly easy to show “macro” and “micro” in one plot by using the same xlim and ylim arguments to focus on an area of a plot.

p +
  facet_zoom(xlim = c(-155, -160.5), ylim = c(19, 22.3))

Skip the coordinates

Another cool feature of facet_zoom() is the ability to set the zoom region based on a row selection. To do this, simply pass an expression that you would use in a function such as filter() to the facet. So instead of using coordinates, I just tell the facet to zoom on anything that has a Pacific/Honolulu time zone.

p +
  facet_zoom(xy = tzone == "Pacific/Honolulu")

Putting it all together, with three lines of code

Using what has been covered so far, it is easy go from a very simple point plot to a sophisticated and nice-looking visualization with just three lines of code, thanks to ggforce.

p +
  geom_mark_hull(aes(label = tzone, fill = tzone), show.legend = FALSE, expand = unit(3, "mm")) +
  theme_no_axes() +
  facet_zoom(x = tzone == "America/Los_Angeles")

“What is a Voronoi?”

This section title is based on my first reaction when I heard the word “Voronoi”. I have since learned about it, and can see why the Voronoi Diagram can be useful for very specific use cases. The good news is that if you encounter one of those use cases, you know that it is easy to draw it up in ggplot using geom_voronoi_segment().

The idea behind a Voronoi diagram is to split the area of the plot into as many sections as there are points. Unlike a grid or heat map, Voronoi draws custom shapes for each point based on the proximity to other points. It returns a plot that looks like stained glass. This can be good to determine the closest point inside each area. For example, a retailer can use it to see the area their store locations cover, and can help them make decisions to optimize their location based on the size of each Voronoi shape.

The following example will focus on airports in Alaska. The ggplot will zoom into that state’s general location, and then trace a hull shape. The hull will provide a quasi-map overlay. The final step is to add the Voronoi diagram layer by calling the function: geom_voronoi_segment()

p +
  geom_mark_hull(aes(fill = tzone), expand = unit(3, "mm")) +
  coord_cartesian(xlim = c(-130, -180), ylim = c(50, 75))  +
  geom_voronoi_segment()

Parallel to alluvial

The geom_parallel... functions allow visualizing interactions between categorical variables. The implementation is generic enough to create Sankey or alluvial charts.

For this, I will use the Manufacturer and Engine data from the planes table inside nycflights13. In this case, some simple data preparation is needed first.

prep_planes <- planes %>%
  filter(year > 1998, year < 2005) %>%
  filter(engine != "Turbo-shaft") %>%
  select(manufacturer, engine) %>%
  head(500)

prep_planes

## # A tibble: 500 x 2
##    manufacturer     engine   
##    <chr>            <chr>    
##  1 EMBRAER          Turbo-fan
##  2 AIRBUS INDUSTRIE Turbo-fan
##  3 AIRBUS INDUSTRIE Turbo-fan
##  4 EMBRAER          Turbo-fan
##  5 AIRBUS INDUSTRIE Turbo-fan
##  6 AIRBUS INDUSTRIE Turbo-fan
##  7 AIRBUS INDUSTRIE Turbo-fan
##  8 AIRBUS INDUSTRIE Turbo-fan
##  9 AIRBUS INDUSTRIE Turbo-fan
## 10 EMBRAER          Turbo-fan
## # … with 490 more rows

Prep for plotting with one line

The gather_set_data() is a convenience function that, just like gather(), creates a single line for each combination of categorical variables. The table contains three new columns - id, x, and y - which contain the combinations that each new row represents, and the row ID number from the original table.

prep_planes %>%
  gather_set_data(1:2)

## # A tibble: 1,000 x 5
##    manufacturer     engine       id x            y               
##    <chr>            <chr>     <int> <chr>        <chr>           
##  1 EMBRAER          Turbo-fan     1 manufacturer EMBRAER         
##  2 AIRBUS INDUSTRIE Turbo-fan     2 manufacturer AIRBUS INDUSTRIE
##  3 AIRBUS INDUSTRIE Turbo-fan     3 manufacturer AIRBUS INDUSTRIE
##  4 EMBRAER          Turbo-fan     4 manufacturer EMBRAER         
##  5 AIRBUS INDUSTRIE Turbo-fan     5 manufacturer AIRBUS INDUSTRIE
##  6 AIRBUS INDUSTRIE Turbo-fan     6 manufacturer AIRBUS INDUSTRIE
##  7 AIRBUS INDUSTRIE Turbo-fan     7 manufacturer AIRBUS INDUSTRIE
##  8 AIRBUS INDUSTRIE Turbo-fan     8 manufacturer AIRBUS INDUSTRIE
##  9 AIRBUS INDUSTRIE Turbo-fan     9 manufacturer AIRBUS INDUSTRIE
## 10 EMBRAER          Turbo-fan    10 manufacturer EMBRAER         
## # … with 990 more rows

The ggplot is primed with x for x, and then new aesthetics: id, split and value. For id, we pass the id column, split takes y, and finally, value is fixed to 1. The value is used to express the amount of “thickness” to add to that particular relationship; using 1 means that all combinations are weighted the same. At this point, the only argument to pass geom_parallel_sets() will be the color fill; in this case we will use engine.

Plotting with parallel

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine))

The plot shows how a specific plane’s engine relates to each of the manufacturers. Next geom_parallel_sets_axes() provides a terminal box; the axis.width argument is the only one necessary to use at this stage, and we will set it to 0.1. The labels are added by using geom_parallel_sets_labels(), and they are automatically rotated.

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine)) +
  geom_parallel_sets_axes(axis.width = 0.1) +
  geom_parallel_sets_labels()

The following is done to finalize the plot:

geom_parallel_sets() - Hide the legend and lower the alpha
geom_parallel_sets_axes() - Change the fill color and font color
geom_parallel_sets_labels() - Remove the rotation of the label

prep_planes %>%
  gather_set_data(1:2) %>%
  ggplot(aes(x, id = id, split = y, value = 1))  +
  geom_parallel_sets(aes(fill = engine), show.legend = FALSE, alpha = 0.3) +
  geom_parallel_sets_axes(axis.width = 0.1, color = "lightgrey", fill = "white") +
  geom_parallel_sets_labels(angle = 0) +
  theme_no_axes()

Danger zone!

When visualizing the combination of a continuous and a categorical variable, it is common practice to resort to a bar or column plot. Cases that require representing this in a single circle shape usually involve modifying a polar bar in ggplot. But, this is much easier now with ggforce. I start with the total number of planes by engine planes:

planes %>%
  count(engine)

## # A tibble: 6 x 2
##   engine            n
##   <chr>         <int>
## 1 4 Cycle           2
## 2 Reciprocating    28
## 3 Turbo-fan      2750
## 4 Turbo-jet       535
## 5 Turbo-prop        2
## 6 Turbo-shaft       5

and then pipe those results into ggplot using geom_arc_bar() to create the circle-shaped plot. The new aesthetics employed here are: x0, y0, r0, r, amount, and explode. The x, y, and r aesthetics refer to the position and the radius of the circle. Since only one plot is needed, I fix x and y to 0. For radius, the r0 refers to the inside of the circle, and r to the outside. Setting r0 to 0.7 and r to 1 will create a sort of doughnut with a 0.3 thickness. Finally, I use “pie” as the stat.

planes %>%
  count(engine) %>%
  ggplot() +
  geom_arc_bar(aes(x0 = 0, y0 = 0, r0 = 0.7, r = 1, amount = n, fill = engine), alpha = 0.3, stat = "pie")

Another cool thing this geom does is to make it east to “break-away” one or several segments of the plot. The explode aesthetic controls that. To break away the “Turbo-jet” results, I create a new column called focus, setting it to 0.2 if it is part of that engine group, and to 0 if it is not, then finish up with theme_no_axes().

planes %>%
  count(engine) %>%
  mutate(focus = ifelse(engine == "Turbo-jet", 0.2, 0)) %>%
  ggplot() +
  geom_arc_bar(aes(x0 = 0, y0 = 0, r0 = 0.7, r = 1, amount = n, fill = engine, explode = focus), alpha = 0.3, stat = "pie") +
  theme_no_axes()

This section is titled “Danger Zone”, because hanging the r0 in geom_arc_bar() may change the look of the plot to one that has fallen out of favor. That plot type happens to be the same name of the stat that we are using.

Closing remarks

ggforce is a great package that does a lot more than what I covered today. My hope is to have shared one or two things that will encourage you to try ggforce in your everyday work.

Special thanks to Thomas Pedersen, the author of the package and a co-worker of mine. His contributions to the R community also include the tidygraph and ggraph packages, which I wrote about in this blog post a few months back.