Quandl on R Views

Mapping Quandl Data with Shiny

Wed, 14 Jun 2017 00:00:00 +0000

Today, we are going to wrap our previously built Quandl/world map Notebook into an interactive Shiny app that lets users choose both a country and a data set for display. As usual, we did a lot of the heavy lifting in the Notebook to make our work more reproducible and our app more performant. The final app is available here.

Devotees of this Reproducible Finance blog series will note similarities to this Shiny app, but today’s app will have different and richer functionality.

First, we are going to be pulling in data from Quandl, so we won’t be using ticker symbols, but rather will be using country codes plus data set codes. That would allow us to open this app to the vast number of data sets available via Quandl.

Second, the user will be able to use the sidebar to select different data sets, rather than being restricted to a country ETF. For example, the user might want to select GDP-per-capita, or the exchange rate, or any data set that we wish to make available. Our app today will include 6 economic indicators.

Third, we will display a chart using highcharter instead of dygraphs. There’s not a substantive reason to be doing this, beyond that it’s a good chance to start exploring highcharter, which is a popular tool in the financial world.

Lastly, we’ll port raw data to a data.table and include a few buttons for easy download, in case our end users want to reproduce our charts or import data for their own work. We are going to use the as_tibble function from the fantastically useful new tidyquant package to facilitate our data.table. We won’t have space to cover the data download section in this post, but all the code is available in the live app.

Without further ado, let’s get to it.

The first line of substance in this app loads in the leaflet map we constructed in our Notebook. We named the object leaf_world and saved it in a file called wdiMapData.RDat. If you want to refresh your memory on how we did that, have a look back at the previous post. Apologies if that sounds tedious, but hopefully it emphasizes the workflow of doing the heavy map-building in the Notebook when possible. If you or your team ever want to use that map as the basis for another Shiny app, that Notebook will be convenient to reuse.

To load our map, we run the following:

# Load the .RDat file where the leaflet map is saved.
load('wdiMapData.RDat')

Now we can access whatever R objects were saved in that file and, in this case, that means one R object called leaf_world. That might not be a great name, but it’s the leaflet map of the world we built in the Notebook. When we are ready to build the map in our Shiny app, we’ll simply render that object.

Before we get to the map, though, let’s construct a sidebar where the user can choose which data set to display. We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators WDI. The Quandl code for WDI is WWDI and thus we’ll append WWDI/ to each data set call - but note that this will not appear in the sidebar because we won’t actually pass any data to Quandl there. We have to wait for the user to click on a county, and thus will handle the data passing in a future code chunk. This sidebar has one purpose: for the user to select the code for the economic indicator.

We want to give the user a drop-down of choices, but we don’t want the choices to be the Quandl codes. We want the choices to be the familiar names of the data sets. In other words, the user will see as a choice ‘GDP Per Capita’, instead of the Quandl code _NY_GDP_PCAP_KN. For that reason, we’ll first create an object called dataChoices that holds our name-value pairs. The name is the familiar title of the time series - for example, ’GDP Per Capita` - and the value is the Quandl code.

dataChoices <- c("GDP Per Capita" = "_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "_FR_INR_RINR",
                  "Exchange Rate" = "_PX_REX_REER",
                  "CPI" = "_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "_SL_TLF_ACTI_ZS")

Next, in our selectInput statement, we will set choices equal to dataChoices, which will allow the user to see intuitive names but choose the Quandl codes.

selectInput("indicatorSelect", 
            "Choose an economic indicator",
            # Give the user a choice from the object we created above.
            choices = dataChoices,
            selected = "GDP Per Capita")

Our simple sidebar is finished, and the end result is that when a user makes a choice, we have an input reactive with the value of the data set code. Let’s put that code aside for a minute and turn our attention to the map.

Remember, we did the map-building work on this in our Notebook, then loaded the object in the setup code chunk. That leaves us with a simple call to renderLeaflet() to pass it leaf_world.

leafletOutput("map1")

output$map1 <- renderLeaflet({
  leaf_world
  })

Alright, that wasn’t too cumbersome, and perhaps the simplicity of that step makes the hard work in the previous Notebook a bit more tolerable.

And now the fun part, wherein we build the machinery to let a user click the map and grab data from Quandl. We will proceed in four steps:

capture the country code of the country that gets clicked
call the economic time series that the user selected in the sidebar
paste the country code and the time series together to create one data set code
pass that one data set code to Quandl and import the time series

On to step 1, capturing the clicked country.

clickedCountry <- eventReactive(input$map1_shape_click, {
  return(input$map1_shape_click$id)
})

It’s same process we used in this post, but there is one difference. Let’s review what happened here.

Recall that we set layerID = ~iso_a3 when we built the map in our Notebook. That’s because Quandl appends the iso_a3 country codes to the data set code. In other words, if we want GDP-per-capita for Brazil, we need the Brazil country code ‘BRA’ so we can later append it to the GDP-per-capita code ’_NY_GDP_PCAP_KN’.

In the code chunk above, we used an observeEvent function to capture the layerID of whatever shape a user clicks, and in the Notebook, we had set that ID to be the ~isa_a3 code. When a user clicks Brazil, our reactive captures the country code ‘BRA’.

Next, we need to append that country code to the data set code. Recall that the user chose this code in our sidebar. We just need to grab his selection via another reactive.

# Nothing fancy here, a reactive value from our sidebar input.
indicatorValue <- reactive({input$indicatorSelect})

Let’s pause and take inventory of what we have captured thus far: the country code from the click on the map is stored as clickedCountry(), and the data set code from the sidebar is stored as indicatorValue(). Now we want to paste those together and pass them to Quandl. We’ll do that via another reactive, but we are also going to pass that pasted chain to Quandl in the reactive. Thus, the chunk below has one reactive that does two things.

First, it pastes together our inputs to form one data set code. Second, it passes that code to Quandl and returns the desired time series.

Notice that when we paste the codes together, we start with WWDI/. That’s because all of our data comes from the WWDI data source on Quandl. We are not letting the user choose a different data source. We could have done so in the sidebar, but consider how that would have complicated our data set code reactive input in the sidebar.

Back to our data import chunk wherein we create our data set code by pasting inputs together and then pass them to Quandl:

countryData <- reactive({
  
  # WWDI is the World Bank data set. 
  # We aren't giving the user a choice of data sources, but we could.
  dataSet <- paste("WWDI/",
                   # The country that was clicked.
                   as.character(clickedCountry()),
                   # The time series that was chosen in the sidebar.
                   as.character(indicatorValue()), 
                   sep = "")
  # Now pass that pasted data set code object to Quandl.
  Quandl(dataSet, type = "xts")
})

We now have an object called countryData that is an xts object imported from Quandl. It holds the time series (1) from the WWDI data source, (2) for the country that was clicked, (3) for the economic indicator that was chosen in the sidebar

From a data import perspective, we’re done. We could fit a model to this data, run forecasts, etc., but I am going to head straight to visualization using highcharter.

As mentioned at the outset, dygraph would have done a fine job here, but highcharter seems to be a more popular visualization tool in the financial industry, and that’s a good reason to start exploring its capabilities.

The code chunk below might look heavy, but most of the lines are for aesthetics, creating a nice title and removing scroll bars that seem clunky to me. The key substantive statements below are highchart(type = "stock") and hc_add_series(countryData()).

The first statement tells highcharter to use its built-in stock format (this is good for graphing all sorts of economic time series, not just stocks), and the second passes our data object to the graphing function hc_add_series. It’s a well-named function because it adds a series to the chart. If we wanted to include another series, we would use that same function and pass in another xts object.

output$highchart <- renderHighchart({
  
  # Make a nice title for the chart.
  # The next three linese are purely aesthetic to that our graph has an intuitive
  # title. 
  indicatorName1 <- names(dataChoices[dataChoices == input$indicatorSelect])
  countryName1 <- countrycode(as.character(clickedCountry()), "iso3c", "country.name")
  title <- paste(countryName1, indicatorName1, sep = " ")
  
  # Call highchart and ask nicely for it to use the built-in 'stock' format.
  highchart(type = "stock") %>% 
    hc_title(text = title) %>%
    # Pass our time series object countryData to highcharter function hc_add_series.
    hc_add_series(countryData(), name = title) %>%
    # I don't like the look of the navigator/scrollbar in a small space, but others might. 
    # Change these to enabled = TRUE and check out the results.
    hc_navigator(enabled = FALSE) %>% 
    hc_scrollbar(enabled = FALSE)
})

That’s all for today. This post completes what we can retrospectively consider our 3-part series of Quandl Shiny apps. The other two posts are here and here. There are definitely interesting ways to combine the functionalities in these apps, but I’ll leave that to the awesome community of Shiny developers.

See you next time, when we visit the wonderful world of portfolio theory and volatility!

A Shiny App for Exploring Commodities Prices and Economic Indicators, via Quandl

Fri, 02 Jun 2017 00:00:00 +0000

In a previous post, we created an R Notebook to explore the relationship between the copper/gold price ratio and 10-year Treasury yields (if you’re curious why we might care about this relationship, have a quick look at that previous post), relying on data from Quandl. Today, we’ll create a Shiny app that lets users choose which different commodities ratios and different economic indicators to investigate. Perhaps users don’t care about Dr. Copper and Treasury yields, but instead want to explore the oil/gold price ratio and how it correlates with the US inflation rate, or the EU inflation rate. Let’s give them some flexibility!

The finished app is available here.

Before we get to it, note a few issues that we have seen in the past.

Very similar to our previous Shiny app, in the code chunk below, we have some important decisions about how a user selects a commodity. We could use textInput to allow the user to enter the code for the desired data set which would not limit the user in any way - he or she could choose any dataset on Quandl. The cost would be that the user would need to know, or go to Quandl and look up, the code for any data set.

For example, to import iron ore prices, the user would have to type in ODA/PIORECR_USD. That’s a big problem if your end users are not familiar with and have no interest in data set codes. To emphasize convenience and usability we will go with selectInput instead of textInput, meaning our app will show a drop-down of a few choices. The user just clicks on “Iron Ore” instead of typing ODA/PIORECR_USD, or clicks on “copper” instead of typing CHRIS/CME_HG1.1. But, if a user wants to work with a data set that we haven’t included, said user is out of luck.

Another big decision is how many choices to give the user. I’ve included a few: copper, oil, iron, platinum, palladium and silver. That’s a nice smattering of commodities whose prices tend to rise with a growing economy, plus silver which does not. I included silver so we could take a look at a commodity that should behave differently from the rest. As always, our choice here is driven by how broad this app should be. We could have added steel, lead, zinc, and tin, or we could have included just copper, oil and iron, depending on the use case. Either way, the number of drop downs is another tradeoff between usability and flexibility.

The final decision is a bit more nuanced and requires looking ahead to how these inputs will be used further down in the app. Have a peak at the object called commodityChoices and you might notice that we don’t strictly need that object. We could have put the vector of choices as an argment to selectInput, so that our code would have read choices = c("Copper" = "CHRIS/CME_HG1.1", ...) instead of choices = commodityChoices. In that choice assignment, “copper” is called the name and “CHRIS/CME_HG1.1” is called the value (together we can think of them as a name-value pair). The reason for building a separate commodityChoices object is that we want the ability to extract either the name or the value of the name-value pair. Usually we would care only about the value, because we want to pass the value to Quandl and import the data, but that name is going to be useful when we label our graphs.

Without further adieu, let’s look at commodityChoices, econIndicatorChoices and the use of selectInput.

# Create a vector of commodity choices. 
commodityChoices <- c(
                  "Copper" = "CHRIS/CME_HG1",
                  "WTI oil" = "FRED/DCOILWTICO",# 
                  "Iron Ore" = "ODA/PIORECR_USD", # monthly
                  "Platinum" = "LPPM/PLAT", 
                  "Palladium" = "LPPM/PALL",
                  "Silver" = "LBMA/SILVER") 

# Make those commodity choices avaible via selectInput.
selectInput("commodity",
            "Commodity",
            choices = commodityChoices, 
            selected = "Copper")

# Create a vector of economic indicator choices.
econIndicatorChoices <- c(
                  "10-Yr Yield" = "FRED/DGS10", # daily
                  "US CPI" = "RATEINF/INFLATION_USA",# monthly
                  "Japan CPI" = "RATEINF/INFLATION_JPN",
                  "EU CPI" = "RATEINF/INFLATION_EUR")

# Make those economic indicator choices avaible via selectInput.
selectInput("econIndicator",
            "Economic Indicator",
            choices = econIndicatorChoices, 
            selected = "10-yr Yield")
 
# A standard date range input.
dateRangeInput("dateRange",
               "Date range",
               start = "1990-01-01",
               end   = "2016-12-31")

Now that we have the inputs in a sidebar for the user, it’s back to Quandl to import the data for the chosen commodity, gold and the chosen economic indicator. There’s a common date range for all three so we’ll start by creating start and end date objects.

ratio_indicator <- reactive({

Quandl.api_key("your API key here")

start_date <- format(input$dateRange[1])
end_date <- format(input$dateRange[2])

We could now write three separate calls to Quandl for each of the data sets but, instead, let’s make use of the map() function from the purrr package. If you’re not familiar with purrr, have a look here. I’ll just say that you’ll probably never have to use lapply() again (and that should be motivation enough), but, in short, the family of map() functions takes a function and applies it to the elements in a vector, similar to the apply() functions.

Before we can use map() though, we need a vector to feed it. Let’s create a vector of Quandl codes.

# Create a vector of 3 data set codes
# 1) commodity chosen by user
# 2) gold quandl code
# 3) economic indicator chosen by user

gold_code <- "CHRIS/CME_GC1.1"
# Vector of Quandl codes.
data_set_codes <- c(input$commodity, gold_code, input$econIndicator)

Then we’ll apply the Quandl() function by piping our vector of codes and using map().

# Pipe the data_set_codes vector to Quandl via the map() function
# Note we can still set the start and end date and object type
# as we always can with Quandl.

 quandlData<- data_set_codes  %>% 
        # Pipe the datasets vector to Quandl via the map() function.
        map(Quandl,
            start_date = start_date,
            end_date = end_date,
            collapse = "monthly",
            type = "xts")     %>%

Next, we will use map() to apply the na.locf() function to our time series and ensure that no NAs remain.

        # Replace all NAs using map() and na.locf().
        map(na.locf, formLast = TRUE) %>%

If we stopped here, we would have a list of three xts series, but I don’t want a list, I want one xts object. So, we’ll pipe our list of three and use the reduce() + merge() to combine our list of 3 time series into one xts object.

        # Merge to one xts object using map() and merge().
        reduce(merge) %>% 
        # Add nicer column names.
        `colnames<-`(c(names(commodityChoices[commodityChoices == input$commodity]), 
                            "Gold",
                            names(econIndicatorChoices[econIndicatorChoices == input$econIndicator])))

Alright, after running our Quandl codes through that series of mapped functions, we have three time series stored in one xts object, and now we want to calculate the price ratio of the chosen commodity/gold.

To create that price ratio, we need to divide the first column by the second column and we’ll store it in a new column called ratio.

Then we will save just that ratio and the economic indicator column data into their xts object. That is not necessary but it makes things cleaner and easier when we pass to dygraph().

# Create a column and add the price ratio.
quandlData$ratio <- quandlData[,1]/quandlData[,2]

# Save just the ratio and the economic indicator data.
ratio_indicator  <- merge(quandlData$ratio, quandlData[,3])

# Add more general names.
colnames(ratio_indicator) <- c("ratio","indicator")       

return(ratio_indicator)
})

Now we just need to pass our reactive object ratio_indicator() to dygraph() and follow the same steps as we did when testing in our Notebook.

We will use dyRoller() to smooth out our chart and make each point an average of the number of periods specified with rollPeriod = X. This won’t affect our xts object, where we store the data, it just makes the chart more readable.

Remember also that we are charting two time series on the same chart and they are on different scales, so we want to add a right-hand-side y-axis.

To do so, we need to invoke dyAxis() for the left-hand axis, called “y”. Then we invoke dyAxis() for the right-hand axis, called “y2”. We also need to set independentTicks = TRUE so that we can use a unique, independent value scale for the right-hand side. Next, in our dySeries() call for each time series, we assign each one to an axis. Here we assign “ratio” with axis = 'y', so that the commodity-gold price ratio will be on the left-hand scale, and we assign “indicator” with axis = 'y2', so the economic indicator will be on the right-hand scale.

dygraphOutput("ratio_indicator")

output$ratio_indicator <- renderDygraph({
  dygraph(ratio_indicator()) %>% 
    
  # Add the rollPeriod for smoothing.
  dyRoller(rollPeriod = 3) %>% 
    
  # Create two independent axes, just we did in the Notebook.
  dyAxis("y", label = "USD") %>%
  dyAxis("y2", label = "Percent (%)", independentTicks = TRUE) %>%
    
  # Assign each time series to an axis. 
  # Use the name from the name-value pair to create nice labels for each.  
  dySeries("ratio", axis = 'y', 
           label = paste(names(commodityChoices[commodityChoices == input$commodity]), 
                         "/Gold (LHS)", sep = ""), color = "blue") %>% 
  dySeries("indicator", axis = 'y2', 
           label = paste(names(econIndicatorChoices[econIndicatorChoices == input$econIndicator]), 
                         "(RHS)", sep = ""), color = "red")
})

We could end things here but let’s go ahead and add a chart to show the rolling correlation between the ratio and the indicator. We’ve done so much work to calculate and wrangle these time series, might as well put them to use!

First, we’ll calculate the rolling correlation using the rollapply() function. Nothing too complicated here.

dygraphOutput("rollingCorrelation")

output$rollingCorrelation <- renderDygraph({
  
rolling_cor <- rollapply(ratio_indicator(), 24,
                         function(x) cor(x[, 1], x[, 2], use = "pairwise.complete.obs"),
                         by.column = FALSE)
# Make a nicer name for the xts object that stores the rolling correlation. 
# This name will be displayed when a user hovers on the dygraph. 
names(rolling_cor) <- paste(names(commodityChoices[commodityChoices == input$commodity]),
                            "/Gold ",
                            names(econIndicatorChoices[econIndicatorChoices == input$econIndicator]), 
                            " Correlation", sep = "")

It’s not necessary, but I like to display the mean, minimum and maximum rolling correlations on the chart. We’ll store those in three objects: avg, mini, and maxi.

  avg  <- round(mean(rolling_cor, na.rm = T), 2)
  mini <- round(min(rolling_cor,  na.rm = T), 2)
  maxi <- round(max(rolling_cor,  na.rm = T), 2)

Now we pass our rolling_cor xts object to dygraph() and pass the mean, minimum and maximum objects to dyLimit().

dygraph(rolling_cor, main = paste(names(commodityChoices[commodityChoices == input$commodity]),
                            "/Gold ",
                            names(econIndicatorChoices[econIndicatorChoices == input$econIndicator]), 
                            " Correlation", sep = "")) %>% 
  dyRangeSelector(dateWindow = c("2015-01-01", "2016-12-31"))  %>% 
  # Add a line for the mean, min and max.
  dyLimit(avg, color = 'purple') %>% 
  dyLimit(mini, color = 'red') %>% 
  dyLimit(maxi, color = 'blue') %>% 
  # Add an event for the US election.
  dyEvent("2016-11-08", "Trump!", labelLoc = "bottom")
})

And, we’re done! It’s fun to explore different relationships amongst different time series with this app. And once we have this template in the toolkit, all sorts of different data sets can be substituted in for exploration. For example, we might want to port this work over to a currencies dashboard, or a country GDP dashboard. The nice thing is, it’s just a matter of finding the right Quandl codes and imagining new hypotheses to explore.

Things got a little choppy today with all the piping, so just a reminder that if you want the reusable code for this app, it’s available via the source code button at the top right of the live app. Thanks, and see you next time.

Mapping Quandl Macroeconomic Data

Wed, 10 May 2017 00:00:00 +0000

In previous posts, we built a map to access global ETFs and a simple Shiny app to import and forecast commodities data from Quandl. Today, we will begin a project that combines those previous apps. Our end goal is to build an interactive map to access macroeconomic data via Quandl, allowing the user to choose an economic indicator and click on a country to access that indicator’s time series. For example, if a user wants to import and graph GDP-per-capita data from China, that user will be able to select GDP per Capita from a drop-down menu, and then click on China to display the time series. As usual, we’ll start with a Notebook for data import, test visualizations, and map building.

Careful readers will note that we are not simply reusing code from those previous projects; that is because we are going to introduce some new packages to our toolkit. Specifically:

Instead of downloading a spatial data frame from Natural Earth, we will use the new R package rnaturalearth and it’s function ne_countries() to import our geographic data.
Relatedly, we are not going work with a “spatial” data frame, but rather with a simple features data frame of class sf. Learn more here.
Finally, when we test our data import of several macroeconomic data sets, we will use the map() function from the purrr package instead of relying on lapply(), as we did previously when passing ticker symbols to getSymbols(). If you’re like me, you won’t be sad to be done with lapply().

Let’s get to it!

We are going to be working with a macroeconomic data source from the World Bank called World Development Indicators (WDI). The Quandl code for WDI is WWDI, and thus we’ll prepend WWDI/ to each data set call.

One difference from our earlier work with Quandl is that previously we imported and visualized global time series, such as the price of oil, gold or copper. By global, I mean these time series were not associated with a particular country. That simplified our job a bit because we did not need to supply a country code to Quandl. Today, we do need to include a country code.

Let’s start with a simple example and import GDP-per-capita data for China, whose country code is CHN. We will pass the string WWDI/CHN_NY_GDP_PCAP_KN to Quandl, which consists of the WDI code WWDI/, appended by the China country code CHN, appended by the GDP-per-capita code _NY_GDP_PCAP_KN.

library(Quandl)
# Pass the code string to Quandl. 
China_GDPPC <- Quandl("WWDI/CHN_NY_GDP_PCAP_KN", type = 'xts') %>% 
  # Add a nice column name
  `colnames<-`("GDP Per Capita")

# Take a look at the result.
tail(China_GDPPC, n = 6)

##            GDP Per Capita
## 2010-12-31       30876.04
## 2011-12-31       33658.85
## 2012-12-31       36126.73
## 2013-12-31       38737.58
## 2014-12-31       41354.61
## 2015-12-31       43991.55

That looks pretty good, but note that the data only goes back to December of 2015 - not ideal, but fine for our illustrative purposes.

Our eventual Shiny app, though, will give the user a choice among many economic time series, and we want to test the import and visualization of all those choices. In addition to GDP-per-capita, let’s test GDP-per-capita growth, real interest rate, exchange rate, CPI, and labor force participation rate. Those offer a nice snapshot of a country’s economy.

To run our test in the code chunk below, we will first build a vector of data set codes. Note that each of them start with WWDI/, then CHN, and then the relevant code. Next, we will pipe that vector to Quandl using the map() function. map() takes a function, in this case the Quandl() function, and applies it to the elements in a vector, in this case our vector of data set codes. We also want to specify type = "xts".

If we stopped our pipe there, the output would be a list of six time series and that would suit our testing purposes, but I don’t love having to look at a list of six. Thus, we’ll keep on piping and use the reduce() function from the purrr package and call merge() to combine the six lists into one xts object. That is much easier to deal with, and easier to pass to dygraphs for our test visualization. The last piped function will invoke colnames<- to clean up the column names.

library(purrr)
# Create a vector of economic indicators that can be passed to Quandl via map().
# Include names and values for easy naming of xts columns.
econIndicators <- c("GDP Per Capita" = "WWDI/CHN_NY_GDP_PCAP_KN",
                  "GDP Per Capita Growth" = "WWDI/CHN_NY_GDP_PCAP_KD_ZG",
                  "Real Interest Rate" = "WWDI/CHN_FR_INR_RINR",
                  "Exchange Rate" = "WWDI/CHN_PX_REX_REER",
                  "CPI" = "WWDI/CHN_FP_CPI_TOTL_ZG",
                  "Labor Force Part. Rate" = "WWDI/CHN_SL_TLF_ACTI_ZS")

# You might want to supply your Quandl api key. It's free to get one.
Quandl.api_key("d9EidiiDWoFESfdk5nPy")

China_all_indicators <- 
  # Start with the vector of Quandl codes
  econIndicators %>% 
  # Pass them to Quandl via map(). 
  map(Quandl, type = "xts") %>% 
  # Use the reduce() function to combine them into one xts objects.
  reduce(merge) %>% 
  # Use the names from the original vector to set nicer column names.
  `colnames<-`(names(econIndicators))
# Have a look.
tail(China_all_indicators, n = 6)

##            GDP Per Capita GDP Per Capita Growth Real Interest Rate
## 2011-12-31       33658.85              9.012854          -1.472149
## 2012-12-31       36126.73              7.332031           3.523236
## 2013-12-31       38737.58              7.226936           3.692594
## 2014-12-31       41354.61              6.755778           4.732429
## 2015-12-31       43991.55              6.376423           4.270386
## 2016-12-31             NA                    NA                 NA
##            Exchange Rate      CPI Labor Force Part. Rate
## 2011-12-31      102.6904 5.410850                   76.7
## 2012-12-31      108.4435 2.624921                   77.0
## 2013-12-31      115.2980 2.627119                   77.3
## 2014-12-31      118.9863 1.996847                   77.6
## 2015-12-31      131.6298 1.442555                     NA
## 2016-12-31      124.1864 2.007556                     NA

That seems to have done the job, and note that both CPI and Exchange have data up to December of 2016. Much better, and a sign that there will be date variation among countries and indicators. That won’t be crucial today, but it could be if we needed consistent dates for all of our data sets.

As a final step, let’s use the dygraph() function to visualize these time series to make sure things work properly before moving to Shiny.

library(dygraphs)

## Warning: package 'dygraphs' was built under R version 3.3.2

dygraph(China_all_indicators$`GDP Per Capita`, main = "GDP Per Capita")

dygraph(China_all_indicators$`GDP Per Capita Growth`, main = "GDP Per Capita Growth")

dygraph(China_all_indicators$`Real Interest Rate`, main = "Real Interest Rate")

dygraph(China_all_indicators$`Exchange Rate`, main = "Exchange Rate")

dygraph(China_all_indicators$`CPI`, main = "CPI")

dygraph(China_all_indicators$`Labor Force Part. Rate`, main = "Labor Force Part. Rate")

GDP-per-capita has been on a steady march higher, labor force participation plummeted in the 1990’s and 2000’s, probably as the labor market became more market-oriented, and real interest rates look quite choppy. Plenty to learn, and it signals that our Shiny app might be a nice exploratory tool.

For now, it’s on to map-building!

In previous posts, we have started map-building by grabbing a spatial object from the internet. Today, we will use the new rnaturalearth package and its fantastic function ne_countries(). We will specify type = "countries" and returnclass = 'sf'. We could have set returnclass = 'sp' if we wanted to work with a spatial data frame, but I have been migrating over to the sf package and simple features data frames for mapping projects. Lots to learn here, but in general, it seems to use more tidy concepts than the sp package.

library(rnaturalearth)
library(sp)
world <- ne_countries(type = "countries",  returnclass = 'sf')

# Take a peek at the name, gdp_md_est column and economy columns. 
# The same way we would peek at any data frame.
head(world[c('name', 'gdp_md_est', 'economy')], n = 6)

## Simple feature collection with 6 features and 3 fields
## geometry type:  MULTIPOLYGON
## dimension:      XY
## bbox:           xmin: -73.41544 ymin: -55.25 xmax: 75.15803 ymax: 42.68825
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
##                   name gdp_md_est                   economy
## 0          Afghanistan      22270 7. Least developed region
## 1               Angola     110300 7. Least developed region
## 2              Albania      21810      6. Developing region
## 3 United Arab Emirates     184300      6. Developing region
## 4            Argentina     573900   5. Emerging region: G20
## 5              Armenia      18770      6. Developing region
##                         geometry
## 0 MULTIPOLYGON(((61.210817091...
## 1 MULTIPOLYGON(((16.326528354...
## 2 MULTIPOLYGON(((20.590247430...
## 3 MULTIPOLYGON(((51.579518670...
## 4 MULTIPOLYGON(((-65.5 -55.2,...
## 5 MULTIPOLYGON(((43.582745802...

This output looks similar to an sp object, but it’s not identical. In particular, note the geometry column, which contains the multipolygon latitude and longitude coordinates. It’s a matter of personal preference, but I find this sf object more intuitive in terms of how the geometry corresponds with the rest of the data.

Now we’ll create our shading and popups. Note that this code is identical to when we were working with spatial data frames.

library(leaflet)
# Create shading by GDP. Let's go with purples.
gdpPal <- colorQuantile("Purples", world$gdp_md_est, n = 20)

# Create popup country name and income group so that something happens
# when a user clicks the map.
popup <- paste0("<strong>Country: </strong>", 
                world$name, 
                "<br><strong>Market Stage: </strong>", 
                world$income_grp)

Now, it’s time to build our leaflet map. This code is nearly identical to our code from previous projects, but with one major change. As we saw above, we need to pass in a country code to Quandl, not a country name and not a ticker symbol. Thus, we don’t want to set layerId = name, but we also don’t need to add anything special to our world object. It already contains a column called iso_a3, and that column contains the country codes used by Quandl. Fantastic!

We just need to set layerId = ~iso_a3 and we will be able to access country codes in our Shiny app when a user clicks the map. This is quite fortunate, but it also highlights a good reason to spend time studying the world object. It might contain other data that will be useful in future projects.

leaf_world <- leaflet(world) %>%
  addProviderTiles("CartoDB.Positron") %>% 
  setView(lng =  20, lat =  15, zoom = 2) %>%
      addPolygons(stroke = FALSE, smoothFactor = 0.2, fillOpacity = .7, 
      
      # Note the layer ID. Not a country name! It's a country code! 
      
      color = ~gdpPal(gdp_md_est), layerId = ~iso_a3, popup = popup)

leaf_world

Because Quandl uses the ISO 3-letter code to identify a country, and because the rnaturalearth sf object already contains a column with iso_a3 country codes, building our map wasn’t very hard. It’s worth pointing out that in various projects, we have now built maps with three different layerIDs. In the Global ETF Map project, we built one map with layerID = name, using the built-in name column in the rnaturalearth object, and we built another map with layerID = tickers, using a column of tickers that we added ourselves with the merge() function. Now, today, we have a map with layerID = iso_a3, using the built-in column of country codes. We are building up a nice portfolio of maps with different layerIDs should we ever want or need them in the future.

That’s all for today. We’ll save that leaflet_world object for use in our Shiny app! See you next time.

A Shiny App for Importing and Forecasting Commodities Prices from Quandl

Fri, 21 Apr 2017 00:00:00 +0000

In a previous post, we imported oil data from Quandl and applied a simple model to it. Today, we’ll port that work over to a Shiny app (by way of flexdashboard, of course) that allows a user to choose a commodity (oil, copper or gold), choose a frequency for the time series, and choose how many periods ahead to forecast. The app will display the price history and the forecasted price. To see it in action, have a look here.

As with the previous Notebook, the main purpose of this post is to build a template where a more sophisticated or proprietary model could be used for forecasting. By setting it up so that models could be selected as inputs, several different models could be included. We are not as concerned with the modeling as much as we are with providing a format that is friendly to both end users and any future collaborator that might want to take this app and expand upon it.

Let’s get to it! In the code chunk below, we are immediately faced with a few important decisions. The first of those is the format of the user input. At the extreme, we could use textInput and simply allow the user to enter the code for the desired data set. The benefit is that we would not limit the user in any way; he or she could choose any dataset on Quandl. The cost would be that the user would need to know - or go to Quandl and look up - the code for any data set.

For example, to import WTI oil prices, the user would have to type in “FRED/DCOILWTICO”. That’s no problem if most of the end users know that code, but it’s a big problem otherwise. We want to emphasize convenience and broad usability, so we are going with selectInput instead of textInput, meaning our app will show a drop-down of a few choices. The user just selects “WTI oil” instead of typing “FRED/DCOILWTICO”“, or selects”copper" instead of typing “ODA/PCOPP_USD”. But, if a user wants to work with a data set that we haven’t included, said user is out of luck.

Another big decision is how many choices to give the user. I have included only 3: oil, gold and copper. In industry, you would probably include several more, perhaps all of the industrial metals, but there’s is a cutoff somewhere. Or maybe we prefer one choice, because this app is just for oil analysis. Either way, the number of options in the drop-down menu is another trade-off between usability and flexibility.

The final decision is a bit more nuanced, and requires looking ahead to how these inputs will be used further down in the app. Have a peek at the object called dataChoices, and you might notice that we don’t strictly need that object. We could have put the vector of choices as an argument to selectInput, so that our code would have read choices = c("WTI oil" = "FRED/DCOILWTICO", ...) instead of choices = dataChoices. In that choice assignment, “WTI oil” is called the name and “FRED/DCOILWTICO” is called the value (together we can think of them as a name-value pair). The entire reason for building a separate dataChoices object is that we want the ability to extract either the name or the value of the name-value pair. Usually we would care only about the value, because we want to pass the value to Quandl and import the data, but that name is going to be useful as well when we label our graph.

The ability to extract names and values will become even more useful when we get to the frequency of the time series and forecasting. For now, let’s look at dataChoices and selectInput.

# Notice a tradeoff here: we're making it easy on our users because they don't need to 
# remember the naming conventions. But, we're also forced to severely limit their choices. 
# The dataChoices object is going to allow us to add a nicer label to the graph. 
# Notice also how easily we can include datasets from different sources, and not worry about
# their formats. Thanks, Quandl!

dataChoices <- c("WTI oil" = "FRED/DCOILWTICO", # oil data from Fred
                 "Copper" = "ODA/PCOPP_USD", # copper data from ODA
                 "Gold" = "CHRIS/CME_GC1") # gold data from CME

selectInput("dataSet",
            "Commodity",
            choices = dataChoices,
            selected = "WTI oil")

Alright, we have given the user the ability to choose a commodity. Next we want to ask about the frequency, and this gets a bit more complicated. We need to tell Quandl the frequency of the time series to import using the “daily”, “weekly”, “monthly” convention, so we set those to the values. But, further down the app when we want to store the forecast results, we’ll need to use the “days”, “weeks”, “months” phrasing and we’ll need to pull out the names from the name-value pair in frequencyChoices. We knew that this would be necessary because when we built our Notebook for importing, wrangling, and testing, we noted the different conventions and started thinking about how to deal with them in the Shiny context.

# The frequencyChoices object is going to allow us to pass different period conventions
# to different places further down the app. 

frequencyChoices <- c("days" = "daily",
                      "weeks" = "weekly", 
                      "months" = "monthly")

selectInput("frequency",
            "freq",
            choices = frequencyChoices, 
            selected = "months")

The remainder of the inputs should look familiar as they are in standard format.

dateRangeInput("dateRange",
               "Date range",
               start = "1980-01-01",
               end   = "2016-12-31")

numericInput("periods", "Periods to Forecast", 6, min = 1, max = 100)

Now that we’ve built the sidebar inputs, let’s put them to use. First, we will import the commodity time series data from Quandl. This will be familiar from the Notebook, but note in particular that we will use the value from the input$frequency choice because Quandl uses the “daily/weekly/monthly” frequency format.

# Let's pull in the Quandl data. 
# Nothing fancy here except we are going to use the reactive inputs.
# We will pass in the value from the input$dataSet key-value pair.
# We will also pass in the value from the input$frequency key-value pair. 

commodity <- reactive({
  # It might be a good idea to include your quandl api key if this will be used more than 50 times in a day. 
  Quandl.api_key("your_apikey_here")
  
  commodity <- Quandl(input$dataSet,
                      start_date = format(input$dateRange[1]),
                      end_date = format(input$dateRange[2]),
                      order = "asc",
                      type = "xts",
                      collapse = as.character(input$frequency)
  )
})

We have imported a time series object for the dataset, date range and frequency chosen by the user. Now we want to do some forecasting and create a visualization. We’ll first use the forecast() function, then we’ll combine the forecasted prices and the historical prices into one xts object that can be passed to dygraph. Let’s handle this in one reactive.

First, we’ll call forecast and pass it the periods input from the user.

combined_xts <- reactive({
  
  # Just like the Notebook, except periods is a reactive input.
  
  forecasted <- forecast(commodity(), h = input$periods)

Now we need to combine that forecasted object with the commodity object that was created in a previous code chunk. Here we will finally thank ourselves for thinking about the name-value issue when assigning frequencyChoices. When we create a dataframe to hold the forecasted time series, we build a ‘date’ column using the seq() function. The column starts on the end date of the historical data (which the user selected with the dateRange input) and runs for as many periods as the user chose in the periods input. But, we need to supply a value to the by = ... argument of the seq() function so that it knows if we want to move 6 days, 6 weeks, 6 months, etc. To do that, we need to extract the name (“days/weeks/months”) from the frequencyChoices name-value pair and pass it to seq. The way we extract the name is with this selection statement: names(frequencyChoices[frequencyChoices == input$frequency]).

  forecast_dataframe <- data.frame(
    date = seq(input$dateRange[2], 
               # The next line is very important, and it's the reason we 
               # created the frequencyChoices object.
               by = names(frequencyChoices[frequencyChoices == input$frequency]),
               length.out = input$periods),
    Forecast = forecasted$mean,
    Hi_95 = forecasted$upper[,2],
    Lo_95 = forecasted$lower[,2])

Now we convert the forecast_dataframe object to an xts and combine that new object with the commodity xts object. Remember, we imported the commodity data from Quandl in the form of an xts object, which saved us from having to do a conversion. This chunk should look very similar to the Notebook.

  forecast_xts <- xts(forecast_dataframe[,-1], order.by = forecast_dataframe[,1])
  
  combined_xts <- cbind(commodity(), forecast_xts)
  
  # Add a nicer name for the first column.
  
  colnames(combined_xts)[1] <- "Actual"
  
  # This is the combined object that will be passed to dygraphs below. 
  combined_xts
})

Next, we will create a chart of the actual price history. Nothing fancy here, except again we are going extract the ‘name’ portion of a name-value pair, this time from the dataSet object (the first one we created in the first code chunk). We want to label the graph with ‘WTI oil’ and not the Quandl code, so we select the name with names(dataChoices[dataChoices==input$dataSet]).

dygraphOutput("commodity")

output$commodity <- renderDygraph({
  dygraph(commodity(),
          # We pull out the name of the selected name-value input$dataSet like so: 
          # names(dataChoices[dataChoices==input$dataSet])
          main = paste("Price history of", names(dataChoices[dataChoices==input$dataSet]), 
                       sep = " ")) %>%
    dyAxis("y", label = "$") %>%
    dyOptions(axisLineWidth = 1.5, fillGraph = TRUE, drawGrid = TRUE)
})

Last but not least, let’s graph the historical and forecasted time series together on one graph. We could have ported the code directly from the Notebook, but I couldn’t help tinkering just a bit. I wanted to focus the graph on the time period around where the actual data ends and the forecast begins, because that is probably what’s of most interest to the user. To do that, we’ll use the dyRangeSelector() function and pass two values to the dateWindow variable: a start date and end an date. This is pure aesthetics, but I think it’s worth the effort here.

We are going to use seq() and names(frequencyChoices[frequencyChoices == input$frequency]) once again.

dygraphOutput("forecasted")

output$forecasted <- renderDygraph({

# We want the user to be able to see the forecasted area, so let's focus in on that 
# by truncating the view of the dygraph.
# We need to give the graph a start date and an end date. 
start_date <- tail(seq(input$dateRange[2], by = "-1 months", length = 6), 1)
end_date <- tail(seq(input$dateRange[2], 
                     by = names(frequencyChoices[frequencyChoices == input$frequency]), 
                     length = input$periods), 1)

Now we will supply those date objects to a piped dygraphs chain, and that will be a wrap. Have a close look at the chunk below and spot where we again extract the name from the input$dataSet name-value pair.

dygraph(combined_xts(), 
        # Name about to be extracted!
        main = paste(names(dataChoices[dataChoices==input$dataSet]), 
                     ": Historical and Forecast", sep = "")) %>%
  # Add the actual series.
  dySeries("Actual", label = "Actual") %>%
  # Add the three forecasted series.
  dySeries(c("Lo_95", "Forecast", "Hi_95")) %>% 
  # A range selector to focus on the where historical data ends and 
  # foracated data begins.
  # Note the user can still use the range selector to zoom out if so desired.
  dyRangeSelector(dateWindow = c(start_date, end_date))
})

That’s all for today! We’ve built an app that can be extended in many directions - more data set choices, an input selector so the user can choose different models, more visualizations - and in that sense it can serve as a template for ourselves and anyone that wants to build upon this work.

This app also relies on and highlights one of the tremendous benefits of Quandl: the ability to import datasets from different sources and have them arrive in the IDE in a consistent format. This makes using Quandl with Shiny quite enjoyable after the initial skeleton is built. For example, if we or a colleague did want to change this app and substitute in different commodities, there would be no worry beyond the finding the right code on Quandl and adding another name-value pair to dataChoices.