R Language on R Views

May 2023: "Top 40" New CRAN Packages

Wed, 28 Jun 2023 00:00:00 +0000

This will be my final R Views post offering my totally idiosyncratic selection of 40 CRAN packages. It has been a good run that began in August 2016. I would like to thank all of you who followed these posts. Your readership kept me going. I hope that the “Top 40” posts were somewhat useful for helping you to keep up with the explosion of creativity in the R Community.

I would also like to thank the guest authors who contributed posts to R Views over the years. It has been a pleasure and privilege to work with all of you. Please find a way to keep writing about R. When you do have something to share on R Views, I am certain you will enjoy working with R Views’ new senior editor, Isabella Velásquez as I have. It has been my great pleasure to collaborate with Isabella.

One hundred ninty-eight new packages stuck to CRAN in May. (I identified eight packages that were removed between the end of may and the 24th of June.) Here are my “Top 40” selections in fifteen categories Astronomy, Biology, Chemistry, Climate Science, Computational Methods, Data, Ecology, Engineering, Finance, Genomics, Mathematics, Medicine, Statistics, Utilities, and Visualization.

Astronomy

snvecR v3.7.7: Provides functions to calculate precession and obliquity from an orbital solution and assumed or reconstructed values for tidal dissipation and dynamical ellipticity. See Zeebe and Lourens (2022) for background and the vignette for examples.

Biology

dwctaxon v2.0.2: Provides tools to edit and validate taxonomic data in compliance with Darwin Core standards (Darwin Core Taxon class). There is an Indroduction, an Example, and there are vignettes on Editing and Validating texon data.

Chemistry

anabel v3.0.1: Implements tools suitable for high-throughput analysis of 1:1 molecular interaction studies. See Kraemer et al. (2019) and Norval et al. (2019) for background and the vignette for examples.

Climate Science

myClim v1.0.1: Provides functions for working with microclimate data including reading, converting data formats, sorting into localities, computing microclimate variables, and plotting. See the vignette.

Computational Methods

matrixset v0.1.1: Provides functions to create, store, manipulate and code with matrix ensembles, matrices that share common properties. See the Gentle Introduction, the example, and the vignettes on annotation, applying functions.

zonohedra v0.2-2: Provides functions to compute and plot zonohedra from vector generators. See Hecbert (1985) for the optimization methods. Vignettes include a User Guide, Matroids, Ray tracing zonohedron boundaries, Transition subcomplex and surface, and Zonotopes.

woyler v0.0.5: Provides functions for generating a tour path by interpolating between d-D frames in p-D space using Givens rotations. See Buja et al (2005) for background on high dimensional rotations, the Introduction and the vignette on Plotting.

Data

appeears v1.0: Implements a programmatic interface to the NASA Application for Extracting and Exploring Analysis Ready Samples services (AppEEARS) providing easy access to analysis ready earth observation data. See the vignette.

BRVM v5.0.0: Provide real-time access to data from the Regional Securities Exchange SA, commonly known as the BRVM stock exchange.

denguedatahub v1.0.4: Provides a weekly, monthly, yearly summary of dengue cases by state, province, or country. Look here for information on the data provided.

Ecology

forestat v1.0.1: Provides functions for evaluating the quality of a natural forest including tree height classification, tree height model establishment, sectional area growth and stock growth modeling, and for calculating actual and potential forest productivity. see the vignette.

hemispheR v0.2.0: Provides functions for processing hemispherical canopy images including functions to import and classify canopy fish-eye images, estimate angular gap fraction and derive canopy attributes like leaf area index and openness. See Chianucci & Macek (2023) for details and README for examples.

traitstrap v0.1.0: Calculates trait moments from trait and community data using the methods developed in Maitner et al (2021). See the vignette for an introduction and the playlist, a definite innovation in R package vignettes.

Engineering

navigation v0.0.1: Implements the framework presented in Cucci et al. (2023) which allows analyzing the impact of sensor error modeling on the performance of integrated navigation (sensor fusion) based on inertial measurement unit (IMU), Global Positioning System (GPS), and barometer data. See the vignettes Compare Models and Model Evaluation.

Finance

FinNet v0.1.1: Provides classes, methods. and functions to build and manipulate financial networks that include multiple types of relationships including ownership, board interlocks and legal entities. Look here for detailed information.

intradayModel v0.0.1: Provides functions to model, analyze, and forecast intraday signals via a univariate state-space model for intraday trading volume provided by Chen (2016). See the vignette.

StockDistFit v1.0.0: Provides functions for fitting probability distributions to stock price data using maximum likelihood estimation to find the best-fitting distribution for a given stock. See Siew et al. (2008) for background and the vignette for details.

Genomics

ApackOfTheCLones v0.1.2: Provides functions to visualize T-cell clonal expansion via circle-packing that can integrate single-cell RNA sequencing and T-cell receptor libraries from the 10X genomics Single Cell Immune Profiling and Cell Ranger pipeline, to produce a simple, publication-ready visualization of the clonal expansion. See README for examples.

baldur v0.0.1: Implements a hierarchical Bayesian model for statistical decisions in proteomics that includes two regression models for describing the mean-variance trend, a gamma regression or a latent gamma mixture regression. The regression model is then used as an Empirical Bayes estimator for the prior on the variance in a peptide. See Berg & Popescu (2023) for the theory and the UPS and Yeast tutorials.

Mathematics

resde v1.1: Implements maximum likelihood estimation for univariate reducible stochastic differential equation models for discrete, possibly noisy observations, not necessarily evenly spaced in time. See Garcia (2019) for the theory and the vignette for some math and examples.

Medicine

PatientProfiles v0.2.0: Provides methods to identify the characteristics of patients in data mapped to the Observational Medical Outcomes Partnership (OMOP) common data model. See the vignettes Get cohort intersections and Get cohort patients characteristics.

PANACEA v1.0.0: Provides a collection of personalized anti-cancer drug prioritization approaches utilizing network methods that utilize personalized “driverness” scores from driveR to rank drugs and map them onto a protein-protein interaction network. See the vignette to get started.

Statistics

boostingDEA v0.1.0: Provides functions to estimate the production frontier and predict ideal output in the Data Envelopment Analysis (DEA) context using both standard models from DEA and Free Disposal Hull (FDH) and boosting techniques. See Guillen et al. (2023) for background and the vignette for examples.

gFormulaMI v1.0.0: Implements the G-Formula method for causal inference with time-varying treatments and confounders using Bayesian multiple imputation methods, as described by Bartlett et al. (2023). See the vignette.

nestedLogit v0.3.2: Provides functions for specifying and fitting nested dichotomy logistic regression models for a multi-category response, and methods for summarizing and plotting those models. There are vignettes on Nested dichotomies Logit models, Plotting, and Standard Errors.

nlpsem v0.1.1: Provides computational tools for intrinsically nonlinear longitudinal models in multiple scenarios including multivariate models with time varying covariates, latent class, and a goal to assess correlation or causation among multiple longitudinal variables. Liu (2023) elaborates on the details. The vignettes provide examples for Latent Change Score Models, Latent Growth Curve Models, Multivariate Longitudinal Models, Longitudinal Mixture Models, Longitudinal Mediation Models, and Time-varying Covariate Models.

npcurePK v1.0-2: Provides functions to perform nonparametric estimation in mixture cure models when the cure status is partially known. See Safari et al. (2021), Safari et al. (2022), and Safari et al. (2023) and the vignette for an example.

semlbci v0.10.3: Implements likelihood-based confidence intervals for parameters in structural equation modeling. See Cheung and Pesigan (2023), Pek and Wu (2018), and Falk (2018) for details. There are vignettes: Get Started, Log Profile Likelihood of a Parameter, and Searching for One Bound.

ssMRCD v0.1.0: Provides functions to estimate the Spatially Smoothed Minimum Regularized Determinant (ssMRCD) estimator and use it for outlier detection as described in Puchhammer and Filzmoser (2023) See the Introduction.

survstan v0.0.2: Implements parametric survival regression models under the maximum likelihood approach via Stan including accelerated failure time models, proportional hazards models, proportional odds models, accelerated hazard models, and Yang and Prentice models. See Bennett (1982, Chen and Wang(2000), Demarqui and Mayrink (2021) for background, README for some math, and the vignette for an example.

voi v1.0: Implements methods to calculate the expected value of information from a decision-analytic model. See Jackson et al. (2022) for the theory, the Package Overview and the vignette on Plots.

Utilities

crc32c v0.0.2: Implements hardware-based support for CRC32C cyclic redundancy checksum function for x86_64 systems with SSE2 support as well as for the arm64. The functionality is exported at the C-language level for use by other packages. See README for some background.

glossary v1.0.0: Implements tools to add glossaries to markdown and quarto documents by tagging individual words. Definitions can be provided inline or in a separate file. See the vignette.

iterors v1.0: Attempts to improve the performance of iterators in R. The package is cross-compatible with iterators and includes a collection of iterator constructors and combinators ported and refined from the iterators, itertools, and itertools2 packages. There are several vignettes including an Introduction, a Guide on using the package, and a vignette on Writing Custom Iterators..

rebib v0.2.2: Implements tools to convert embedded LaTeX bibliographies into a close BibTeX equivalent that is useful for web publishing systems (such as the R Journal) that do not support embedded bibliographies. See the Introduction and the vignettes general usage, parser mechanics, and use with RJ article.

sasr v0.1.2: Provides a SAS interface that enables creating SAS sessions, executing SAScode on remote SAS servers, exchanging data sets between SAS and R and more. See the vignette.

tidytlg v0.1.2: Implements tools to generate tables, listings, and graphs (TLG) using the tidyverse. Tables can be created functionally, using a standard process, or by specifying table and column metadata to create generic analysis summaries. There are several vignettes including examples for Freauency Analysis, Table Manipulation, and a Get Started guide.

Visualization

ggautomap v0.3.2: Extends ggplot2 to convert place names to coordinates without having to work directly with the underlying geospatial tools. See the Getting Started Guide.

ggplend v0.1.0: v0.1.0: Implements an algebra of operations for blending, copying, adjusting, and compositing layers in ggplot2 that supports copying and adjusting the aesthetics or parameters of an existing layer, partitioning a layer into multiple pieces for re-composition, applying affine transformations to layers, and combining layers. See README for examples.

ggtricks v0.1.0: Provides collection of ggplot2() geoms to create graphics using Cartesian coordinate system and avoid the use of polar coordinates. Look here for examples.

isopleuros v1.0.0: Provides functions to display data in ternary space, to add or tune graphical elements and to display statistical summaries. It also includes common ternary diagrams which are useful for archaeologists. See README for examples.

tidygam v0.2.0: Provides functions that compute predictions from Generalized Additive Models (GAMs) fitted with mgcv and return them as a tibble. See the vignette to get started.

April 2023: "Top 40" New CRAN Packages

Thu, 25 May 2023 00:00:00 +0000

One hundred fifty-six new packages made it to CRAN in April. Here are my “Top 40” selections in twelve categories: Computational Methods, Data, Ecology, Economics, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

clarabel v0.4.1: Implements Clarabel, a versatile interior point solver that solves linear programs, quadratic programs, second-order cone programs, and problems with exponential and power cone constraints. See the vignette.

condor v1.0.0: Provides functions to access the Condor high performance computing environment. Files are first uploaded to a submitter machine and the resulting job is then passed on to Condor. Look here for the code.

GPUmatrix v0.1.0: Extends R to use GPUs for matrix computations. See the vignette.

hydroMOPSO v0.1-3: Implements a state-of-the-art Multi-Objective Particle Swarm Optimiser (MOPSO), based on the algorithm developed by Lin et al. (2018) with improvements described by Marinao-Rivas & Zambrano-Bigiarini (2020) which can be used for global optimization of non-smooth and non-linear R functions and other models that need to be run from the system console, e.g. SWAT+.

Data

dataverifyr v0.1.5: Provides a thin wrapper around dplyr, data.table, arrow, and DBI to allow users to define rules which can be used to verify a given dataset. See README to get started.

neotoma2 v1.0.0: Provides functions to access and manipulate data in the Neotoma Paleoecology Database. See the vignette.

rpaleoclim v1.0.0: Implements an interface to PaleoClim, a set of free, high resolution paleoclimate surfaces covering the whole globe that includes data on surface temperature, precipitation and the standard bioclimatic variables commonly used in ecological modelling. See Brown et al. (2019) for background and the vignette.

zctaCrosswalk v2.0.0: Contains the US Census Bureau’s 2020 ZCTA to County Relationship File, as well as convenience functions to translate between States, Counties and ZIP Code Tabulation Areas (ZCTAs). See the Introduction and the vignettes Workflow with tidycensus, and Developer Notes.

Ecology

EWSmethods v1.1.2: Implements methods for forecasting tipping points at the community level that include rolling and expanding window approaches to assessing abundance based early warning signals, non-equilibrium resilience measures, and machine learning. See Dakos et al. (2012), Deb et al. (2022), and Drake and Griffen (2010) for background and the vignette for an introduction.

fqacalc v1.0.0: Provides functions for calculating Floristic Quality Assessment (FQA) metrics using regional FQA databases that have been approved or approved with reservations as ecological planning models by the U.S. Army Corps of Engineers (USACE). For information on FQA see Spyreas (2019). There is an Introduction.

Economics

clptheory v0.1.0: Provides functions to compute the uniform rate of profit, the vector of price of production and the vector of labor values, and also compute measures of deviation between relative prices of production and relative values. See Basu and Moraltis (2023) for background and README for an introduction.

Genomics

BREADR v1.0.1: Implements a method for estimating degrees of relatedness for extreme low-coverage genotype data and includes functions to quantify and visualize the level of confidence in the estimated degrees of relatedness. See Rohrlach et al. (2023) for package details and README for examples.

crosshap v1.2.2: Implements a local haplotyping visualization toolbox to capture major patterns of co-inheritance between clusters of linked variants, while connecting findings to phenotypic and demographic traits across individuals. See Marsh et al. (2022) for a detailed example and README for an introduction.

DAISIEprep v0.3.2: Extracts colonization and branching times of island species for analysis with the DAISIE package. There is a Tutorial and there are vignettes on Performance and Sensitivity.

Machine Learning

CCMMR v0.1: Implements the convex clustering through majorization-minimization algorithm described in Touw, Groenen, and Terada (2022) to minimize the convex clustering loss function. See README for examples.

rcccd v0.3.2: Provides functions to fit class cover catch digraph classification models. Methods are explained in Priebe et al. (2001), Priebe et al. (2003), and Manukyan and Ceyhan (2016). README contains some description.

TheOpenAir v0.1.0: Implements a wrapper using the OpenAI API as a back end to integrate ChatGPTinto diverse data-related tasks, such as data cleansing and automating analytics scripts. See README to get started.

Mathematics

cyclotomic v1.1.0: Implements algorithms from the GAP project to work with cyclotomic numbers: complex numbers that can be thought of as the rational numbers extended with the roots of unity. They have applications in number theory, algebraic geometry, algebraic number theory, coding theory, in the theory of graphs and combinatorics, and in the theory of modular functions and modular curves. See README for examples.

markovmix v0.1.1: Provides functions to fit a mixture of Markov chains of higher orders from multiple sequences along with various utility functions to derive transition patterns, transition probabilities per component and component priors. See README for examples.

Medicine

DiDforBigData v1.0: Provides a big-data-friendly and memory-efficient difference-in-differences estimator for staggered (and non-staggered) treatment contexts. See the Get Started Guide the vignettes Background, Examples, and Theory.

predictNNB v0.1.0: Provides tools to estimate when and where a model-guided treatment strategy may outperform a treat-all or treat-none approach using Monte Carlo simulation and evaluation of the Net Monetary Benefit. See Parsons et al. (2023) for details, the Introduction, and the vignettes on creating functions, summarising results, and detailed example.

predRupdate v0.1.0: Provides functions to evaluate the predictive performance of existing clinical prediction model given a new dataset. See Su et al. (2018), Debray et al. (2014), and Martin et al. (2018) for background and the vignettes Introduction and Technical Background.

SPARRAfairness v0.0.0.1: Provides functions to analyse the behavior and performance of the Scottish Patients At Risk of admission and Re-Admission risk score which estimates yearly risk of emergency hospital admission using electronic health records for most of the Scottish population. Analysis focuses on differential performance over demographically-defined groups. See the vignette.

Science

kronos v1.0.0: Implements a framework to analyse circadian or otherwise rhythmic data using the familiar R linear modelling syntax, while taking care of the trigonometry under the hood. Look here for examples.

mpmsim v1.0.0: Provides functions to to simulate matrix population models with particular characteristics based on aspects of life history such as mortality trajectories and fertility trajectories, and allows the exploration of sampling error due to small sample size. See the vignettes on robustness, sampling error & propagation, and PCA.

Statistics

BGFD v0.1: Implements the probability density function, cumulative distribution function, quantile function, random numbers, survival function, hazard rate function, and maximum likelihood estimates for the family of Bell-G and Complementary Bell-G distributions. See Fayomi et al. (2022), Alanzi et al.(2023), and Algarni (2022) for details.

D3mirt v1.0.3: Provides functions for identifying, estimating, and plotting descriptive multidimensional item response theory models, restricted to 3D and dichotomous or polytomous data that fit the two-parameter logistic model or the graded response model. See the vignette for an extensive introduction.

funStatTest v1.0.2: Implements two sample comparison procedures based on median-based statistical tests for functional data, described in Smida et al. (2022), Chakraborty and Chaudhuri (2015), Horvath et al. (2013, and Cuevas et al. (2004). See the vignette for examples.

lessSEM v1.4.16: Provides regularized structural equation modeling (regularized SEM) with non-smooth penalty functions (e.g., lasso) building on lavaan. There are nine vignettes including: lessSEM, The Structural Equation Model, and Mixed Penalties.

panelhetero v1.0.0: Provides tools for estimating the degree of heterogeneity across cross-sectional units in the panel data analysis using the methods developed by Okui and Yanagi (2019) and Okui and Yanagi (2020). See the vignette.

tdsa v1.0-1: Provides functions to perform time-dependent sensitivity analysis by calculating time-dependent state and parameter sensitivities for both continuous- and discrete-time deterministic models. See Ng et al. (in review) for background and the vignette to get started.

Utilities

crew.cluster v0.1.0: Extends the mirai-powered crew package with worker launcher plugins for traditional high-performance computing systems to enable statisticians and data scientists to asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. Look here to get started.

duke v0.0.1: Provides functions to generate visualizations with Duke’s official suite of colors in a color blind friendly way. There is an Overview and four additional vignettes including one on the theme_duke() function.

grateful v0.2.0: Facilitates the citation of R packages used in analysis projects by providing functions to scan projects for packages used and produces documents with citations in the preferred bibliography format. Functions may be used within rarkdownor quarto documents. See README for examples.

hightR v0.3.0: Implements the HIGHT block cipher encryption algorithm developed to provide confidentiality in low power consumption computing environments such Radio-Frequency Identification and Ubiquitous Sensor Network. Look here for more information.

myCRAN v1.0: Provides functions to plot the daily and cumulative number of downloads of R packages, obtaining daily and cumulative counts in one run. See the vignette.

woodendesc v0.1.0: Provides functions to simplify obtaining available packages, their version codes and dependencies from any R repository. Uses extensive caching for repeated queries. See READMEfor examples.

Visualization

fxl v1.6.3: Provides functions to prepare and design single case design figures that are typically prepared in spreadsheet software. See the vignette for theory and examples.

ggragged v0.1.0: Extends ggplot2 facets to panel layouts arranged in a grid with ragged edges with rows and columns of potentially varying lengths. These may be useful in representing nested or partially crossed relationships between faceting variables. See README for examples.

nndiagram v1.0.0: Generates LaTeX code for drawing well-formatted neural network diagrams with TikZ. Users define the number of neurons on each layer, neuron connections to keep or omit, layers considered to be oversized, and neurons to draw with lighter color. See README for instructions.

PlotTools v0.2.0: Provides functions to manipulate irregular polygons and annotate plots with legends for continuous variables and color spectra using the base graphics plotting tools. See README for an example.

March 2023: "Top 40" New CRAN Packages

Fri, 28 Apr 2023 00:00:00 +0000

Accounting

debkeepr v0.1.1: Provides tools to analyze historical, non-decimal currencies and value systems that use tripartite or tetrapartite systems such as pounds and shillings in the context of double-entry bookkeeping. See the Getting Started guide and the vignettes Analysis of Richard Dafforne’s Journal and Ledger and Transactions in Richard Dafforne’s Journal.

Computational Methods

ABM v0.3: Implements a high-performance, flexible and extensible framework to develop continuous-time agent based models capable of simulating millions of agents in which state transitions may be either spontaneous or caused by agent interactions. See README for multiple examples including Simulate an agent based SEIR model and Simulate contact tracing on an SIR model.

rvMF v0.0.7: Provides functions to generate pseudo-random vectors that follow an arbitrary von Mises-Fisher distribution on a sphere including functions to generate random variates, compute the density for the distribution of an inner product between von Mises-Fisher random vector and its mean direction. Look here for an example.

Data

FertNet v0.1.1: Provides tools to processes data from The Social Networks and Fertility Survey including functions for correcting respondent errors and for transforming network data into network objects to facilitate analyses and visualization. See README to get started.

oldbailey v1.0.0: Provides functions to fetch trial data from the Old Bailey Online API. Data includes the names of the first person speakers, defendants, victims, their recorded genders, verdicts, punishments, crime locations, and dates. Look here for an example.

webtrackR v0.0.1: Implements data structures and methods to work with web tracking data, including data preprocessing steps, methods to construct audience networks as described in Mangold & Scharkow (2020) and metrics of news audience polarization described in Mangold & Scharkow (2022). Look here to get started.

Ecology

GIFT v1.0.0: Provides functions to retrieve regional plant checklists, species traits and distributions, and environmental data from the Global Inventory of Floras and Traits database and to visualize the map of available flora. There is an introductory Tutorial, an Advanced Tutorial, and a vignette on Queries.

rTLsDeep v0.0.5: Uses terrestrial laser scanning and deep learning to classify post-hurricane damage severity at the individual tree level. See Klauberg et al. (2023) for details, and look here for an example.

Finance

HSRFA v0.1.1: Implements two algorithms to do robust factor analysis by considering the Huber loss: one is based on minimizing the Huber loss of the idiosyncratic error’s L2 norm, the other is based on minimizing the element-wise Huber loss. See He et al. (2023) for background, Bai (2003) for PCA code, and He et al. (2022), and Chen et al. (2021) for the Quantile Factor Analysis method.

PCRA v1.0: Provides a collection of functions and several real-world data sets that support teaching a quantitative finance MS level course on Portfolio Construction and Risk Analysis. See the vignette: Introduction to CRSP Stocks and SPGMI Factors in PCRA.

Genomics

GESciLiVis v1.1.0: Provides tools to visualize publication activity per gene based on a gene list and a user-defined set of keywords to perform an NCBI database search as in PubMed. See the vignette.

ggpicrust2 v1.6.0: Provides tools to analyze and visualize PICRUSt2 output with pre-defined plots and functions, including a one-click option for creating publication-level plots. For more details, see Yang et al. (2023). Look here for examples.

gsdensity v0.1.2: Implements a computational tool for pathway centric analysis of single-cell data including scRNA-seq data and spatial genomics data. Given a gene set and a cell-by-gene matrix, ask the question: is this gene set somehow enriched by a subpopulation of the cells? See README for examples.

metaGE v1.0.0: Provides tools for conducting genome-wide association studies for studying Genotype x Environment interactions, including functions to collect the results of GWAS data from different files, infer the inter-environment correlation matrix, perform global test procedure for quantitative trait loci detection, and perform tests of contrast or meta-regression. See De Walsche et al. (2023) for the details.

Machine Learning

FACT v0.1.0: Implements an algorithm agnostic framework for feature attribution while preserving the integrity of the data and facilitating the understand of the mapping procedure of an algorithm that assigns instances to clusters. See README to get started.

lpda v1.0.1: Implements the linear programming classification method described by Nueda, et al. (2022) which is advantageous when variable distributions are unknown or when the number of variables is much greater than the number of observations. See the vignette.

UBayFS v1.0: Implements the user-guided Bayesian framework proposed by Jenul et al. (2022) for ensemble feature selection. See the Introduction and the vignette on Block feature selection.

Mathematics

qfratio v1.0.1: Provides functions to evaluate moments of ratios and products of quadratic forms in normal variables using recursive algorithms developed by Bao and Kan (2013) and and Hillier et al. (2014). See README for examples.

Medicine

gsDesign2 v1.0.7: Provides tools to enable fixed or group sequential design under non-proportional hazards assumptions that support flexible enrollment, time-to-event and time-to-dropout assumptions. Design methods include average hazard ratio, the weighted logrank tests in Yung and Liu (2019), and MaxCombo tests. See the vignette to get started.

NCC v1.0: Provides functions to simulate and analyze platform trials with non-concurrent controls. See Bofill Roig et al. (2022), Saville et al. (2022), and Schmidli et al. (2014) for background. There is a brief Introduction and there are vignettes on simulating binary data, continuous data, and How to run a simulation study.

Pharma

DrugExposureDiagnostics v0.4.1: Provides ingredient specific diagnostics for drug exposure records in the Observational Medical Outcomes Partnership (OMOP) common data model. See the Introduction and the Summary of checks vignette.

rlistings v0.1.1: Provides functions to create and display listings for clinical trials. See the Getting Started Guide.

Science

LCMSQA v1.0.0: Provides functions to check the quality of liquid chromatograph/mass spectrometry (LC/MS) experiments using an interactive shiny application. Tests include total ion current chromatogram, base peak chromatogram, mass spectrum, and extracted ion chromatogram. See the Introduction.

Statistics

lmw v0.0.1: Provides functions to compute the implied weights of linear regression models for estimating average causal effects and provides diagnostics based on these weights. See Chattopadhyay and Zubizarreta (2022) where several regression estimators are represented as weighting estimators, in connection with inverse probability weighting. Look here for examples.

ptable v1.0.0: Implements the cell-key statistical disclosure control perturbation technique to protect confidential information. See Giessing (2016) for the technical details and the vignette for examples.

snha v0.1.3: Implements the St. Nicolas House Analysis to explore interacting variables and create correlation networks. See the vignette.

satdad v1.1: Implements theoretical and non-parametric tools to analyze tail dependence in sample based or theoretical models. A goal is to generate multivariate extreme value models in any dimension. See the extensive vignette.

sr v0.1.0: Implements the Gamma test based smooth regression method for measuring smoothness in multivariate relationships, finding causal connections in precision data, finding lags and embeddings in time series, and training neural networks. See Evans & Jones (2002) and Jones (2004) for details and the vignette for examples.

wqspt v1.0.1: Implements a permutation test method for the weighted quantile sum (WQS) regression used to evaluate the effect of complex exposure mixtures on an outcome. See Carrico et al. (2015) and Day et al. (2022) for the theory and the vignette for examples.

Time Series

coconots v1.1.1: Provides tools for fitting, validating, and forecasting practical convolution-closed time series models for low counts. The models are described in Jung and Tremayne (2011), and the model assessment tools are presented in Czado et al. (2009), Gneiting and Raftery (2007), and, Tsay (1992). See README for examples.

sparseDFM v1.0: Implements various estimation methods for dynamic factor models (DFMs) including PCA, see Stock and Watson (2002), and EM, see Banbura and Modugno (2014) and DFMs Mosley et al. (2023). There are vignettes on Nowcasting UK Trade in Goods and Inflation.

Utilities

askgpt v0.0.2: Implements a connection to the OpenAI API to answer questions about R. See the vignette.

cellKey v1.0.1: Implements a method to protect statistical data by computing cell keys for individual cells in statistical tables. The theory behind the method is described in Thompson, Broadfoot and Elazar (2013) and Giessing and Tent (2019).

occupationMeasurement v0.2.0: Implements an interface for performing interactive occupation coding during interviews as described in Peycheva et al. (2021) and Schierholz et al. (2018). There are several vignettes including a Getting Started guide, Using the API, and Custom Questionnaires.

pracpac v0.1.0: Provides functions to streamline the creation of Docker images with R packages and dependencies embedded. See Nagraj and Turner (2023) for details and the vignettes Basic usage and Use cases.

RmdConcord v0.1.6: Supports concordances in R Markdown documents to easily find the source in the .Rmd file of errors detected by HTML tidy. See README for details and note that the vignette serves as a practice file.

symbol.equation.gpt v1.1.1: Provides an interface for adding symbols, smileys, arrows, and building mathematical equations using LaTeX or r2symbols for Markdown and Shiny development. See the vignette.

tinysnapshot v0.0.3: Provides snapshots for unit tests using the tinytest framework and includes expectations to test base R and ggplot2 plots as well as console output from print(). See README for usage.

Visualization

PlotBivInvGaus v0.1.0: Provides functions to create bivariate inverse Gaussian distribution contour plots for non-negative random variables. See the vignette.

textBoxplacement v1.0: Provides functions to compute a non-overlapping layout of text boxes to label multiple overlaying curves. See the vignette.

Multistate Models for Medical Applications

Wed, 19 Apr 2023 00:00:00 +0000

Clinical research studies and healthcare economics studies are frequently concerned with assessing the prognosis for survival in circumstances where patients suffer from a disease that progresses from state to state. Standard survival models only directly model two states: alive and dead. Multi-state models enable directly modeling disease progression where patients are observed to be in various states of health or disease at random intervals, but for which, except for death, the times of entering or leaving states are unknown. Multi-state models easily accommodate interval censored intermediate states while making the usual assumption that death times are known but may be right censored.

A natural way to conceptualize modeling the dynamics of disease progression with interval censored states is as continuous time Markov chains. The following diagram illustrates a possible disease progression model where there is some possibility of dying from any state, but otherwise a patient would progress from being healthy, to mild disease, to severe disease and then perhaps death.

## Loading required package: shape

It is true that in that the Markov assumption implies that the time patients spend in the various states are exponentially distributed. However, the mathematical theory of stochastic multi-state processes is very rich and can accommodate more realistic models with state dependent hazard rates that vary over time and other relaxations of the Markov assumption. Moreover, there is robust software in R (and other languages) that make multi-state stochastic survival models practical.

In the remainder of this post, I present a variation of a disease progression model discussed by Ardo van den Hout in some detail in his incredibly informative and very readable monograph Multi-State Survival Models for Interval Censored Data . Also note that van den Hout’s model is itself an elaboration of the main example presented by Christopher Jackson in the vignette to his msmpackage. This post presents a slower development of the model developed by Jackson and van den Hout that might be easier for a person not already familiar with the msm package to follow.

library(tidyverse)
library(tidymodels)
library(msm)

The Data

The data set explored by both Jackson and van den Hout is the Cardiac Allograft Vasculopathy (CAV) data set which contains the individual histories of angiographic examinations of 622 heart transplant recipients collected at the Papworth Hospital in the United Kingdom. This data is included in the msm package and is a good candidate to be the iris dataset for progressive disease models. It is a rich data set with 2846 rows and multiple covariates, including patient age and time time since transplant, both of which can be use for time scales, multiple state transitions among four states and no missing values. Observations of intermediate states are interval censored and have been recorded varying time intervals. Deaths are “exact” or right censored.

The following code creates a new variable that preserves the original state data for each observation and displays the data in tibble format.

set.seed(1234)
df <- tibble(cav) %>% mutate(o_state = state)

df

## # A tibble: 2,846 × 11
##     PTNUM   age years  dage   sex pdiag cumrej state firstobs statemax o_state
##     <int> <dbl> <dbl> <int> <int> <fct>  <int> <int>    <int>    <dbl>   <int>
##  1 100002  52.5  0       21     0 IHD        0     1        1        1       1
##  2 100002  53.5  1.00    21     0 IHD        2     1        0        1       1
##  3 100002  54.5  2.00    21     0 IHD        2     2        0        2       2
##  4 100002  55.6  3.09    21     0 IHD        2     2        0        2       2
##  5 100002  56.5  4       21     0 IHD        3     2        0        2       2
##  6 100002  57.5  5.00    21     0 IHD        3     3        0        3       3
##  7 100002  58.4  5.85    21     0 IHD        3     4        0        4       4
##  8 100003  29.5  0       17     0 IHD        0     1        1        1       1
##  9 100003  30.7  1.19    17     0 IHD        1     1        0        1       1
## 10 100003  31.5  2.01    17     0 IHD        1     3        0        3       3
## # ℹ 2,836 more rows

The state table which presents the number of times each pair of states were observed in successive observation times shows that 46 transitions from state 2 (Mild CAV) to state 1 (No CAV), 4 transitions from state 3 (Severe CAV) to Healthy and 13 transitions from Severe CAV to Mild CAV.

statetable.msm(state = state, subject = PTNUM, data = df)

##     to
## from    1    2    3    4
##    1 1367  204   44  148
##    2   46  134   54   48
##    3    4   13  107   55

I will follow van den Hout and assume these backward transitions are misclassified and alter the state variable so there is no back sliding. The following code does this in a tidy way and also creates a new variable b_age which records the baseline age at which patients entered the study. (Note: you can find van den Hout’s code here)

df1 <- df %>% group_by(PTNUM) %>% 
                     mutate(b_age = min(age),
                            state = cummax(state)
                     )

This transformation will make the state transition table conform to the diagram above, but with state 1 representing No CAV rather than Health.

statetable.msm(state = state, subject = PTNUM, data = df1)

##     to
## from    1    2    3    4
##    1 1336  185   40  139
##    2    0  220   52   49
##    3    0    0  140   63

Setting Up and Running the Model

The next step is to set up the model using the function msm() whose great flexibility means that some care must be taken to set parameter values.

First, we set up the initial guess for the intensity matrix, Q, which determines the transition rates among states for a continuous time Markov chain. For the msm function, positive values indicate possible transitions.

# Intensity matrix Q:
q <- 0.01
Q <- rbind(c(0,q,0,q), c(0,0,q,q),c(0,0,0,q),c(0,0,0,0))
qnames <- c("q12","q14","q23","q24","q34")

Next, we set up the covariate structure which van den Hout discusses in his monograph, but does not show in the code on the book’s website referenced above. For this model, transitions from state 1 to state 2 and from state 1 to state 4 depend on time,years, the age of the patient at transplant time b_age, and dage, the age of the donor. The other transitions depend only on dage. So, we see that msm() can deal with time varying covariates as well as permitting individual state transitions to be driven by different covariates.

covariates = list("1-2" = ~ years + b_age + dage , 
                  "1-4" = ~ years + b_age + dage ,
                  "2-3" = ~ dage,
                  "2-4" = ~ dage,
                  "3-4" = ~ dage)

Now, we set the remaining parameters for the model.

obstype <- 1
center <- FALSE
deathexact <- TRUE
method <- "BFGS"
control <- list(trace = 0, REPORT = 1)

obstype = 1 indicates that observations have been taken at arbitrary time points. They are snapshots of the process that are common for panel data.

center = FALSE means that covariates will not be centered at their means during the maximum likelihood estimation process. The default for this parameter is TRUE.

deathexact = TRUE indicates that the final absorbing state is exactly observed. This is the defining assumption survival data. In msm this is equivalent to setting obstupe = 3 for state 4, our absorbing state.

** method = BFGS** signals optim() to use the optimization method published simultaneously in 1970 by Broyden, Fletcher, Goldfarb and Shanno. (look here). This is a quasi-Newton method that uses function values and gradients to build up a picture of the surface to be optimized.

control = list(trace=0,REPORT=1) indicates more parameters that will be passed to optim().

REPORT sets the the frequency of reports for the “BFGS”, “L-BFGS-B” and “SANN” methods if control$trace is positive. Defaults to every 10 iterations for “BFGS” and “L-BFGS-B”, or every 100 temperatures for “SANN”. (Note: SANN is a variant of the simulated annealing method presented by C. J. P. Belisle (1992) Convergence theorems for a class of simulated annealing algorithms on R^d Journal of Applied Probability.)

trace is also passed tooptim(). trace must be a non-negative integer. If positive, tracing information on the progress of the optimization is produced. Higher values may produce more tracing information.

model_1 <- msm(state~years, subject = PTNUM, data = df1, center= center, 
             qmatrix=Q, obstype = obstype, deathexact = deathexact, method = method,
             covariates = covariates, control = control)

First check to see if the model has converged. For the BFGS method, possible convergence codes returned by optim() are: 0 indicates convergence, 1 indicates that the maximum iteration limit has been reached, 51 and 52 indicate warnings.

#Model Status
conv <- model_1$opt$convergence; cat("Convergence code =", conv,"\n")

## Convergence code = 0

Next, look at a measure of how well the model fits the data proposed by using a visual test proposed by Gentleman et al. (1994) which plots the observed numbers of individuals occupying a state at a series of times against forecasts from the fitted model, for each state. The msm function plot.prevalence.msm() produces a perfectly adequate base R plot. However, to emphasize that msm users are not limited to base R plots, I’ll do a little extra work to use ggplot(). When a package author is kind enough to provide an extractor function you can do anything you want with the data.

The prevalence.msm() function extracts both the observed and forecast prevalence matrices.

prev <- prevalence.msm(model_1)

This not very elegant, but straightforward code reshapes the data and plots.

# reshape observed prevalence
do1 <-as_tibble(row.names(prev$Observed)) %>% rename(time = value) %>% 
          mutate(time = as.numeric(time))
do2 <-as_tibble(prev$Observed) %>% mutate(type = "observed")
do <- cbind(do1,do2) %>% select(-Total)
do_l <- do %>% gather(state, number, -time, -type)
# reshape expected prevalence
de1 <-as_tibble(row.names(prev$Expected)) %>% rename(time = value) %>% 
          mutate(time = as.numeric(time))
de2 <-as_tibble(prev$Expected) %>% mutate(type = "expected")
de <- cbind(de1,de2) %>% select(-Total) 
de_l <- de %>% gather(state, number, -time, -type) 

# bind into a single data frame
prev_l <-rbind(do_l,de_l) %>% mutate(type = factor(type),
                                     state = factor(state),
                                     time = round(time,3))


# plot for comparison
prev_gp <- prev_l %>% group_by(state)
pp <- prev_l %>% ggplot() +
     geom_line(aes(time, number, color = type)) +
     xlab("time") +
     ylab("") +
     ggtitle("")
pp + facet_wrap(~state)

The agreement of the observed and forecast prevalence for states 1 through 3 look pretty good. After about 8 years the observed deaths are notably higher than the forecast. As Jackson points out (See the msm Manual page 33), this kind of discrepancy could indicate that the underlying process is not homogeneous. I have attempted to capture this non-homogeneity by having some of the transitions depend on time. And, although the plot above looks a little better that than the plot in the manual, which does not attempt to model non-homogeneity, it is apparent that there is room to find a better model!

Survival Curves and Calculations

Now, we can jump straight to the major result and look at the fitted survival curves. There is a plot() method for msm that will directly plot these curves. However, just to emphasize that if a package author is kind enough to provide a plot method, it will probably not be too difficult to hack the code for the method to use an alternative plotting system. To save space, I will not show my code, but you can easily recreate it by stating with the plot.msm() function, deleting the plotting parts and returning the values for time and the states “Health, Mild_CAV, and Severe_CAV which are used int the code below. Check my hack by running plot(model_1)

# plot_prep was obtained from plot.msm()
res <- plot_prep(model_1)
time <- res[[1]]
Health <- res[[2]]
Mild_CAV <- res[[3]]
Severe_CAV <- res[[4]]
df_w <- tibble(time,Health, Mild_CAV, Severe_CAV)
df_l <- df_w %>% gather("state", "survival", -time)
p <- df_l %>% ggplot(aes(time, 1 - survival, group = state)) +
     geom_line(aes(color = state)) +
     xlab("Years") +
     ylab("Probability") +
     ggtitle("Fitted Survival Probabilities")
p

These curves indicate that a treatment that could prevent CAV or at least delay progression from mild CAV to severe CAV might prolong survival. Additionally, the Markov structure of the model permits extracting information that relates to disease progression and the total time spent in each state.

The function totlos.msm() estimates the total expected time that a patient will spend in each state. Parameter settings for this function include:

start = c(1,0,0,0) specifies that patients will start in state 1 with probability 0.

fromt = 0 indicates starting at the beginning of the process.

covariates = “mean” indicates that the covariates will be set to their mean values for the calculation.

total_state_time <-totlos.msm(model_1,start = c(1,0,0,0), from = 0, covariates = "mean")
total_state_time

## State 1 State 2 State 3 State 4 
##   7.002   2.473   1.621     Inf

The table indicates that the mean time a patient can expect to avoid CAV is about 7 years. After progressing to a Mild_CAV, a patient can expect five additional years.

A more direct calculation based on the intensity matrix, Q, give the expected time to the “absorbing” state, Death, from each of the “transient” living states.

time_to_death <- efpt.msm(model_1, tostate = 4, covariates = "mean")
time_to_death

## [1] 11.097  5.836  3.005  0.000

This agrees with the total state times calculated above.

sum(total_state_time[1:3])

## [1] 11.1

Within the scope of the information provided by the covariates, it is also possible to generate more individualized forecasts. For example, here is the expected time to death for a person starting off with no CAV at age 60, who received a heart from a 20 year old donor, 5 years after the transplant.

efpt.msm(model_1, tostate = 4, start = c(1,0,0,0), covariates = list(years = 5, b_age = 60, dage = 20))

##       [,1]
## [1,] 7.953

A related quantity, mean sojourn time, is the mean time that each visit to each state is expected to last. Since, we are assuming a progressive disease model where each patient visits each state only once, the estimate should be close to total time spent in each state. However, Jackson notes that in a progressive model, sojourn time in the disease state will be greater than the expected length of stay in the disease state because the mean sojourn time in a state is conditional on entering the state, whereas the expected total time in a diseased state is a forecast for an individual, who may die before getting the disease. (See help(totlos.msm)). And indeed, that is what we see here for states 2 and 3.

sojourn.msm(model_1, covariates="mean")

##         estimates     SE     L     U
## State 1     7.002 0.4024 6.256 7.837
## State 2     3.525 0.3020 2.980 4.169
## State 3     3.005 0.3748 2.353 3.837

Hazard Ratios

model_1 will also product estimates of hazard ratios which show the estimate effect on transition intensities for each state.

Here is the table of Hazard Ratios:

model_1

## 
## Call:
## msm(formula = state ~ years, subject = PTNUM, data = df1, qmatrix = Q,     obstype = obstype, covariates = covariates, deathexact = deathexact,     center = center, method = method, control = control)
## 
## Maximum likelihood estimates
## Baselines are with covariates set to 0
## 
## Transition intensities with hazard ratios for each covariate
##                   Baseline                         years              
## State 1 - State 1 -0.032750 (-0.0607897,-0.017644)                    
## State 1 - State 2  0.030957 ( 0.0160470, 0.059721) 1.112 (1.061,1.166)
## State 1 - State 4  0.001793 ( 0.0004703, 0.006836) 1.093 (1.012,1.182)
## State 2 - State 2 -0.395633 (-0.6488723,-0.241227)                    
## State 2 - State 3  0.264310 ( 0.1488153, 0.469441) 1.000              
## State 2 - State 4  0.131323 ( 0.0330133, 0.522385) 1.000              
## State 3 - State 3 -0.434548 (-0.9113857,-0.207192)                    
## State 3 - State 4  0.434548 ( 0.2071918, 0.911386) 1.000              
##                   b_age                dage                 
## State 1 - State 1                                           
## State 1 - State 2 1.001 (0.9884,1.014) 1.0281 (1.0159,1.040)
## State 1 - State 4 1.053 (1.0271,1.079) 1.0208 (1.0039,1.038)
## State 2 - State 2                                           
## State 2 - State 3 1.000                0.9932 (0.9756,1.011)
## State 2 - State 4 1.000                0.9757 (0.9298,1.024)
## State 3 - State 3                                           
## State 3 - State 4 1.000                0.9906 (0.9672,1.015)
## 
## -2 * log-likelihood:  3466

The table shows that time, the covariate years, affects disease progression represented by the the transition from state 1 to state 2, but has a smaller effect on the transition from state 1 to state 4.

The covariate b_age, the baseline age of patient at transplant time has a larger effect on dying before the onset of CAV than on the transition to CAV.

The covariate dage has a minor effect on the transitions from state 1 but apparently has no effect thereafter.

The hazard ratios are computed by calculating the exponential of the estimated covariate effects on the log-transition intensities for the Markov process which are stored in the model object.

To see how these work, first look at the baseline hazard ratios.

model_1$Qmatrices$baseline

##          State 1  State 2 State 3  State 4
## State 1 -0.03275  0.03096  0.0000 0.001793
## State 2  0.00000 -0.39563  0.2643 0.131323
## State 3  0.00000  0.00000 -0.4345 0.434548
## State 4  0.00000  0.00000  0.0000 0.000000

These baseline hazard ratios are computed from the model intensity matrix, Q, assuming no covariates. They can also be directly extracted from the model by qmatrix.msm() extractor function with covariates set to zero.

qmatrix.msm(model_1,  covariates = 0)

##         State 1                          State 2                         
## State 1 -0.032750 (-0.0607897,-0.017644)  0.030957 ( 0.0160470, 0.059721)
## State 2 0                                -0.395633 (-0.6488723,-0.241227)
## State 3 0                                0                               
## State 4 0                                0                               
##         State 3                          State 4                         
## State 1 0                                 0.001793 ( 0.0004703, 0.006836)
## State 2  0.264310 ( 0.1488153, 0.469441)  0.131323 ( 0.0330133, 0.522385)
## State 3 -0.434548 (-0.9113857,-0.207192)  0.434548 ( 0.2071918, 0.911386)
## State 4 0                                0

The 95% confidence limits are computed by assuming normality of the log-effect.

A more representative value for the intensity matrix for this model can be obtained by setting the covariates to their expected mean values.

qmatrix.msm(model_1,  covariates = "mean")

##         State 1                      State 2                     
## State 1 -0.14281 (-0.15984,-0.12760)  0.10019 ( 0.08742, 0.11482)
## State 2 0                            -0.28369 (-0.33556,-0.23984)
## State 3 0                            0                           
## State 4 0                            0                           
##         State 3                      State 4                     
## State 1 0                             0.04262 ( 0.03339, 0.05441)
## State 2  0.21821 ( 0.17636, 0.26999)  0.06548 ( 0.03872, 0.11075)
## State 3 -0.33281 (-0.42498,-0.26064)  0.33281 ( 0.26064, 0.42498)
## State 4 0                            0

Next, we may want to examine the contribution of the covariate covariates to the hazard ratios. To take a particular example, look at the dage to the hazard ratios

model_1$Qmatrices$dage

##         State 1 State 2   State 3   State 4
## State 1       0 0.02771  0.000000  0.020575
## State 2       0 0.00000 -0.006783 -0.024625
## State 3       0 0.00000  0.000000 -0.009439
## State 4       0 0.00000  0.000000  0.000000

and focus on the contribution of dage to the intensity matrix for the transition from state 3 to state 4 which is given as -0.009439 in the table above. Taking the exponential of this value, yields the hazard ratio for the dage state 3 to 4 transition in the hazard ratio’s table we got by printing out model_1 above.

exp(-.009439)

## [1] 0.9906

The hazard ratio tables for the remaining covariates are given by:

model_1$Qmatrices$years

##         State 1 State 2 State 3 State 4
## State 1       0  0.1064       0 0.08933
## State 2       0  0.0000       0 0.00000
## State 3       0  0.0000       0 0.00000
## State 4       0  0.0000       0 0.00000

and

model_1$Qmatrices$b_age

##         State 1   State 2 State 3 State 4
## State 1       0 0.0009645       0 0.05152
## State 2       0 0.0000000       0 0.00000
## State 3       0 0.0000000       0 0.00000
## State 4       0 0.0000000       0 0.00000

Exploring Transition Probabilities and Intensities

It is also possible to look at the state transition matrix at different times and see how these probabilities change over time. Here we compute the transition matrix at 1 year.

pmatrix.msm(model_1, t = 1, covariates = "mean")

##         State 1 State 2  State 3 State 4
## State 1  0.8669 0.08101 0.008493 0.04357
## State 2  0.0000 0.75300 0.160340 0.08666
## State 3  0.0000 0.00000 0.716903 0.28310
## State 4  0.0000 0.00000 0.000000 1.00000

and at 5 years.

pmatrix.msm(model_1, t = 5, covariates = "mean")

##         State 1 State 2 State 3 State 4
## State 1  0.4897  0.1761 0.07871  0.2556
## State 2  0.0000  0.2421 0.23419  0.5237
## State 3  0.0000  0.0000 0.18937  0.8106
## State 4  0.0000  0.0000 0.00000  1.0000

Additionally, it is possible to examine the effect of covariates on transition probabilities. Here are the 5 year transition probabilities for a patient with a baseline age of 35.

pmatrix.msm(model_1, t = 5, covariates = list(years = 5, b_age = 35, dage = 20))

##         State 1 State 2 State 3 State 4
## State 1  0.5474  0.1674 0.07575  0.2095
## State 2  0.0000  0.2112 0.21622  0.5726
## State 3  0.0000  0.0000 0.16547  0.8345
## State 4  0.0000  0.0000 0.00000  1.0000

and those who had the procedure at age 60.

pmatrix.msm(model_1, t = 5, covariates = list(years = 5, b_age = 60, dage = 20))

##         State 1 State 2 State 3 State 4
## State 1  0.3863  0.1409 0.06784  0.4050
## State 2  0.0000  0.2112 0.21622  0.5726
## State 3  0.0000  0.0000 0.16547  0.8345
## State 4  0.0000  0.0000 0.00000  1.0000

Note that the transitions from CAV states are unaffected.

To summarize: Continuous Time Markov Chains provide a natural framework for working with multi-state survival models. The msm package is sufficiently sophisticated to permit modeling clinical process with level of fidelity that may provide insight about clinically observed disease progression. The software is relatively easy to use and there is plenty of documentation to help you get started.

Learning More About Multi-State Survival Models

To dive deeper into multi-state survival models, I am sure you will find Ardo van den Hout’ Multi-State Survival Models for Interval-Censored Data extraordinarily helpful. There are many good textbooks about the basics of Continuous Time Markov Chains. I recommend J.R.Norris’ - Markov Chains which is still modestly priced. There are also many expositions freely available on the internet including:

David F. Anderson - Chapter 6: Continuous Time Markov Chains from Lecture Notes on Stochastic Processes with Applications in Biology
Miranda Holmes-Cerfon - Lecture 4: Continuous-time Markov Chains
Søren Feodor Nielsen - Continuous-time homogeneous Markov chains
Karl Sigman - Continuous-Time Markov Chains

A data analyst workflow, part 1: SQL & tidyverse

Thu, 06 Apr 2023 00:00:00 +0000

Vidisha Vachharajani works in the EdTech industry, where she enjoys developing data-driven strategy solutions for learners. She has been an R user for over 15 years.

As a data professional, I have enjoyed learning and using multiple tools for my workflows. For me, everything used to begin and end with R. Today, SQL is a must-know. Not being able to pull your own custom tables from a warehouse can make things tricky. Then there is tidyverse, the master collection of packages for data science & analytics. As an OG R user, I cannot envision data work without tidyverse.

In this first part of a 2-part article, I want to demonstrate how a data analyst can use one OR the other for the initial stages of data exploration, and then double down on tidyverse, leveraging ggplot2 for a deeper exploration. By no means does this preclude the extensive use of SQL for data wrangling. Rather, this post showcases the wonders of tidyverse (a collection of R packages designed for data science, sharing an underlying design philosophy, grammar, and data structures) and specifically, ggplot2 (the language of elegant graphics) for a SQL user’s benefit.

1. The dataset and the goal

The dataset I am using is clinical. Sourced from the UCI machine learning repo, it is the Diabetes 130-US hospitals for years 1999-2008 Data Set. The dataset is large, ~100K rows and 51 columns in its raw format. It is, however, clean data. For the purpose of this article, in order to show SQL and tidyverse language in tandem, I will split it up into 5 parts, and we will assume that the data is actually available to us in these 5 different pieces, rather than as the whole, cleaned data, since this is typically the case in real life.

I will skip the portion about dbplyr, referring readers to the hyperlinked article that will show you how to actually pull data from a remote database using tidyverse’s dbplyr. Typically, this is done using SQL, butdbplyr allows you to do this within R. Rather, I will focus on the initial stages of data exploration, using both SQL and tidyverse for the same output, while extending the tidyverse portion to include ggplot2 visualization examples, using different plot types for each use case. Note that in each case, you can use SQL first, and then use the SQL output as an input for the ggplot2 visualization.

2. Reading in the data

The data has been split into 5 parts – demographic, medical, hospital visits, outcome, test results. To learn more about the actual data, see here. Each part is connected with the other through a UID that is a concatenation of the patient encounter ID and the patient number (using either one doesn’t work to make the ID unique). Note that all of the analyses in this post will be done at the UID level, rather than patient level.

#rm(list=ls())
library(sqldf)
library(dplyr)
library(readxl)
#library(dbplyr)
library(ggplot2)

data_path <- "./dataset_diabetes/diabetic_data.xlsx"
dem <- read_excel(data_path, "demo")
meds <- read_excel(data_path, "medications")
visits <- read_excel(data_path, "hosp_visits")
y <- read_excel(data_path, "readmissions")
results <- read_excel(data_path, "test_results")

3. Early explorations

Let’s begin using SQL and tidyverse to answer some initial questions related to the dataset. The primary hypothesis for this data is the impact of HbA1c measurement on readmission rates, where “readmission” is our response. We will also answer a number of other questions along the way to understand the data better, using ggplot2 when we can.

3.1 Look at the data

3.1.1 Get some counts

Let’s take a look at medications and get a sample size for it, first using SQL and then R.

sqldf('SELECT * FROM meds where 1=0') # SQL see col names

##  [1] uid                      metformin                repaglinide             
##  [4] nateglinide              chlorpropamide           glimepiride             
##  [7] acetohexamide            glipizide                glyburide               
## [10] tolbutamide              pioglitazone             rosiglitazone           
## [13] acarbose                 miglitol                 troglitazone            
## [16] tolazamide               examide                  citoglipton             
## [19] insulin                  glyburide-metformin      glipizide-metformin     
## [22] glimepiride-pioglitazone metformin-rosiglitazone  metformin-pioglitazone  
## [25] change                   diabetesMed             
## <0 rows> (or 0-length row.names)

sqldf('SELECT uid, metformin, repaglinide, nateglinide, chlorpropamide FROM meds LIMIT 5') # SQL

##               uid metformin repaglinide nateglinide chlorpropamide
## 1 2278392-8222157        No          No          No             No
## 2 149190-55629189        No          No          No             No
## 3  64410-86047875        No          No          No             No
## 4 500364-82442376        No          No          No             No
## 5  16680-42519267        No          No          No             No

head(meds, n=5) # dplyr

## # A tibble: 5 × 26
##   uid    metfo…¹ repag…² nateg…³ chlor…⁴ glime…⁵ aceto…⁶ glipi…⁷ glybu…⁸ tolbu…⁹
##   <chr>  <chr>   <chr>   <chr>   <chr>   <chr>   <chr>   <chr>   <chr>   <chr>  
## 1 22783… No      No      No      No      No      No      No      No      No     
## 2 14919… No      No      No      No      No      No      No      No      No     
## 3 64410… No      No      No      No      No      No      Steady  No      No     
## 4 50036… No      No      No      No      No      No      No      No      No     
## 5 16680… No      No      No      No      No      No      Steady  No      No     
## # … with 16 more variables: pioglitazone <chr>, rosiglitazone <chr>,
## #   acarbose <chr>, miglitol <chr>, troglitazone <chr>, tolazamide <chr>,
## #   examide <chr>, citoglipton <chr>, insulin <chr>,
## #   `glyburide-metformin` <chr>, `glipizide-metformin` <chr>,
## #   `glimepiride-pioglitazone` <chr>, `metformin-rosiglitazone` <chr>,
## #   `metformin-pioglitazone` <chr>, change <chr>, diabetesMed <chr>, and
## #   abbreviated variable names ¹metformin, ²repaglinide, ³nateglinide, …

sqldf('SELECT COUNT(uid) FROM meds') # SQL

##   COUNT(uid)
## 1     101766

nrow(meds) # R

## [1] 101766

How many patients with a diabetes diagnosis, vs respiratory, circulatory, etc.?

sqldf('SELECT primary_diag, COUNT(*) FROM results GROUP BY primary_diag') # SQL

##   primary_diag COUNT(*)
## 1  circulatory    30437
## 2     diabetes     8757
## 3        other    48149
## 4  respiratory    14423

results %>% group_by(primary_diag) %>% count(primary_diag) # R

## # A tibble: 4 × 2
## # Groups:   primary_diag [4]
##   primary_diag     n
##   <chr>        <int>
## 1 circulatory  30437
## 2 diabetes      8757
## 3 other        48149
## 4 respiratory  14423

How many women came in through an emergency admission type?

sqldf('SELECT gender, admission_type_id, COUNT(*) AS n FROM dem LEFT JOIN visits USING(uid) WHERE admission_type_id=1 GROUP BY gender')  # SQL

##            gender admission_type_id     n
## 1          Female                 1 29448
## 2            Male                 1 24540
## 3 Unknown/Invalid                 1     2

visits %>% left_join(dem, by=join_by(uid)) %>% subset(admission_type_id==1) %>% count(gender, admission_type_id) # dplyr

## # A tibble: 3 × 3
##   gender          admission_type_id     n
##   <chr>                       <dbl> <int>
## 1 Female                          1 29448
## 2 Male                            1 24540
## 3 Unknown/Invalid                 1     2

3.1.2 A mosaic plot

Instead of extracting counts manually, let’s use a mosaic plot to get a sense of how 2 count variables are distributed relative to each other. In this case, age and admission type. This plot sheds light into data availability and asymmetric distributions. For example, here, we see that most patients come from emergency, urgent care, or as an elective, and that there is missing or “not available” admission type data. It is important to retain these 2 categories separately, since they mean different things. Note that in the ggplot parameters, I have not yet introduced axes label cleanup, etc.

p0 <- dem %>% left_join(visits, by=join_by(uid)) %>%  
  mutate(admission_type=ifelse(admission_type_id==1, "1:Emergency", 
                               ifelse(admission_type_id==2, "2:Urgent", 
                               ifelse(admission_type_id==3, "3:Elective", 
                               ifelse(admission_type_id==4, "4:Newborn", 
                               ifelse(admission_type_id==5, "5:Not Available",
                               ifelse(admission_type_id==6, "6:NULL", 
                               ifelse(admission_type_id==7, "7:Trauma Center", 
                                      "8:Not Mapped")))))))) %>% 
  group_by(admission_type, age) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) 
ggplot(p0, aes(x=age, y=admission_type)) +
  geom_tile(aes(fill=n)) + scale_fill_gradient(low="white", high="blue")

3.1.3 A simple join

Let’s join all 5 datasets and look at it. Note that in SQL, in order to look only at the first few columns, we need to know the column names, which is what we first do here.

# (Output suppressed)
sqldf('SELECT * FROM dem LEFT JOIN visits USING(uid) LEFT JOIN results USING(uid) LEFT JOIN meds USING(uid) LEFT JOIN y USING(uid) where 1=0') # SQL

sqldf('SELECT uid, race, gender, age, weight FROM dem LEFT JOIN visits USING(uid) LEFT JOIN results USING(uid) LEFT JOIN meds USING(uid) LEFT JOIN y USING(uid) LIMIT 5') # SQL

##               uid            race gender     age weight
## 1 2278392-8222157       Caucasian Female  [0-10)      ?
## 2 149190-55629189       Caucasian Female [10-20)      ?
## 3  64410-86047875 AfricanAmerican Female [20-30)      ?
## 4 500364-82442376       Caucasian   Male [30-40)      ?
## 5  16680-42519267       Caucasian   Male [40-50)      ?

dem %>% left_join(visits, by=join_by(uid)) %>% left_join(results, by=join_by(uid)) %>% left_join(meds, by=join_by(uid)) %>% left_join(y, by=join_by(uid)) %>% print(n=5) # dplyr, by default shows 10 rows, so we ask it to print 5

## # A tibble: 101,766 × 50
##   uid          race  gender age   weight admis…¹ disch…² admis…³ time_…⁴ payer…⁵
##   <chr>        <chr> <chr>  <chr> <chr>    <dbl>   <dbl>   <dbl>   <dbl> <chr>  
## 1 2278392-822… Cauc… Female [0-1… ?            6      25       1       1 ?      
## 2 149190-5562… Cauc… Female [10-… ?            1       1       7       3 ?      
## 3 64410-86047… Afri… Female [20-… ?            1       1       7       2 ?      
## 4 500364-8244… Cauc… Male   [30-… ?            1       1       7       2 ?      
## 5 16680-42519… Cauc… Male   [40-… ?            1       1       7       1 ?      
## # … with 101,761 more rows, 40 more variables: medical_specialty <chr>,
## #   num_lab_procedures <dbl>, num_procedures <dbl>, num_medications <dbl>,
## #   number_outpatient <dbl>, number_emergency <dbl>, number_inpatient <dbl>,
## #   diag_1 <chr>, diag_2 <chr>, diag_3 <chr>, number_diagnoses <dbl>,
## #   max_glu_serum <chr>, A1Cresult <chr>, primary_diag <chr>, metformin <chr>,
## #   repaglinide <chr>, nateglinide <chr>, chlorpropamide <chr>,
## #   glimepiride <chr>, acetohexamide <chr>, glipizide <chr>, glyburide <chr>, …

3.2 Explore the response: readmissions

3.2.1 Lab procedures

Let’s start with the simplest question – for the primary response variable, “readmitted”, how many lab procedures were done by each category of the response? Note here that “number of lab procedures” is one of a handful of continuous design covariate – rest of the ~45 covariates are all categorical/discrete.

# How many lab tests performed for readmitted patients
sqldf('SELECT readmitted, SUM(num_lab_procedures) AS n FROM visits LEFT JOIN y USING(uid) GROUP BY readmitted') # SQL

##   readmitted       n
## 1        <30  502275
## 2        >30 1558172
## 3         NO 2325224

visits %>% left_join(y, by=join_by(uid)) %>% group_by(readmitted) %>% 
  summarise(n=sum(num_lab_procedures)) %>% mutate(freq = n / sum(n)) # dplyr, w/ an added proportion

## # A tibble: 3 × 3
##   readmitted       n  freq
##   <chr>        <dbl> <dbl>
## 1 <30         502275 0.115
## 2 >30        1558172 0.355
## 3 NO         2325224 0.530

Since the above doesn’t really tell us much, other than actual counts, proportions by response categories, let’s use ggplot2 to explore the distribution of “number of lab procedures”, using a barplot/histogram approach, with “readmitted” as the fill element. This helps us get a better picture of their relationship; we see here how, for a strikingly normally distributed “number of lab procedures” (other than 1 outlier), on average, the higher the volume of procedures, the more the proportion of readmitted.

# How many lab tests performed for readmitted patients, use ggplot2
p1 <- visits %>% left_join(y, by=join_by(uid))
ggplot(data = p1 ,aes(x=num_lab_procedures,fill=readmitted)) + geom_bar() + labs(x="Number of lab procedures", y="counts") + scale_y_continuous(
    labels = function(n) scales::comma(abs(n)))

Let’s also do this using ggplot’s beautiful density plots. It is a slightly different type of visual, and tells us how the distribution of X shifts left or right by the response or fill.

ggplot(p1, aes(num_lab_procedures)) + geom_density(aes(fill=factor(readmitted)), alpha=0.8) + labs(x="Number of lab procedures")

3.2.2 Demographics

Next, we ask how readmissions differ across age groups and gender. Let’s also plot this to understand the output better. We first use a population pyramid approach to get the counts and then barplot the proportions to get a better understanding of the variance in readmissions across these groups.

# What are readmission rates by the different age groups?
sqldf('SELECT age, readmitted, COUNT(*) AS n FROM dem LEFT JOIN y USING(uid) GROUP BY age')  # SQL

##         age readmitted     n
## 1    [0-10)         NO   161
## 2   [10-20)        >30   691
## 3   [20-30)         NO  1657
## 4   [30-40)         NO  3775
## 5   [40-50)         NO  9685
## 6   [50-60)        >30 17256
## 7   [60-70)         NO 22483
## 8   [70-80)        >30 26068
## 9   [80-90)         NO 17197
## 10 [90-100)         NO  2793

p21 <- dem %>% left_join(y, by=join_by(uid)) %>% group_by(gender, age, readmitted) %>% summarise(n=n()) %>%
mutate(pct = 100 * n / sum(n), readmission=ifelse(readmitted=="NO", "not readmitted", "readmitted")) %>% 
  ungroup() %>% subset(gender=="Male"|gender=="Female") %>%
  ggplot() +
  geom_col(aes(x = ifelse(readmission == "readmitted", -n, n),
               y = age,
               fill = readmission)) +
  facet_wrap(~ gender) +
  scale_x_continuous(
    labels = function(n) scales::comma(abs(n))) +
  xlab("Counts")
p21

p22 <- dem %>% left_join(y, by=join_by(uid)) %>% group_by(gender, age, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) %>% subset(gender=="Male"|gender=="Female")
ggplot(data=p22, aes(x=age, y=freq, fill=readmitted)) + geom_col() + facet_wrap(~ gender) + labs(y="proportions") + geom_text(aes(label = paste0(round(freq, 4) * 100, "%")), position = position_stack(vjust = 0.5), size=2.5, angle=90) + theme(axis.text.x = element_text(angle=90, vjust=.5, hjust=1))

The population pyramid is an intriguing plot type, and already tells us that for most age groups, more women are readmitted. But this could be solely because there are more women than men in the sample. However, from the proportion barchart, we see here that proportion of readmitted women is greater than men, particularly for the 20-30 age group.

3.2.3 Patient diagnoses

Finally, how are readmission rates distributed by patient and patient care features. For example, how is it distributed by patient primary diagnosis? In the final section of this post, we will leverage ggplot2’s visualization power to triangulate patient diagnoses with the key covariate and the response. Like in the previous section, we use proportions, adding the relevant labels to more easily infer that we see higher readmission rates for a diabetes diagnosis.

We change around quite a few of the plotting parameters in ggplot2 to make it look much more eye-catching.

# How is readmitted rate distributed by diagnoses?
sqldf('SELECT primary_diag, readmitted, COUNT(*) as n FROM results LEFT JOIN y USING(uid) GROUP BY primary_diag, readmitted')  # SQL

##    primary_diag readmitted     n
## 1   circulatory        <30  3485
## 2   circulatory        >30 10839
## 3   circulatory         NO 16113
## 4      diabetes        <30  1137
## 5      diabetes        >30  3318
## 6      diabetes         NO  4302
## 7         other        <30  5332
## 8         other        >30 15856
## 9         other         NO 26961
## 10  respiratory        <30  1403
## 11  respiratory        >30  5532
## 12  respiratory         NO  7488

p3 <- results %>% left_join(y, by=join_by(uid)) %>% group_by(primary_diag, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n))
ggplot(data=p3, aes(x=primary_diag, y=n, fill=readmitted)) + geom_bar(position = "dodge", stat = "identity", color="black") + scale_fill_manual(values=c("#999999", "#E69F00", "#56B4E9")) + theme_minimal() + labs(x="Primary diagnoses", y="Counts (proportions as labels)") + geom_text(aes(label = paste0(round(freq, 4) * 100, "%")), position = position_dodge(width = 1), vjust=-0.7, size=3) + scale_y_continuous(labels = function(n) scales::comma(abs(n)))

3.2.4 HbA1c measurement

One of the key questions this dataset seeks to answer is the impact of the A1C test (decision to test) on readmission rates, in the presence of covariates (especially the primary diagnosis). Output in its raw form (i.e. untransformed) doesn’t always give us the answer clearly. To get around this, we will use CASE WHEN in SQL and mutate in tidyverse.

Let’s plot this in 2 ways – a barplot with labels, and a spineplot. The latter allows us to see the “weight” of the underlying categories.

# What is the readmission rate profile of patients who had their A1C measured?
sqldf('SELECT CASE WHEN A1Cresult = "None" THEN "not measured" ELSE "measured" END AS a1c, readmitted,
   COUNT(*) FROM results LEFT JOIN y USING(uid) GROUP BY a1c, readmitted ORDER BY 1') # SQL

##            a1c readmitted COUNT(*)
## 1     measured        <30     1676
## 2     measured        >30     5800
## 3     measured         NO     9542
## 4 not measured        <30     9681
## 5 not measured        >30    29745
## 6 not measured         NO    45322

p4 <- results %>% left_join(y, by=join_by(uid)) %>% mutate(a1c=ifelse(A1Cresult=="None", "not measured", "measured")) %>% 
  group_by(a1c, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) 
ggplot(data=p4, aes(x=a1c, y=freq, fill=readmitted)) + geom_col() + labs(x="HbA1c test measurement", y="proportions") + geom_text(aes(label = paste0(round(freq, 4) * 100, "%")), position = position_stack(vjust = 0.5), size=3)# dplyr

# Spineplot
library(ggmosaic)
p5 <- results %>% left_join(y, by=join_by(uid)) %>% left_join(dem, by=join_by(uid))%>% mutate(a1c=ifelse(A1Cresult=="None", "not measured", "measured")) %>% subset(gender=="Male"|gender=="Female")
per <- p5 %>% group_by(a1c, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) 
g <- ggplot(p5) + geom_mosaic(aes(x = product(a1c),fill = readmitted)) 

g + geom_text(data = ggplot_build(g)$data[[1]] %>% 
                group_by(x__a1c) %>%
                mutate(pct = .wt/sum(.wt)*100), 
              aes(x = (xmin+xmax)/2, y = (ymin+ymax)/2, label=paste0(round(pct, 2), "%")))

We observe a lower readmission rate (<30 days) when there is an A1C measurement taken, vs when it is not measured at all. In the 2nd/spineplot, we see this without actually calculating the percentages, while also inferring that number of patients not measured is much higher than those measured. We do however, manually add in the percentages to the spineplot to get a more complete picture on the relationship between HbA1c measurement and readmission rates. These are key findings which we will explore in greater detail, using tidyverse and ggplot2 more extensively, in the next part of this blog series, including cutting these plots across multiple covariates to explore how HbA1c affects readmissions in the presence of other patient groupings. Stay tuned!

February 2023: "Top 40" New CRAN Packages

Tue, 28 Mar 2023 00:00:00 +0000

One hundred seventy-three new packages made it to CRAN in February. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Ecology, Economics, Machine Learning, Mathematics, Medicine, Pharma, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

dcTensor v1.0.1: Implements semi-binary and semi-ternary matrix methods based on non-negative matrix factorization (NMF) and singular value decomposition (SVD). For the details see the reference section of GitHub README.md. There are seven vignettes including Discretized Singular Value Decomposition, Discretized Partial Least Squares, and Discretized Non-negative Tucker Decomposition.

Data

dhis2r v0.1.1: Implements a connection to DHIS2, a global open-source project coordinated by the HISP Centre at the University of Oslo. See the vignette to get started.

ispdata v1.1: Provides access to data from the Rio de Janeiro Public Security Institute including criminal statistics, data on gun seizures and femicide. See README to get started.

ohsome v0.2.1: Implements a client for Heidelberg Institute for Geo information Technology’s OpenStreatMap API and provides functions to analyze the rich data source of OpenStreetMap (OSM) history. See the vignette for examples.

OlympicRshiny v1.0.0: Implements a Shiny App to visualize Olympic Data from 1896 to 2016 residing in a Kaggle Dataset. Look here to get started.

whitewater v0.1.2: Provides methods for retrieving United States Geological Survey (USGS) water data using sequential and parallel processing. See Bengtsson (2022) for background on the parallel methods and README for examples.

Ecology

birdscanR v0.1.2: Provides functions to extract bird and insect data from Birdscan MR1 SQL vertical-looking radar databases, filter, and process the data into Migration Traffic Rates, e.g. #objects per hour and per km, etc. See Haest et al. (2021) and Schmid et al. (2019) for background and the vignette for examples.

Economics

echoice2 v0.2.3: Implements choice models based on economic theory, including MCMC based estimation prediction, Hierarchical Multinomial Logit and Multiple Discrete-Continuous (Volumetric) models. See Allenby, Hardt and Rossi (2019), Kim, Hardt, Kim and Allenby (2022), and Hardt and Kurz (2020) for the underlying theory, and the vignettes on Importing lists of lists and Volumetric Demand and Conjunctive Screening.

Machine Learning

ODRF v0.0.3: Implements oblique decision random random forests as an ensemble of oblique decision trees which use linear combinations of predictors for partitioning trees. See the vignette.

spinner v1.1.0: Provides a torch implementation of Graph Net architecture allowing different options for message passing and feature embedding. Look here for examples.

sMTL v0.1.0: Implements L0-constrained Multi-Task Learning and domain generalization algorithms which are coded in Julia allowing for fast coordinate descent and local combinatorial search algorithms. See Loewinger et al. (2022) for details and look here for an introduction.

tfevents v0.0.2: Provides a convenient way to log scalars, images, audio, and histograms in the tfevent record file format. Logged data can be visualized on the fly using TensorBoard, a web based tool that focuses on visualizing the training progress of machine learning models. Look here for an example.

tidyAML v0.0.1: Implements a simple interface for automatic machine learning that uses the fits tidymodels framework to fit models. Look here for examples.

Mathematics

deFit v0.1.2: Provides functions that use numerical optimization to fit first and second order ordinary differential equations to time series data in order to examine the dynamic relationships between variables or the characteristics of a dynamical system. Look here for examples.

fitlandr v0.1.0: Provides a toolbox for estimating vector fields from intensive longitudinal data and construct potential landscapes. The vector fields can be estimated with two nonparametric methods: the Multivariate Vector Field Kernel Estimator by Bandi & Moloche (2018) and the Sparse Vector Field Consensus algorithm by Ma et al. (2013). The potential landscapes are simulated with the simlandr package of Cui et al. (2021) or with the method of Bhattacharya et al. (2011). See README for examples.

Medicine

CodelistGenerator v1.0.0: Provides functions to generate a candidate code list for the Observational Medical Outcomes Partnership common data model based on string matching. For a given search strategy, a candidate code list will be returned. See the Introduction and the vignettes on Candidate codes, Options, and Codelists for medicaitons.

simaerep v0.4.3: Implements bootstrap based simulation methods to detect clinical trials that may be under reporting adverse events. See Koneswarakantha (2021) for background and README for an example.

Pharma

metalite v0.1.1: Provides A metadata structure for clinical data analysis and reporting based on Analysis Data Model (ADaM) datasets which simplifies clinical analysis and reporting tool development by defining standardized inputs, outputs, and workflow. See Zhang et al. (2022), the package Introduction and the vignettes AE Listing, AE Specification, AE Summary, and Miettinen and Nurminen Test.

Science

gcplyr v1.1.0: Implements tools to import, manipulate and analyze bacterial growth curve data as commonly output by plate readers including for reshaping common plate reader outputs into tidy formats. See README for documentation. There are several vignettes including an Introduction and Analyzing data and Dealing with noise.

locaR v0.1.2: Provides functions to conduct acoustic source localization, as well as organize and check localization data and results. Cobos et al. (2010) gives details of the algorithms. Vignettes include an Introduction, Detecting sound sources, and localize Multiple.

PooldiloutionR v1.0.0: Pool dilution is a isotope tracer technique wherein a biogeochemical pool is artificially enriched with its heavy isotopologue and the gross productive and consumptive fluxes of that pool are quantified. This package calculates gross production and consumption rates from closed-system isotopic pool dilution time series data. See Fischer and Hedin (2002) for background and the vignette for examples.

Statistics

counterfactuals v0.1.1: Implements a modular R6 interface for counterfacual explanation methods including Burghmans et al. (2022), Dandl et al. (2020), and Wexler et al. (2019). See the the Introduction and the vignettes How to extend the package and Other types of models.

semfindr v0.1.4: Provides functions for sensitivity analysis in structural equation modeling using influence measures and diagnostic plots. It supports the leave-one-out case wise sensitivity analysis of Pek and MacCallum (2011). There is an introduction and three additional vignettes: Approximate Case Influence, Selecting Cases in Rerun, and Use Case IDs.

stxplore v0.1.0: Implements statistical tools for spatio-temporal data exploration, including simple plotting functions, covariance calculations and computations similar to principal component analysis for spatio-temporal data. Look here for examples and see the vignettes Exploration using dataframes and Using stars objects.

SurrogateRsq v0.2.0: Implements the surrogate R-squared measure for categorical data analysis proposed in Liu et al. (2022). See the vignette.

Time Series

setartree v0.1.0: Implements the forecasting-specific tree-based model proposed by Godahewa et al. (2022) that is particularly suitable for global time series forecasting. Look here to get started.

StructuralDecompose v0.1.1: Provides functions, which perform very well in the presence of significant level shifts, to explain the behavior of a time series by decomposing it into trend, seasonality and residuals. See the short vignettes Decomposition and Example Walkthrough.

sufficientForecasting v0.1.0: Implements a sufficient forecasting method for a single time series using many predictors and a possibly nonlinear forecasting function. Assuming that the predictors are driven by some latent factors, the SF first conducts factor analysis and then performs sufficient dimension reduction. See Fan et al. (2017), Luo et al. (2022), and Yu et al. (2022) for background and the vignette to get started.

tseriesTARMA v0.3-2: Implements routines for nonlinear time series analysis based on Threshold Autoregressive Moving Average models and provides methods for model fitting and forecasting, tests for threshold effects and unit-root tests. See README for examples.

Utilities

currr v0..1.2: Implements the family of map() functions with frequent saving of the intermediate results. This enables stopping the evaluation and then restarting it from where you left off by reading the already evaluated work from cache. See README to get started.

dataMojo v1.0.0: Implements a grammar of data manipulation with data.table by providing a consistent a series of utility functions that help to solve the most common data manipulation challenges. See the vignette.

hexfont v0.3.1: Contains all the hex font files from the GNU Unifont Project compressed by ‘xz’. GNU Unifont is a duo spaced bitmap font that attempts to cover all the official Unicode glyphs plus several of the artificial scripts in the Under-ConScript Unicode Registry. See the vignette.

parabar v0.10.1: Provides a simple interface in the form of R6 classes for executing tasks in parallel, tracking their progress, and displaying accurate progress bars. See README for examples.

rang v0.2.0: Provides tools to resolve the dependency graph of R packages at a specific time point based on the information from various R-hub web services. The dependency graph can then be used to reconstruct the R computational environment with Rocker. See README and the FAQ.

tablexlsx v0.1.0: Provides functions to export data frames to excel workbooks. See the vignette.

xmpdf v0.1.3: Provides functions to edit XMP metadata in a variety of media file formats as well as edit bookmarks and documentation in pdf files. Functions can detect and use a variety of command-line tools to perform these operations including exiftool, ghostscript, and pdftk. See the Introduction and the FAQ vignette.

Visualization

animate v0.3.9.4: Implements a web-based graphics device to extend base R graphics functions to support frame-by-frame animation and keyframes animation. Target use cases include real-time animated visualizations, agent-based models, dynamical systems, and animated diagrams. See the Introduction and the Q&A vignette.

Durga v1.0.0: Implements a system for plotting grouped data effect sizes. compatible with base R methods for combining plots. See Khan & McLean (2023) and the vignette for examples.

ggrain v0.0.3: Extends ggplot2 to create raincloud plots. See the vignette.

PieGlyph v0.1.0: Extends ggplot2 to replace points in a scatter plot with pie-chart glyphs showing the relative proportions of different categories. There are several vignettes including PieGlyph, Multinomial Classificatio, and Time series example.

January 2023: "Top 40" New CRAN Packages

Tue, 28 Feb 2023 00:00:00 +0000

One hundred sixty-five new packages made it to CRAN in January. Here are my “Top 40” selections in thirteen categories: Actuarial Statistics, Archaeology, Computational Methods, Ecology, Genomics, Mathematics, Medicine, Machine Learning, Science, Statistics, Time Series, Utilities, Visualization.

Actuarial Statistics

actuaRE v0.1.3: Provides functions to fit random effects models using either the hierarchical credibility model alone or combined with a glm or with a Tweedie generalized linear mixed model. See Campo & Antonio (2023) for background and the vignette for an example.

Archaeology

archeoViz v0.2.2: Implements a shiny application for the visualisation, interactive exploration, and web communication of archaeological excavation data. It includes interactive 3D and 2D visualisations, th generation of cross sections and maps, basic spatial analysis methods, and excavation timeline visualisations. There is a short vignette in English and French.

shoredate v1.0.0: Provides tools for shoreline dating Stone Age sites located on the Norwegian Skagerrak coast using methods presented in Roalkvam (2023). See the vignette.

Computational Methods

em v1.0.0: Implements a generic Expectation-Maximization (EM) algorithm within a maximum likelihood framework based on Dempster, Laird, and Rubin (1977). See the vignette for some theory and examples.

ROptimus v3.0.0: Implements a general-purpose optimisation engine that supports Monte Carlo optimisation with the Metropolis criterion and acceptance ratio replica exchange Monte Carlo optimization. See the foundational papers Metropolis et al. (1953), Hastings (1970), Kirkpatrick et al. (1983) and Černý (1985) for background.

Data

CopernicusMarine v0.0.6: Provides functions to import data on the ocean’s physical and biogeochemical state from EU Copernicus Marine Service Information. See README to get started.

itol.toolkit v1.1.0: Provides helper functions to access the Interactive Tree of Life including functions to edit and annotate trees interactively. See the vignette to get started.

rdracor v0.7.2: Provides an interface to the Drama Corpora Project (DraCor) API.

usdoj v1.0.0: Provides functions to fetch data from the U.S Department of Justice API such as press releases, blog entries, and speeches. Look here for notes on data structure.

Ecology

pastclim v1.2.3: Implements methods to extract and manipulate palaeoclimate reconstructions for ecological and anthropological analyses as described in Leonardi et al. (2022). See the Overview and the vignettes available datasets and custom dataset.

PERK v0.0.9.2: Implements a shiny web application to predict and visualize concentrations of pharmaceuticals in the aqueous environment. See Jagadeesan et al. (2022) and the vignette for a walk through.

Gemomics

autoGO v0.9.1: Implements a framework to enable automated, high quality gene ontology enrichment analysis visualizations and a wrapper for differential expression analysis using the DESeq2 package described in Love et al. (2014). See the vignette.

phylter v0.9.6: Provides functions to detect and remove outliers in phylogenomics datasets that build on the Distatis approach described in Abdi et al. (2005). See the vignette.

poolHelper v1.0.0: Provides functions to simulate pooled sequencing data under a variety of conditions, and also evaluate the average absolute difference between allele frequencies computed from genotypes and those computed from pooled data. See Carvalho et al. (2022) for the details and the vignette for an introduction to the package.

Machine Learning

BT v0.3: Implements adaptive boosting trees for Poisson distributed response variables, using log-link function. See Trufin & Denuit (2021), Denuit et al. (2019), and Hainaut & Trufin (2022) for background, and the vignette for an example.

chatgpt v0.1.5: Implements a ChatGPT coding assistant for the RStudio IDE.

evreg v1.0.1: Implements a evidential neural network for the regression model recently introduced in Denoeux (2023) in which prediction uncertainty is quantified by Gaussian random fuzzy numbers as introduced in Denoeux (2023). See the vignette.

FuzzyDBScan v0.0.3: Provides an interface to the Fuzzy DBScan clustering algorithm described in Ienco and Bordogna (2018). See README for an example.

Mathematics

rgudhi v0.1.0: Implements and interface to the C++ library, GHUDI for topological data analysis (TDA) and offers state-of-the-art data structures and algorithms to construct simplicial complexes and compute persistent homology.

Medicine

injurytools v1.0.1: Provides standardized routines and utilities to simplify the data analysis of sports injuries in order to identify and describe the magnitude of sports injury problems and determine the potential risk factors. See the vignettes estimate-epi-measures, prepare-injury-data. and visualize-injury-data.

simIDM v0.0.5: Provides functions to simulate oncology trials using an illness - death model. See Meller, Beyersmann and Rufibach (2019) for background and the vignettes Getting Started and Power and Type 1 Error Correlations.

Science

gravmagsubs v1.0.1: Provides functions to compute the gravitational and magnetic anomalies generated by 3-D vertical rectangular prisms at specific observation points using the method of Plouff (1976). See the vignette.

QurvE v1.0: Implements three methods for solving high-throughput analysis of growth curves and fluorescence data: linear regression, growth model fitting, and smooth spline fits. A shiny application provides access to all features without requiring any programming knowledge. There is a User Manual and there are vignettes on Flouresence Curve Evaluation and Growth Curve Evaluation.

Statistics

BCClong v1.0.0: Implements a Bayesian consensus clustering (BCC) model for multiple longitudinal features via a generalized linear mixed model that allows simultaneous clustering of mixed-type (e.g., continuous, discrete and categorical) longitudinal features. There are vignettes for continuous and mixed type data.

clarify v0.1.2: Provides functions to perform simulation-based inference as an alternative to the delta method for obtaining valid confidence intervals and p-values for regression post-estimation quantities such as average marginal effects and predictions at representative values. The methodology is described in King et al. (2000). See the vignette for examples.

emplikAUC v0.3: Provides functions to test hypotheses and construct confidence intervals for AUC (Area Under receiver operating characteristic curve) and pAUC (partial area under ROC curve) using the method described in Zhao, Ding & Zhou .

jtdm v0.1-0: Implements The Joint Trait Distribution Model in a Bayesian framework using conjugate priors to compute joint probabilities and multivariate confidence intervals. See the vignette.

mbreaks v1.0.0: Functions provide comprehensive treatments for estimating, inferring, testing and selecting linear regression models with structural breaks. See Bai & Perron (1998) for background and the vignette examples.

sdmTMB v0.3.0: Implements spatial and spatiotemporal predictive-process GLMMs (Generalized Linear Mixed Effect Models) using TMB, INLA, and the SPDE (Stochastic Partial Differential Equation) approximation to Gaussian random fields. See Anderson et al. (2022) for background and the vignette for examples.

snapKrig v0.0.1: Provides functions for geostatistical modeling and kriging with gridded data using spatially separable covariance Kronecker covariance functions. See Koch, Lele, Lewis (2020) for descriptions of the computational methods and the vignette for examples.

Time Series

fnets v0.1.2: Implements methods for network estimation and forecasting of high-dimensional time series exhibiting strong serial and cross-sectional correlations under a factor-adjusted vector autoregressive model. See Barigozzi et al. (2022) for the methodology and Owens et al. (2023) for details of the package.

MSinference v0.0.9: Provides functions to perform multiscale analysis of a nonparametric regression or nonparametric regressions with time series errors. See Khismatullina and Vogt (2020), Khismatullina and Vogt (2022) and Khismatullina and Vogt (2023) for the theory and examples, and the vignette to get started.

Utilities

datetimeoffset v 0.2.1: Provides support for a number of datetime string standards including ISO 8601 and pdfmark and also datetimes with UTC offsets and possibly heterogeneous time zones with up to nanosecond precision. See the package Introduction.

mathml v0.5: Provides functions to translate R expressions to MathML or MathJax so that they can be rendered in rmarkdown documents and shiny applications. See the vignette

options v0.0.1: Provides a simple mechanisms for defining and interpreting package options, including helper functions for interpreting environment variables, global options, defining default values and more. See the vignettes environment variables and options.

parquetize v0.5.3: Provides functions to convertcsv, RData, rds, RSQLite, json, ndjson, SAS, SPSS and other files to the Parquet columnar storage format. See the vignette.

scenes v0.1.0: Provides functions to facilitate switching between shiny UIs depending on the information that is to be passed to the request object. See the vignettes Creating new actions and Changing scenes.

shiny.fluent v0.3.0: Provides shiny components based on the Fluent UI JavaScript library. There are several small vignettes including a Tutorial on creating a full dashboard with shiny and Fluent.

Visualization

ggplate v0.0.1: Provides functions to create simple plots of biological culture plates as well as microplates which can plot both continuous and discrete values. See README for examples.

plotRCS v0.1.3: Extends ggplot2 to draw restricted cubic spline curves from a logistic regression model or a Cox proportional hazards regression model using the method described in Harrell (2015). Look here for examples.

viscomp v1.0.0: Implements several visualization tools for exploring the behavior of the components in a network meta-analysis of multi-component interventions including heat plots of the two-by-two component combinations, leave one component combination out scatter plot, violin plots for specific component combination effects, and density plots for components effects. See the vignette for examples.

December 2022: "Top 40" New CRAN Packages

Mon, 30 Jan 2023 00:00:00 +0000

One hundred sixteen new packages stuck to CRAN in December 2022. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Ecology, Epidemiology, Genomics, Machine Learning, Mathematics, Medicine, Networks, Signal Processing, Statistics, Utilities, and Visualization.

Computational Methods

bgw v0.1.0: Implements the BGW algorithm described by Bunch, Gay and Welsch (1993) which exploits the special structure of statistical estimation problems within a trust-region-based optimization approach to produce an estimation algorithm that is much more effective than the usual practice of using optimization methods originally developed for general optimization. See the vignette.

GPArotateDF v2022.12-1: Provides functions for derivative-free gradient projection algorithms for factor rotation. See Jennrich (2004) and Bernaards and Jennrich (2005) for the theory, and the vignette for examples.

Mestim v0.2.1: Implements a flexible framework for estimating the variance-covariance matrix of parameters by computing the empirical sandwich variance. See Tsiatis et al. (2019) for background and the vignette for some theory and examples.

Data

dafishr v1.0.0: Provides functions to download, clean and analyse raw Vessel Monitoring System, VMS, data from Mexican government. See the vignette to get started.

hcidata v0.1.0: Provides a collection of datasets of human-computer interaction (HCI) experiments. Each dataset is from an HCI paper, with all fields described and the original publication linked. The datasets sources include Bergström et al. (2022), Dalsgaard et al. (2021), Larsen et al. (2019), Lilija et al. (2019), Pohl and Murray-Smith (2013), and Pohl and Mottelson (2022).

Ecology

bamm v0.4.3: Implements species distribution models as a function of biotic and movement factors. See Soberón and Osorio-Olvera (2022) for the theoretical background and the vignette for examples.

gadget3 v0.8-4: Provides a framework for creating marine ecosystem models that generatesR or C++ code which can then be optimized using the TMB package and standard tools. See Kristensen et al. (2016) and Begley & Howell (2004) for background and the vignettes Model Debugging, Model Structure, and Writing G3 Actions.

Epidemiology

finalsize v0.1: Provides functions to calculate the final size of a susceptible-infectious-recovered epidemic in a population with demographic variation in contact patterns and susceptibility to disease. See Miller (2012) for details. Additionally, there is an Introduction and vignettes on modeling uncertainty in R0 and modeling heterogeneous susceptibility.

Genomics

betaclust v1.0.0: Implements the family of novel beta mixture models developed by Majumdar et al. (2022) to appositely model the beta-valued cytosine-guanine dinucleotide (CpG) sites, to objectively identify methylation state thresholds, and to identify the differentially methylated CpG (DMC) sites using a model-based clustering approach. See the vignette.

plasma v0.9.21: Implements tools for supervised analyses of incomplete, overlapping multiomics datasets. Functions apply partial least squares in multiple steps to find models that predict survival outcomes. See the vignettes on Interpretation and Partial Least Squares.

POMS v1.0.1: Provides code to identify functional enrichments across diverse taxa in phylogenetic tree, particularly where these taxa differ in abundance across samples in a non-random pattern. See Douglas et al. (2022) for background and look here for documentation.

vivaldi v1.0.0: Provides functions to analyze minor alleles of viral genomes from sequencing data of viral genomes in Illumina vcf files. See the vignette.

Machine Learning

arf v0.1.2: Implements adversarial random forests, an iterative algorithm that recursively partition data into fully factorized leaves, where features are jointly independent. See Watson et al. (2022) for details and the vignette on Density Estimation.

correctR Provides corrected test statistics for cases when samples are not independent, such as when classification accuracy values are obtained over resamples or through k-fold cross-validation, as proposed by Nadeau and Bengio (2003) and presented in Bouckaert and Frank (2004). See the vignette.

HTT v0.1.1: Implements a novel decision tree algorithm in the hypothesis testing framework that examines the distribution difference between two child nodes over all possible binary partitions. The test statistic enables the algorithm to better detect complex structure and not only the mean difference. See the vignette

janus v1.0.0: Implements a tensorflowbased, deep-neural network recommending system with coarse-to-fine optimization. Look here for an introduction.

reservr v0.0.1: Provides functions to fit distributions and neural networks to censored and truncated data. See Bücher & Rosenstock (2022) for background and the vignettes Working with Distributions and TensorFlow Integration.

Mathematics

qspray v0.1.1: Provides functions for the symbolic calculation and evaluation of multivariate polynomials with rational coefficients. See README for an example.

Medicine

cstime v2022.11.22: Provides consistent time conversion functions for public health purposes including conversions between date, ISO week, ISO yearweek, ISO year, calendar month/year, season, and season week. There is an Introduction and there are additional vignettes Date, year, week conversion and Season week.

GNGTools v1.0.0: Implements a Go/No-Go decision-making framework based on Bayesian posterior probabilities linked to the target product profile. See the vignette.

oncomsm v0.1.2: Implements methods to fit a parametric Bayesian multi-state model to tumor response data. The model can be used to sample from the predictive distribution, to impute missing data, and to calculate probability of success for custom decision criteria during an ongoing trial. See the vignettes avoiding-bias, oncomsm, and prior-choice.

REDCapDM v0.1.0: Provides functions to to read and manage REDCap data and identify missing or extreme values. See the vignette.

sitePickR v0.0.1: Provides functions to perform a two-level process to select a representative sample of sites for a prospective study such as a randomized controlled trial where the possibility of an initially selected site may not want to participate is anticipated. See Deville & Tillé (2004) and Tillé (2011) for background on the method employed and the vignette for a demo.

Networks

metanetwork v0.7.0: Provides a toolbox to represent large trophic networks in space or time across aggregation levels. The layout algorithm uses dimension reduction on a diffusion graph kernel and trophic levels. Look here for examples.

oddnet v0.1.0: Implements a feature-based method to identify anomalies in dynamic, temporal networks. See Kandanaarachchi & Hyndman (2022) for the theory and the vignette for an example.

Signal Processing

ohun v0.1.0: Provides functions to facilitates the automatic detection of acoustic signals, including functions to diagnose and optimize the performance of detection routines. See Hossin & Sulaiman (2015) for background and the vignette for examples.

Statistics

gmvjoint v0.1.0: Implements an EM algorithm for fitting joint survival and glm models of longitudinal data. See Bernhardt (2015) for background and README for examples.

hdcate v0.1.0: Implements a two-step double-robust method to estimate the conditional average treatment effects (CATE) with potentially high-dimensional covariates as described in Fan et al. (2022). See the vignette.

jointVIP v0.1.1: Implements joint variance importance plots to assist with variable selection and parameter tuning. See by Liao et al. (2023) for background. There is a Getting Started Guide and a vignette on Options.

nieve v0.1.1: Provides utility functions and objects for extreme value analysis including probability functions with their exact derivatives and transformations exchanging the two parameterizations of peaks over threshold models. See the vignette.

RelDists v1.0.0: Provides functions for parameter estimation and linear regression models for the reliability distribution families reviewed by Almalki & Nadarajah (2014). Generalized Additive Models are used for location, scale and shape. See Rigby & Stasinopoulos (2005). There are vignettes on the [FWE]() and OQ distributions.

ridgetorus v1.0.1: Implements Principal Component Analysis (PCA) on the torus via density ridge estimation, and includes functions for evaluating, fitting, and sampling these models. See García-Portugués and Prieto-Tirado (2022) for the theory and the vignette for examples.

Utilities

dataset v0.2.0: Implements a subjective interpretation of the W3C DataSet recommendation and the datacube model, which relevant to the global Statistical Data and Metadata eXchange standards, the Dublin Core, and the DataCite standards preferred by European open science repositories to improve the findability, accessibility, interoperability and reusability of the datasets. There are several vignettes including Motivation and Datasets with FAIR Metadata.

nlmixr2rpt v0.1.0: provides functions to generate reporting workflows around nlmixr2 analyses with outputs in Word and PowerPoint including functions to specify figures, tables and report structure in a user-definable YAML file. See the vignettes Accessing Figures and Tables and Reporting nlmixr2 Fit Results.

qtwAcademic v2022.12.13: Provides Quarto website templates commonly used by academics including templates for personal websites and course/workshop websites. There is an Intraduction and there are vignettes for workshop and personal websites.

r2social v1.0: Provides JavaScript and CSS styles to allow easy incorporation of various social media elements in shiny applications, dashboards and rmarkdown documents. The elements include share buttons, connect with us buttons, and hyperlink buttons. See the vignette.

rworkflows v0.99.5: Implements functions for the continuous integration for R packages, automated testing, website building, and containerised deployment. See the vignettes depgraph, docker, repositories report, and rworlflows.

shinyDatetimePicker v1.0.0: Provides three types of datetime pickers for usage in a shiny UI. See README for examples.

Visualization

ddplot v0.0.1: Implements an API to allow users to create D3 based SVG plots using the r2d3 library. See the vignette.

swipeR v0.1.0: Provides tools to create carousels using the JavaScript library Swiper and htmlwidgets. Carousels can be displayed in the RStudio viewer pane, in Shiny applications and in rmarkdown documents. See README for examples.

November 2022: "Top 40" New CRAN Packages

Tue, 03 Jan 2023 00:00:00 +0000

One hundred sixty-seven new packages made it to CRAN in November: Here are my “Top 40” selections in fourteen categories: Climate Modeling: Computational Methods, Data, Ecology, Epidemiology, Genomics, Machine Learning, Mathematics, Networks, Pharma, Statistics, Time Series, Utilities, and Visualization.

Climate Modeling

gtfs2emis v0.1.0: Implements a bottom up model to estimate the emission levels of public transport systems based on General Transit Feed Specification data. Functions estimate several pollutants at high spatial and temporal resolutions. See Viera et al for background. There is a Getting Started Guide, and vignettes on Emission Factors and Preparing Data.

Computational Methods

jmatrix v1.1: A mainly instrumental package that allows other packages with cores written in C++ to read, write and manipulate matrices in a binary format to mitigate memory issues. See the vignette.

lazyNumbers v1.2.1: Implements lazy numbers, a new number type whose arithmetic is exact, contrary to ordinary floating-point arithmetic. The lazy numbers are implemented in C++ with the CGAL library. See README for examples.

miesmuschel v0.0.2: Provides optimization algorithms and functions that can be used to manually construct specialized optimization loops including the mixed integer evolution strategy as described in Li et al. (2013) and the multi-objective optimization algorithms NSGA-II described in Deb et al. (2002). See README for examples.

mlr3mbo v0.1.1: Implements a flexible approach to Bayesian optimization that includes both ready-to-use optimization algorithms as well as fundamental building blocks to construct custom algorithms. See Jones et al. (1998), Knowles (2006), and Ponweiser et al. (2008) for examples of ready to use optimization algorithms and the vignette for examples.

Data

googletraffic v0.1.1: Allows users to create geographically referenced traffic data from the Google Maps JavaScript API. Look here for information to get started.

IPEDS v0.1.1: Implements an interface to the US Post-Secondary Institution Statistics for 2020 which contains information on post-secondary institutions, students, faculty, demographics, financial aid, educational and recreational offerings, and completions. See the vignette.

npi v0.2.0: Provides access the US National Provider Identifier Registry API which contains administrative data linked to a specific individual or organizational healthcare providers. There is an Introduction and a vignette on Advanced Use.

rfars v0.2.0: Implements an interface to the US Fatality Analysis Reporting System. See the vignettes Crash Sequences and Rural Roads.

tidyDisasters v0.1.1: Provides a queryable data set that unites information from three complementary resources: Belgium’s Centre for Research on the Epidemiology of Disasters EMDAT, the US National Consortium for the Study of Terrorism GTD, and the US Federal Emergency Management Agency FEMA. There is a brief User Guide.

Ecology

et.nwfva v0.1.1: Implements a forest management tool developed by the Northwest German Forest Research Institute (NW-FVA) for the five main commercial tree species oak, beech, spruce, Douglas-fir and pine for northwestern Germany. See Albert et al (2022) for background and the vignettes: Beispiele, Nutzerhinweise zum Geleit, and Vergleich der Interpolationsmethoden.

mmodely v0.2.2: Provides functions to perform multivariate modeling of evolved traits, with special attention to understanding the interplay of the multi-factorial determinants of their origins in complex ecological settings. See Stephens (2007), Gruebner (2011), and Garamszegi (2011) for background, and the examples on primate vision and vocal communications.

webSDM v1.1-1: Implements the method of Poggiato et al. (2022) for working with tropic Species Distribution Models. There is an Introduction and there are vignettes on Composit variables, Differences between SDM and Trophic SDM.

Epidemiology

exDE v1.0.0: Provides tools to set up modular ordinary and delay differential equation models for mosquito-borne pathogens, focusing on malaria. See Wu et al. (2022) for a description on the methods implemented. There are ten vignettes including Basic Copmetition Aquatic Mosquito Model and SIP Human Model.

Genomics

idopNetwork v0.1.1: Implements a cartographic tool to chart spatial microbial interaction networks. See Kim et al. (2008) and Wang et al. (2011) for information on functional clustering, Chen et al. (2019) and Cao et al. (2022) for background on the model, and the vignette for examples.

tip v0.1.0: Provides functions to cluster data without specifying the number of clusters using the Table Invitation Prior (TIP) introduced by Harrison et al. (2022). There are vignettes on matrix clustering with CONSTANT and MNIW, Tensor clustering CONSTANT, iris NIW, and usarrests NIW.

scistreer v1.0.1: Enables fast maximum-likelihood phylogeny inference from noisy single-cell data using the ScisTree algorithm described in Wu (2019) making the method applicable to massive single-cell datasets (>10,000 cells). Look here for an example.

Machine Learning

tidyclust v0.1.1: Implements a common interface to specifying clustering models, in the same style as parsnip. Creates unified interface across different functions and computational engines. See README for examples.

VSOLassoBag v0.99.0: Implements an integrated Wrapped LASSO ensemble learning strategy to help select efficient, stable, and high confidential variables from omics-based data. Functions integrate and vote on variables generated from multiple LASSO models to determine the optimal candidates. See Luo et al. (2020) for details and the vignette for examples.

Mathematics

tensorFun v0.1.1: Provides basic functions to handle higher-order tensor data. See Kolda and Bader (2009) for background on tensor decompositions, Bau and Ng (2021) for information on missing values, and the vignette for a short introduction.

TransTGGM V1.0.0: Implements a transfer learning framework for tensor Gaussian graphical models, which takes full advantage of informative auxiliary domains even when non-informative auxiliary domains are present. See Ren, Zhen, and Wang for background and the vignette.

Networks

nett 1.0.0: Provides multiple methods for fitting, model selection and goodness-of-fit testing in degree-corrected stochastic blocks models. Implements the methods in Amini et al.(2013), Bickel and Sarkar (2015), Lei (2016), Wang and Bickel (2017) and more. There are vignettes Community Detection, Visualization, Explore Comm, and dcsbm testing.

Pharma

DrugSim2DR v0.1.0: Implements a tool to predict drug functional similarity for drug repurposing. See the vignette.

parttime v0.1.0: Provides classes for embedding partial missingness as a central part of datetime classes allowing for more ergonomic use of datetimes for challenging datetime computation, including calculations of overlapping date ranges, imputations, and more. See README for examples.

Statistics

GAGAs v0.5.1: Implements the Global Adaptive Generative Adjustment Algorithm for generalized liner models used for improving the computational efficiency in the high-dimensional data analysis. See Wang et al. (2022) for the theory.

kfino v1.0.0: Implements method for detecting outliers with a Kalman filter on impulsed noised outliers and prediction on cleaned data. See Cloez et al. (2022) for the details and the vignettes on outlier detection and outlier detection with parallelization.

pwrss v0.2.0: Implements power and sample size calculations for a number of one, two and three sample tests. See the vignette for examples.

spatstat.models v3.0-2: A member of the spatstat family of packages, it provides multiple functions for exploratory data analysis and nonparametric analysis of spatial data, mainly spatial point patterns. See spatstat.org for documentation.

vMF v0.0.1: Provides functions for fast sampling from von Mises-Fisher distribution using the method proposed in Wood (1994). See the vignette.

Time Series

ardl.nardl v1.2.2: Implements linear and nonlinear autoregressive distributed lag (ARDL & NARDL) models and the corresponding error correction models and includes a test for long-run and short-run asymmetry and the Pesaran, Shin & Smith (2001) bounds test for level relationships.

Utilities

cookies v0.2.0: Provides tools for working with cookies in shiny apps, in part by wrapping the js-cookie JavaScript library. See README for an example.

formatdown v0.1.1: Provides a small set of tools for formatting tasks when creating documents in rmarkdown or quarto. See the vignette.

hellorust v1.0.0: Implements tools to use Rust code in R without hacks or frameworks. Includes basic examples of importing cargo dependencies, spawning threads and passing numbers or strings from Rust to R. Look here and here for for documentation.

prismjs v1.1.0: Implements a server-side rendering in R using Prism, a lightweight, extensible syntax highlighter, built with modern web standards in mind such that no JavaScript library is required in the resulting HTML documents. Look here for examples.

rjtools v1.0.9: Provides tools to create an R Journal rmarkdown template article, that will generate HTML and PDF versions of your paper. There are vignettes rjtools-template, Check, Create article, and Format details.

Visualization

funkyheatmap v0.1.0: Provides functions for generating heatmap-like visualizations for benchmark data frames, which can be fine-tuned with annotations for columns and rows. See Saelens et al. (2019) for background and the vignettes Recreating the dynbenchmark figures and Demo with mtcars for examples.

ggbrain v0.8.0: Implements a ggplot2-consistent approach to generating 2D displays of volumetric brain imaging data from multiple NIfTI images. There is an Introduction and there are vignettes on Aesthetic refinement and Annotations and labels.

ggpcp v0.2.0: Provides a Grammar of Graphics implementation of parallel coordinate plots that incorporates categorical variables into the plots in a principled manner. Look here for examples.

ggredist v0.0.2: Provides ggplot2 extensions for political map making based on simple features and includes palettes and scales for red to blue color mapping and for discrete maps. See README for an example.

ggsector v1.6.6: Implements functions that use grid and ggplot2 to plot sectors, draw complex heat maps and interact with Seurat to plot gene expression percentages. See the vignette.

sherlock v0.5.1: Provides graphical displays and statistical tools for structured problem solving and diagnosis with functions that are especially useful for applying the process of elimination as a problem diagnosis technique. See Hartsborne (2020) for background and README for examples.

October 2022: "Top 40" New CRAN Packages

Mon, 28 Nov 2022 00:00:00 +0000

One hundred seventy-four new packages made it to CRAN in October. Here are my “Top 40” selections in sixteen categories: Astronomy, Biology, Business, Computational Methods, Data, Ecology, Finance, Genomics, Mathematics, Machine Learning, Medicine, Pharma, Statistics, Time Series, Utilities, Visualization.

Astronomy

skylight v1.1: Provides a function to calculate sky illuminance values (in lux) for both the sun and moon. The model is a verbatim translation of the code by Janiczek and DeYoung (1987). There are vignettes for Use and Advanced Use.

Biology

palaeoverse v1.0.0: Provides tools to support data preparation and exploration for palaeobiological analyses including functions for data cleaning, binning (time and space), summarisation and visualisation with the goals of improving code reproducibility and accessibility and establishing standards for the palaeobiological community. See Jones et al. for details, and the contribution guide to get involved.

pirouette v1.6.5: Implements a method to create a Bayesian posterior from a phylogeny that depicts the true evolutionary relationships. See Richèl et al. (2020) for background. There are several vignettes including a Tutorial, a demo, and a guide showing how to use the package in a scientific experiment.

Business

bupaverse v0.1.0: Facilitates loading the packages comprising the bupaverse, an integrated suite of R packages for handling and analysing business process data, developed by the Business Informatics research group at Hasselt University, Belgium. See the Getting Started Guide.

Computational Methods

fastWavelets v1.0.1: Provides an Rcpp implementation of the Maximal Overlap Discrete Wavelet Transform (MODWT) and the À Trous Discrete Wavelet Transform. See Quilty & Adamowski (2018) for background and README for examples.

gips v1.0.0: Employs the methods described in Graczyk et al. (2022) to find the permutation symmetry group under which the covariance matrix of the data is invariant. See the vignettes Optimizers, Theory, and gips.

HomomorphicEncryption v0.1.0: Implements the Brakerski-Fan-Vercauteren (2012), Brakerski-Gentry-Vaikuntanathan (2014), and Cheon-Kim-Kim-Song (2016) schema for fully homomorphic encryption. There are seven short vignettes including BFV, BGV, and CKKS.

rxode2random v2.0.9: Implements parallel random number generation. See Wang et al. (2016) and Fidler et al (2019) for background and README for an example..

Data

airnow v0.1.0: Provides functions to retrieve U.S. Government AirNow air quality data. See README to get started.

amazonadsR v0.1.0: Provides functions to collect data on digital marketing campaigns using the Windsor.ai API. See the tutorial for an example and also look at the related new packages: bingadsR, facebookadsR, googleadsR, instagramadsR, linkedinadsR, pinterestadsR, redditadsR, snapchatadsR, ticktokadsR, twitteradsR. Pablo Sanchez was on a roll in October.

congress v0.0.1: Provides functions to download and read data on United States congressional proceedings through the Congress.gov API of the Library of Congress. See README for an example.

Ecology

canaper v1.0.0: Provides functions to analyze the spatial distribution of biodiversity especially useful in the categorical analysis of neo- and paleo-endemism (CANAPE) as described in Mishler et al. (2014) and for statistical tests to determine the types of endemism that occur in a study area while accounting for the evolutionary relationships of species. There are vignettes on CANAPE, randomization, and parallel computing.

EcoEnsemble v1.0.1: Provides functions to fit and sample from the ensemble model described in Spence et al (2018). There is an Introduction and there are two additional vignettes: ExploringPriors and SyntheticData.

rTRIPLEXCWFlux v0.2.0: Encodes the carbon uptake submodule and evapotranspiration submodule of the TRIPLEX-CW-Flux model to run the simulation of carbon-water coupling. See Zhou et al. (2008) Monteith (1965) for background and the vignette for examples.

stopdetection v0.1.1: Enables stop detection in time stamped trajectory by implementing the Stay Point detection algorithm originally described in Ye (2009) that uses time and distance thresholds to characterize spatial regions as stops. See the vignette for examples.

Finance

highOrderPortfolios v0.1.0: Implements methods to select portfolios using high order moments to characterize return distributions. See Zhou & Palomar (2021) and Wang et al. (2022) for the theory and the vignette to get started.

MSTest v0.1.0: Implements hypothesis testing procedures described in Hansen (1992), Carrasco, Hu, & Ploberger (2014) and Dufour & Luger (2017) that can be used to identify the number of regimes in Markov switching models. See README for an example.

Genomics

metevalue v0.1.13: Implements the e-value method to correct p-values in omics data association studies. See Hebestreit & Klein (2022) and Akalin et.al (2012) for background and the vignette for an example.

SCpubr v1.0.4: Implements a system that provides a streamlined way of generating publication ready plots for known Single-Cell transcriptomics data. Look here for an online reference manual.

Mathematics

Boov v1.0.0: Provides functions to perform the Boolean operations union, difference and intersection on volumes. Computations are done by the C++ library CGAL. See README for some examples. Also, have a look at the package MinkowskiSum.

fitode v0.1.1: Provides methods and functions for fitting ordinary differential equations that use sensitivity equations to compute gradients of ODE trajectories with respect to underlying parameters. See the vignette for details.

manifold v0.1.1: Implements operations for Riemannian manifolds including geodesic distance, Riemannian metric, and exponential and logarithm maps, and also incorporates a random object generator on the manifolds. See Dai, Lin, and Müller (2021) for details.

Machine Learning

SoftBart v1.0.1: Implements the SoftBart model of described by Linero and Yang (2018) with the optional use of a sparsity-inducing prior to allow for variable selection. The vignette contains theory and examples.

tidyfit v0.5.1: Extends the tidy data environment with functions to fit and cross validate linear regression and classification algorithms on grouped data. There are several vignettes including Predicting Boston House Prices, Multinomial Classification, and Rolling Window Time Series Regression.

Medicine

cities v0.1.0: Provides functions to simulate clinical trials and summarize causal effects and treatment policy estimands in the presence of intercurrent events. Have a look at the demo.

RCT2 v0.0.1: Implements various statistical methods for designing and analyzing two-stage randomized controlled trials using the methods developed by Imai, Jiang, and Malani (2021) and Imai, Jiang, and Malani (2022). There are vignettes on Interference and Causal Inference.

Pharma

DTSEA v0.0.3: Implements a novel tool to identify candidate drugs against a particular disease based on the drug target set enrichment analysis. It assumes the most effective drugs are those with a closer affinity in the protein-protein interaction network to the specified disease. See Gómez-Carballa et al. (2022) and Feng et al. (2022) for disease expression profiles, Wishart et al. (2018) and Gaulton et al. (2017) for drug target information, and Kanehisa et al. (2021) for the details of KEGG database. There is a vignette.

nlmixr2lib v0.1.0: Provides tools to create model libraries for nlmixr2. Models include pharmacokinetic, pharmacodynamic, and disease models used in pharmacometrics. See the vignette Creating a model library.

Statistics

aIc v1.0: Implements set of tests for compositional pathologies including for coherence of correlations as suggested by Erb et al. (2020), compositional dominance of distance, compositional perturbation invariance as suggested by (Aitchison (1992) and singularity of the covariation matrix. See the vignette for details and examples.

ktweedie v1.0.1: Uses Reproducing Kernel Hilbert Space methods to implement Tweedie compound Poisson gamma models with high-dimensional predictors for the analyses of zero-inflated response variables. See the vignette for examples.

missoNet v1.0.0: Implements efficient procedures for fitting conditional graphical lasso models linking predictor variables to response variables or tasks, when the response data may contain missing values. See the vignette for examples.

ShalpeyOutlier v0.1.0: Provides methods to use Shapley values to detect, explain, and cell wise impute multivariate outliers. See Mayrhofer and Filzmoser (2022) for details and the vignette for examples.

SpatialfdaR v1.0.0: Provides functions to that implement finite element analysis methods to spatial functional data analysis. See Sangalli et al. (2013) and Bernardi et al. (2018) for background and the vignette for an example.

Time Series

dfms v0..1.3: Provides a user friendly and computationally efficient approach to estimate linear Gaussian dynamic factor models using Kalman filter and EM algorithm methods. See Doz et al. (2011) and Banbura & Modugno (2014) for background and the vignette for examples.

Utilities

ExclusionTable v1.0.0: Provides functions for creating tables of excluded observations by reporting the number before and after each subset() call together with the number of observations that have been excluded. See the vignette.

shiny.tailwind v0.2.2: Allows TailwindCSS to be used in Shiny apps with just-in-time compiling including custom CSS with @apply directive, and custom tailwind configurations. See README for examples.

Visualization

AlphaHull3D v1.1.0: Provides functions to compute the alpha hull of a set of points (informallly: the shape formed by these points) in 3D space. See README for some visualizations, and also have a look at the related packages MeshesTools, and PolygonSoup.

bangladesh v1.0.0: Provides sf objects, shape files, and functions to draw regional chorpleth maps for Bangladesh. See the vignette.

ggstats v0.1.0: Provides functions to create forest plots of regression model coefficients along with new statistics to compute proportions, weighted mean and cross-tabulation statistics, as well as new geometries to add alternative background color to a plot. There are vignettes on plotting coefficients and on computing cross-tabulation, custom proportions, and weighted means.

jagshelper v0.1.11: Provides tools to streamline Bayesian analyses in JAGSincluding functions for extracting output, streamlining assessment of convergence, and producing summary plots. See the vignette for examples.

roughsf v1.0.0: Provides functions to draw maps, including “sketchy”, hand-drawn-like maps using the Javascript library Roughjs. See README for examples.

Analyzing Projected Calculations Using R

Mon, 21 Nov 2022 00:00:00 +0000

Nicolas Nguyen works in the Supply Chain industry, in the area of Demand and Supply Planning, S&OP and Analytics, where he enjoys developing solutions using R and Shiny. Outside his job, he teaches data visualization in R at the Engineering School EIGSI and Business School Excelia in the city of La Rochelle, France.

Introduction

Demand & Supply Planning requires forecasting techniques to determine the inventory needed to fulfill future orders. We can build end-to-end supply chain monitoring processes with R to identify potential issues and run scenario testing.

In a 3-part series, I will walk through a Demand & Supply Planning workflow.

In October, we published the first post, Using R for Projected Inventory Calculations in Demand & Supply Planning. It is an introduction to projected inventory and coverage methodology. Check it out if you haven’t read it yet!
This post, Analyzing Projected Calculations Using R, presents an analysis of a demo dataset using the planr package.
The final post, Visualizing Projected Calculations with reactable and Shiny, will answer the question: how would you present your results to your boss once the analysis is done?

By the end of the series, you will understand how and why to use R for Demand & Supply Planning calculations. Let’s continue!

The “problem” we aim to solve

In Demand & Supply Planning, we often need to calculate projected inventories (inventory needed to fulfill future orders) and projected coverages (demand needed to cover periods of time). In my previous post, we discussed the methodology behind these calculations. If you haven’t read it, click here!

This post will focus on analyzing a demo dataset using the proj_inv() function from the planr package. This function calculates projected inventories and coverages with also some analysis features. Then, we can apply the projected inventories & coverages methodology to analyze our data using R!

Demand & Supply Planning datasets

A typical Demand & Supply Planning dataset has the following basic elements, which are essential to calculate projected inventories & coverages:

An Item (a SKU or a DFU) or a reference (a group/family of SKUs for example)
A Period
An Opening Inventories
Then a Demand and a Supply Plan

We can also add more features, such as stocks parameters expressed in units, coverages, or a combination of both (for example, ensuring a minimum of one month coverage, and also at least two units of stocks).

Overview of the demo dataset

To illustrate a typical Demand & Supply Planning dataset, we can use the demo dataset, [Blueprint_DB]. This dataset contains dummy data that displays common situations in Demand and Supply Planning (alerts, shortages, overstocks, etc.). [Blueprint_DB] also contains data for the most common type of coverage, minimum and maximum targets.

We can upload the dataset directly from the planr package:

library(tidyr)
library(dplyr)
library(stringr)
library(lubridate)

## 
## Attaching package: 'lubridate'

## The following objects are masked from 'package:base':
## 
##     date, intersect, setdiff, union

library(planr)
library(ggplot2)

Blueprint_DB <- planr::blueprint

glimpse(Blueprint_DB)

## Rows: 520
## Columns: 7
## $ DFU                 <chr> "Item 000001", "Item 000002", "Item 000003", "Item…
## $ Period              <date> 2022-07-03, 2022-07-03, 2022-07-03, 2022-07-03, 2…
## $ Demand              <dbl> 364, 1419, 265, 1296, 265, 1141, 126, 6859, 66, 38…
## $ Opening.Inventories <dbl> 6570, 5509, 2494, 7172, 1464, 9954, 2092, 17772, 1…
## $ Supply.Plan         <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ Min.Stocks.Coverage <dbl> 4, 4, 4, 6, 4, 6, 4, 6, 4, 4, 4, 4, 4, 6, 4, 6, 4,…
## $ Max.Stocks.Coverage <dbl> 8, 6, 12, 6, 12, 6, 8, 10, 12, 12, 8, 6, 12, 6, 12…

The dataset has seven fields: Demand Forecast Unit (DFU), Period, Demand, Opening Inventories, Supply Plan, Minimum Stocks Coverage, and Maximum Stocks Coverage. We can use a reactable table to look through all the information at once. This helps us get a sense of the distribution of the data and what it looks like for each variable over time. (Code for the table below is in the Appendix.)

[Table 1: Display of [Blueprint_DB] Data by Column]

Let’s go through each of these columns.

DFU: this is a single product (also called SKU, “Storage Keeping Unit”) with certain characteristics. For example, a SKU sold in one particular distribution channel. In this dataset, we have 10 items.

unique(Blueprint_DB$DFU)

##  [1] "Item 000001" "Item 000002" "Item 000003" "Item 000004" "Item 000005"
##  [6] "Item 000006" "Item 000007" "Item 000008" "Item 000009" "Item 000010"

Period: the horizon of time during which we will project our inventories calculation. It could be monthly, weekly, daily (and so on) buckets, over a certain range, for example, 24 months or 52 weeks. In this case, it is a weekly bucket over 52 weeks.

range(Blueprint_DB$Period)

## [1] "2022-07-03" "2023-06-25"

Demand: the Demand Forecast for each period of time, expressed in units.
- For example, this could be Sales Forecasts or Replenishment Forecasts (if we ship to one warehouse to replenish it).

Looking at the Demand, there’s a range from no demand for units to a demand of 6859 units.

summary(Blueprint_DB$Demand)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0      62     119     508     690    6859

Opening Inventories: our available stocks at the beginning of the horizon, expressed in units.

The first date in our Period is 2022-07-03. Let’s take a look at the summary statistics, where we can see the range of opening inventory from 1222 to 17,772 units:

July_03 <- Blueprint_DB %>% filter(Period == "2022-07-03")
summary(July_03$Opening.Inventories)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1222    2192    4460    5766    7022   17772

Supply Plan: the quantities that we will receive to replenish our stocks at a given time, expressed in units.

The replenishment amount in the Supply Plan ranges from no replenishment to replenishment of over 16,000 units.

summary(Blueprint_DB$Supply.Plan)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0       0       0     256       0   16104

Minimum Stocks Coverage: a parameter of minimum stocks expressed in the number of periods of coverage.
- For example, if we put two months, we aim to maintain a minimum stock coverage of two months, based on the Demand Forecasts.
Maximum Stocks Coverage: a parameter of maximum stocks expressed in the number of coverage periods.
- If we put six months, it means that we aim to maintain our stocks below the coverage of six months, based on the Demand Forecasts.

Some items, like “Item 000004” have the same min and max stocks coverage throughout the period. Other items, like “Item 000009” have variable min and max stocks coverage over time.

It’s always helpful to look at a specific item to better understand the data. We want to ensure it contains all the information that we need:

Dimensions: [DFU] and [Period]
Demand and Supply Planning Values: [Demand] / [Opening Inventories] / [Supply Plan]
(Target) Stocks Levels: [Min.Stocks.Coverage] and [Max.Stocks.Coverage]

For example, let’s look closer at “Item 000008” and verify all the information is there:

Item_000008 <- filter(Blueprint_DB, Blueprint_DB$DFU == "Item 000008")

summary(Item_000008)

##      DFU                Period               Demand     Opening.Inventories
##  Length:52          Min.   :2022-07-03   Min.   : 500   Min.   :    0      
##  Class :character   1st Qu.:2022-09-30   1st Qu.: 744   1st Qu.:    0      
##  Mode  :character   Median :2022-12-28   Median :1176   Median :    0      
##                     Mean   :2022-12-28   Mean   :2203   Mean   :  342      
##                     3rd Qu.:2023-03-27   3rd Qu.:3432   3rd Qu.:    0      
##                     Max.   :2023-06-25   Max.   :6859   Max.   :17772      
##   Supply.Plan    Min.Stocks.Coverage Max.Stocks.Coverage
##  Min.   :    0   Min.   :6           Min.   :10         
##  1st Qu.:    0   1st Qu.:6           1st Qu.:10         
##  Median :    0   Median :6           Median :10         
##  Mean   :  989   Mean   :6           Mean   :10         
##  3rd Qu.:    0   3rd Qu.:6           3rd Qu.:10         
##  Max.   :16104   Max.   :6           Max.   :10

Looking at our summary statistics, we see:

Our [DFU] is “Item 000008”.
We have a [Period] of 52 weeks from 2022-07-03 until 2023-07-03.
The [Opening Inventory] is 17,772 units.
[Demand] ranges from 500 to 6859 units, with an average of 2203 units.
[Supply] ranges from 0 to 16,104 units, with an average of 989 units.
In this case, the [Min.Stocks.Coverage] coverage is constant at 6 units, and the [Max.Stocks.Coverage] is constant at 10.

Great, we have what we need!

This was an overview of a standard, complete, and tidy database, ready to be used for a Demand & Supply planning calculation. All the needed elements are there, organized in three groups of data: Dimensions / Demand & Supply Planning Values / Stocks Levels. Now, we can move on to our calculations!

Calculation of the Projected Inventories

We are going to calculate two things, applying the methodology we saw previously (read about it in the previous post of the series):

Projected Inventories
- Definition: [Stocks at the beginning of a Period of time] - [Demand of the Period] + [Supply Plan to be received during this Period]
- At the very beginning: [Stocks at the beginning of a Period of time] = [Opening Inventories]
Related Projected Coverages
- Definition: how many periods of Demand (forecasts) do the Projected Inventories of a Period of time cover

Using R means that the analysis can be simple and fast. It provides us with a summary view of the portfolio, and then we can zoom in on the items with risks of shortages or overstocks.

Run `proj_inv()`

To see how simple and fast it can be to calculate projected inventories and coverages, let’s apply the function proj_inv() from the planr package to the whole demo dataset to create this summary view:

# Create a calculated database for the Projected Inventories @ DFU level
Calculated_DB <- proj_inv(
  data = Blueprint_DB,
  DFU = DFU,
  Period = Period,
  Demand = Demand,
  Opening.Inventories = Opening.Inventories,
  Supply.Plan = Supply.Plan,
  Min.Stocks.Coverage = Min.Stocks.Coverage,
  Max.Stocks.Coverage = Max.Stocks.Coverage
)

Let’s have a look at the output’s column headers:

colnames(Calculated_DB)

##  [1] "DFU"                            "Period"                        
##  [3] "Demand"                         "Opening.Inventories"           
##  [5] "Calculated.Coverage.in.Periods" "Projected.Inventories.Qty"     
##  [7] "Supply.Plan"                    "Min.Stocks.Coverage"           
##  [9] "Max.Stocks.Coverage"            "Safety.Stocks"                 
## [11] "Maximum.Stocks"                 "PI.Index"                      
## [13] "Ratio.PI.vs.min"                "Ratio.PI.vs.Max"

After the calculation using proj_inv(), we have new columns that give us a complete database for automated analysis.

1st group: Calculated Columns
- Projected.Inventories.Qty: Calculation of projected inventories
- Calculated.Coverage.in.Periods: Calculation of coverages in number of periods
- Safety.Stocks and Maximum.Stocks: Projected Stocks Targets (minimum & maximum) in units, useful for further analysis
2nd group: Analysis Features
- To automate screening and facilitate the decision taking process:
  - PI.Index: Projected Inventories Index: basically, how the [projected inventories] are doing (OK, alert, shortage, overstocks)
  - Ratio.PI.vs.min: A ratio [Projected Inventories] vs. [Minimum Stocks Target]: used with a threshold filter, it allows us to identify the relevant SKUs quickly
  - Ratio.PI.vs.Max: A ratio [Projected Inventories] vs. [Maximum Stocks Target], same as above

Once we have run proj_inv(), we can automate the screening of our demand & supply. We can quickly know if we need to change our inventories and how to do so. This is very powerful as it means we can easily adjust based on Demand & Supply.

Results from `proj_inv()`

Let’s revisit “Item 000008” using this new dataset with calculated columns using the reactable package:

Table 2: The results after applying the proj_inv() function.

The new columns in yellow, green, and blue colors have been calculated and added to the original database. We can quickly scan through “Item 000008” for any particular period and notice information such as:

The calculated coverage on 2022-07-03 is 1.6 (based on Upcoming Demand, we expect our inventory to last 1.6 weeks). The next week, it is 0.6.
From 2022-07-03 to 2022-09-11, we should be on Alert that we will fall below our minimum stock inventory. On 2022-09-18, we fall into shortage.

We can find specific numbers for each Period by scrolling through the table. That is just from proj_inv(), without having to do any manual calculations or corrections.

Outside the analysis features, proj_inv() is very fast. Below are calculations of the speed on a Windows 10 x64 release. If we manage a portfolio of 200 SKUs over a 52 weeks horizon (weekly bucket), the calculation and analysis are done in only 5 seconds. The calculation is still pretty fast, up to 500 SKUs, which makes it quite handy if we want to run some simulations straight in Shiny.

Chart of a classic range and size of Demand Planning or S&OP Portfolio

When we work on the S&OP (Sales & Operations Planning) Process, we work at an aggregated level (production families of products) and in monthly buckets. There are limited numbers of items and the horizon is typically 24 months. We can use the proj_inv() function to quickly calculate and analyze the projected inventories & coverages through a (sometimes complex) supply chain network at the same time.

Learn more

Thank you for reading this walkthrough of applying proj_inv() on a Demand & Supply Planning dataset! I hope you see what is needed to run projected inventory calculations, and how a single function in R can provide important information on projected inventories and coverages.

The reactable package makes quick, easy, pretty tables that are easy to sort and scan. What if we want to show more visual information, like color highlighting and charts?

…let’s find out in the next blog post..!

In the meantime, here are some useful links:

Previous post: Using R for Projected Inventory Calculations in Demand & Supply Planning
planr package GitHub repository: https://github.com/nguyennico/planr
URL for a Shiny app showing a demo of the proj_inv() and light_proj_inv() functions: Demo: app proj_inv() function (shinyapps.io)

Appendix

Code for the reactable table summarising Blueprint_DB:

library(sparkline)

#-------------------
# Create Value_DB
#-------------------

# aggregation
tab_scan <- blueprint %>% select(DFU, Demand, Opening.Inventories, Supply.Plan,
                            Min.Stocks.Coverage,
                            Max.Stocks.Coverage) %>%
      group_by(DFU) %>%
      summarise(Total.Demand = sum(Demand),
                Opening.Inventories = sum(Opening.Inventories),
                Supply.Plan = sum(Supply.Plan),
                Min.Stocks.Coverage = mean(Min.Stocks.Coverage),
                Max.Stocks.Coverage = mean(Max.Stocks.Coverage)
      )
    
# Get results
Value_DB <- tab_scan

#-------------------
# Create Sparklines for Demand
#-------------------
 
# replace missing values by zero
blueprint$Demand[is.na(blueprint$Demand)] <- 0
    
# aggregate
tab_scan <- blueprint %>%
  group_by(DFU,
           Period) %>%
  summarise(Quantity = sum(Demand))
    
# generate Sparkline
Sparkline_Demand_DB <- tab_scan %>%
  group_by(DFU) %>%
  summarise(Demand.Quantity = list(Quantity))

#-------------------
# Create Sparklines for Supply Plan
#-------------------
    
# replace missing values by zero
blueprint$Supply.Plan[is.na(blueprint$Supply.Plan)] <- 0
    
# aggregate
tab_scan <- blueprint %>%
  group_by(DFU,
           Period) %>%
  summarise(Quantity = sum(Supply.Plan))
    
# generate Sparkline
Sparkline_Supply_DB <- tab_scan %>%
  group_by(DFU) %>%
  summarise(Supply.Quantity = list(Quantity))

#----------------------------------------
# Link both databases
tab_scan <- left_join(Value_DB, Sparkline_Demand_DB)
tab_scan <- left_join(tab_scan, Sparkline_Supply_DB)

# Get Results
Overview_DB <- tab_scan

reactable(
  tab_scan,
  compact = TRUE,
  defaultSortOrder = "asc",
  defaultSorted = c("DFU"),
  defaultPageSize = 20,
  columns = list(
    `DFU` = colDef(name = "Item", minWidth = 150),
    `Opening.Inventories` = colDef(
      name = "Opening Inventories (units)",
      aggregate = "sum",
      footer = function(values)
        formatC(
          sum(values),
          format = "f",
          big.mark = ",",
          digits = 0
        ),
      format = colFormat(separators = TRUE, digits = 0)
      #style = list(background = "yellow",fontWeight = "bold")
    ),
    `Total.Demand` = colDef(
      name = "Total Demand (units)",
      aggregate = "sum",
      footer = function(values)
        formatC(
          sum(values),
          format = "f",
          big.mark = ",",
          digits = 0
        ),
      format = colFormat(separators = TRUE, digits = 0),
      style = list(background = "yellow", fontWeight = "bold")
    ),
    `Supply.Plan` = colDef(
      name = "Supply Plan (units)",
      aggregate = "sum",
      footer = function(values)
        formatC(
          sum(values),
          format = "f",
          big.mark = ",",
          digits = 0
        ),
      format = colFormat(separators = TRUE, digits = 0)
    ),
    `Min.Stocks.Coverage` = colDef(
      name = "Min Stocks Coverage (periods)",
      
      cell = data_bars(tab_scan,
                       #round_edges = TRUE
                       #value <- format(value, big.mark = ","),
                       #number_fmt = big.mark = ",",
                       fill_color = "#32CD32",
                       #fill_opacity = 0.8,
                       text_position = "outside-end")
    ),
    `Max.Stocks.Coverage` = colDef(
      name = "Max Stocks Coverage (periods)",
      cell = data_bars(tab_scan,
                       #round_edges = TRUE
                       #value <- format(value, big.mark = ","),
                       #number_fmt = big.mark = ",",
                       fill_color = "#FFA500",
                       #fill_opacity = 0.8,
                       text_position = "outside-end")
    ),
    Demand.Quantity = colDef(
      name = "Demand",
      cell = function(value, index) {
        sparkline(tab_scan$Demand.Quantity[[index]])
      }
    ),
    Supply.Quantity = colDef(
      name = "Supply",
      cell = function(values) {
        sparkline(values, type = "bar")
      }
    )
  ),
  # close columns list
  
  defaultColDef = colDef(footerStyle = list(fontWeight = "bold")),
  columnGroups = list(
    colGroup(
      name = "Demand & Supply Inputs",
      columns = c("Total.Demand", "Opening.Inventories", "Supply.Plan")
    ),
    colGroup(
      name = "Stocks Targets Parameters",
      columns = c("Min.Stocks.Coverage", "Max.Stocks.Coverage")
    )
  )
) # close reactable

September 2022: "Top 40" New CRAN Packages

Thu, 27 Oct 2022 00:00:00 +0000

Two hundred and two new packages made it to CRAN in September. Here are my “Top 40” selections in fourteen categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Pharmacology, Psychology, Science, Social Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

kimfilter v1.0.0: Provides an Rcpp implementation of the multivariate Kim filter, which combines the Kalman and Hamilton filters for state probability inference. The filter is designed for state space models and can handle missing values and exogenous data in the observation and state equations. See Kim et al. (1999) for details and the vignette for examples.

SparseChol v0.1.1: Provides a C++ implementation of sparse LDL decomposition of symmetric matrices and solvers as described in Davis (2016). See README for an example.

Data

allhomes v0.3.0: Provides tools to extract past sales data for specific suburbs and years from the Australian property website including the address and property details, date, price, block size and unimproved value of properties. See README to get started.

kgp 1.1.0: Provides access to the metadata about populations and data about samples from the 1000 Genomes Project, including the 2,504 samples sequenced for the Phase 3 release and the expanded collection of 3,202 samples with 602 additional trios. The data is described in Auton et al. (2015) and Byrska-Bishop et al. (2022), and raw data is available here. See Turner (2022) for details and look here for examples.

eHDPrep v1.2.1: Provides a tool for the preparation and enrichment of health datasets for analysis including functions to assess data quality and enable semantic enrichment of a dataset by discovering metavariables from relationships among input variables determined from user-provided ontologies. See the vignette.

Genomics

refdb v0.1.1: Implements a reference database manager offering a set of functions to import, organize, clean, filter, audit and export reference genetic data and includes functions to download sequence data from Bold Systems and NCBI GenBank. There is an Introduction and a vignette on Downloading and combining data.

RestoreNet v1.0: Implements a random-effects stochastic model that starts from an Ito-type equation describing the dynamics of cells duplication, death and differentiation at clonal level to detect clonal dominance events in gene therapy studies. See Del Core et al., (2022) for details and the vignette for the math and examples.

Machine Learning

DynForest v1.0.0: Implements a random forests model that uses multiple longitudinal predictors to make survival predictions for individual subjects. See Devaux et al.(2022) for the details and the vignette for an example.

multiview v0.4: Provides functions to fit cooperative learning models which are supervised learning models for multiple sets of features (“views”), as described in Ding et al. (2022). See the vignette for an introduction.

survex v0.1.1: Implements methods for explaining survival models. Methods include SurvSHAP(t) as described in Krzyzinski et al. (2022), SurvLIME introduced in Kovalev et al. (2020), as well as methods described in Biecek et al. (2021). See the vignettes Creating custom extensions and Package usage.

voice v0.4.14: Provides general purpose tools for voice analysis, speaker recognition and mood inference. See thevignette for an example.

Mathematics

collatz v1.0.0: Provides functions to explore the Collatz conjecture including the ability to retrieve the hailstone sequence, the stopping time, total stopping time and tree-graph. There are four vignettes including: collat, Hailstone Sequences, and Tree Graphs.

greta.dynamics v0.2.0: Implements a greta extension for analyzing transition matrices and ordinary differential equations representing dynamical systems. Have a look at the iterate-matrix and ode-solve examples.

Medicine

historicalborrow v1.0.4: Implements a hierarchical model and a mixture model to borrow historical control data from other studies to better characterize the control response a study. See Viele et al. (2013) for a discussion of the methods and the vignettes Methods and Usage.

nphRCT v0.1.0: Provides functions to perform a stratified weighted log-rank test in a randomized controlled trial which can be visualized as a difference in average score on the two treatment arms. See Magirr and Burman (2018), Magirr (2020), and Magirr (2022) for a description of the tests and the vignettes Survival tests as differences-of-means and The weighted log-rank test for examples.

Pharmacology

rPBK v0.2.0: Provides functions to fit and simulate any kind of physiologically-based kinetic model which allows for multiple compartments, links between pairs of compartments, and links between compartments and the external medium. See Charles et al. (2022) for background and the vignette for examples.

xhaz v2.0.1: Provides functions to fit relative survival regression models with or without proportional excess hazards and with the additional possibility to correct for background mortality by one or more parameters. See Touraine et al. (2020), Mba et al. (2020), and Goungounga et al. (2019) for a description of the models and the vignette for an introduction.

Psychology

rempsyc v0.0.9: Provides convenience functions for Psychology including functions to customize plots and tables following the style of the American Psychological Association which are exportable to Microsoft Word. There are nine vignettes including Test linear regression assumptions, Planned Contrasts Analyses, and Publication-ready scatter plots.

Science

Karen v1.0: Implements a stochastic framework that combines biochemical reaction networks with extended Kalman filter and Rauch-Tung-Striebel smoothing allowing biologists to investigate the dynamics of cell differentiation from high-dimensional clonal tracking data subject to measurement noise, false negative errors, and systematically unobserved cell types. See Del Core et al. (2022) for details and the vignette for an example.

LMD v1.0.0: Implements Local Mean Decomposition, an iterative and self-adaptive approach for demodulating, processing, and analyzing multi-component amplitude modulated and frequency modulated signals. See Smith (2005) for background and the vignette to get started.

oceanexplorer v0.0.2: Provides tools to explore the NOAA world ocean atlas including functions to extract NetCDF data and visualize physical and chemical parameters. A shiny app allows interactive exploration. Look here for background information and see the vignette for examples.

WormTensor v0.1.0: Implements a toolkit to detect clusters from distance matrices calculated between the cells of multiple animals (Caenorhabditis elegans) from input time-series matrices. Includes functions to generate, cluster, and visualize distance matrices, and to retrieve calculated distance matrices from figshare. See the vignette.

demcon v0.3.0: Implements an open-source toolkit developed by ISciences and the DANTE Project for exploring popular political, institutional, and constitutional datasets with the goal of reducing barriers to entry in political science research by automating common acquisition and pre-processing procedures. This package focuses on the V-Dem dataset. There are four vignettes including A Brief Review of Constitutional Datasets and Country Coding Considerations for Dataset Harmonization.

sdam v1.1.4: Provides tools for performing social dynamics and complexity analyses about the Ancient Mediterranean in the context of the SDAM project based at the Department of History and Classical Studies at Aarhus University. There are vignettes on Dates, Re-encoding people, Datasets, and Maps and Networks.

Statistics

adjustedCurves v0.9.0: Provides functions to estimate and plot confounder-adjusted survival curves using either direct adjustment, inverse probability weighting, empirical likelihood estimation, or targeted maximum likelihood estimation. See Denz et. al (2022) for details and the vignette for an introduction.

CovRegRF 1.0.1: Implements a method that uses random forests to estimate the covariance matrix of a multivariate response given a set of covariates as described in in Alakus et al. (2022). The vignette provides an example.

greta.gp v0.2.0: Provides the syntax to create and combine full rank or sparse Gaussian process kernels in greta. See Golding (2019) for background on greta and the vignette to get started.

fwb v0.1.1: Implements the fractional weighted bootstrap (aka the Bayesian bootstrap) to be used as a drop-in for functions in the boot package. The fractional weighted bootstrap involves drawing weights randomly that are applied to the data rather than resampling units from the data. See Xu et al. (2020) for the theory and README for an example.

glmmrBase v0.1.2: Provides the R6 classes Covariance, MeanFunction and Model to allow for the flexible specification of generalized linear mixed models, and also functions to produce relevant matrices, values, and analyses. See README for details.

rocbc v0.1.1: Provides functions for inferences and comparisons around the AUC, the Youden index, the sensitivity at a given specificity level, the optimal operating point of the ROC curve, and the Youden based cutoff. See Bantis et al. (2018) and Bantis et al. (2021 and the vignette for examples.

vglmer v1.0.2: Provides functions to estimate hierarchical models using mean-field variational Bayes which can accommodate models with an arbitrary number of random effects and requires no integration to estimate. See Goplerud (2022) for details and README for an example.

Time Series

bsvars v1.0.0: Implements MCMC algorithms for Bayesian estimation of Structural Vector Autoregressive (SVAR) models including a wide range of SVAR models. See Lütkepohl & Woźniak (2020), Waggoner & Zha (2003) for background.

gasmodel v0.1.0: Provides functions to estimate, forecast and simulate generalized autoregressive score (GAS) models of Creal, Koopman, and Lucas (2013) and Harvey (2013). There are two case study vignettes Bookshop Orders and Hockey Rankings and another on Probability Distributions.

kalmanfilter v2.0.0: Uses Rcpp to implement a multivariate Kalman filter for state space models that can handle missing values and exogenous data in the observation and state equations. See Kim & Nelson (1999) for details and the vignette for an example.

MultiGlarmaVarSel v1.0: Provides functions to perform variable selection in high-dimensional sparse GLARMA models. See Gomtsyan et al. (2022) for details and the vignette for examples.

VedicDateTime v0.1.1: Provides functions to facilitate conversion between the Gregorian and Vedic calendar systems. See Bokde (2021) and Ramakumar (2011) and the vignette for an overview with examples.

Utilities

bundle v0.1.0P Provides functions to serialize model objects with a consistent interface. See the vignette to get started.

r2resize v1.3: Implements an automatic resizing toolbar for containers, images and tables for markdown, rmarkdown and quarto documents. There is Welcome vignette and another on New features.

Visualization

figuRes2 v1.0.0: Provides functions and supporting documentation to streamline a variety of figure production tasks. There are vignettes on Basics, Forest plots, KM plots, and Production Workflows.

openairmaps v0.5.1: Combines openair air quality maps with leaflet to plot site maps with directional analysis figures such as polar plots, and air mass trajectories. See README for examples.

Using R in Inventory Management and Demand Forecasting

Thu, 20 Oct 2022 00:00:00 +0000

Introduction

Demand & Supply Planning requires forecasting techniques to determine the inventory needed to fulfill future orders. With R, we can build end-to-end supply chain monitoring processes to identify potential issues and run scenario testing.

In a 3-part series, I will walk through a Demand & Supply Planning workflow:

Using R in Inventory Management and Demand Forecasting: an introduction of projected inventory and coverage methodology (this post)
Analyzing Projected Inventory Calculations Using R: an analysis of a demo dataset using the planr package
Visualizing Projected Calculations with reactable and shiny: once the analysis is done, how would you present your results to your boss?

By the end of the series, you will understand how and why to use R for Demand & Supply Planning calculations. Let’s begin!

The “problem” we aim to solve

When we work in Demand & Supply Planning, it’s pretty common that we need to calculate projected inventories (and related projected coverages). We often have three options to perform this calculation, using:

an APS (Advanced Planning System) software
an ERP, such as SAP or JDE
and of course…Excel!

All are fine and have different pros and cons. For example, we simply sometimes don’t have all the data in our ERP either APS, like when we work with third-party distributors or we want to model a supply chain network that relies on different systems with unconnected data.

How about using R to perform these calculations? How simple and fast could they be? And, could we do more than just the calculations?

For example, could we get an analysis of the projected situation of a portfolio (as an output of a function), so we don’t have to look at each product one by one and can instead:

Easily get a summary view of the portfolio?
Then zoom on the products with risks of shortages or overstocks?

In a series of posts, I will demonstrate how R can help us in Demand & Supply planning. This first post introduces the proj_inv() and light_proj_inv() functions for projected inventory and coverage calculations.

proj_inv(): to calculate projected inventories and coverages with some analysis features
light_proj_inv(): to calculate projected inventories and coverages (only)
- Runs faster than the previous function (as it’s lighter and doesn’t provide any analysis features)

With R, we have an efficient way to run end-to-end supply chain monitoring processes.

Methodology

How to calculate projected inventories

First, let’s have a look at an example of how to calculate projected inventories. Consider that the field Demand = Sales Forecasts.

We start with some Opening Inventory of 1000 units.
During month M, we sell 100 units (the Demand). At the end of the 1st period (Month M), the inventory is 900 units.
Then, there’s a demand of 800 units at the end of the following period (Month M+1).
During the period (Month M+2), we get a Supply of 400 units, and sell 100: it is now 1100 units in stock.

That’s all, this is how we calculate projected inventories ☺

Figure 1: Describes the mechanism of the calculation of projected inventories based on Opening Inventories, Demand and Supply

How to calculate projected coverages

Now, let’s have a look at how to calculate projected coverages. The idea: we look forward.

We consider the projected inventories at the end of a period and evaluate the related coverage based on the Upcoming Demand. See the example below:

Figure 2: Description of the calculation of projected coverages, considering the inventories at a point in time and the Upcoming Demand

If we use Excel, we often see a “shortcut” to estimate the related coverages, like considering an average of the Demand over the next 3 or 6 months. This can lead to incorrect results if the Demand is not constant (if we have some seasonality or a strong trend, for example). However, these calculations become very easy through the proj_inv() and light_proj_inv() functions.

Projected inventory calculations in R

Now, let’s see how the above is done using two functions from the planr package. First, let’s create a tibble of data for the example shown above (we will cover Min.Stocks.Coverage and Max.Stocks.Coverage more thoroughly in another post):

# Install the planr package
# remotes::install_github("nguyennico/planr")

library(planr)
library(dplyr)

Planr_Example <-
  tibble::tribble(
          ~DFU,      ~Period, ~Demand, ~Opening.Inventories, ~Supply.Plan, ~Min.Stocks.Coverage, ~Max.Stocks.Coverage,
    "Item0001", "2022-01-01",    100L,                1000L,           0L,                   0L,                   0L,
    "Item0001", "2022-02-01",    100L,                   0L,           0L,                   0L,                   0L,
    "Item0001", "2022-03-01",    100L,                   0L,         400L,                   0L,                   0L,
    "Item0001", "2022-04-01",    800L,                   0L,           0L,                   0L,                   0L,
    "Item0001", "2022-05-01",    100L,                   0L,           0L,                   0L,                   0L,
    "Item0001", "2022-06-01",    300L,                   0L,           0L,                   0L,                   0L,
    "Item0001", "2022-07-01",    100L,                   0L,         400L,                   0L,                   0L
    )

Now apply the proj_inv() function:

Calculated_Inv <- proj_inv(
  data = Planr_Example,
  DFU = DFU,
  Period = Period,
  Demand = Demand,
  Opening.Inventories = Opening.Inventories,
  Supply.Plan = Supply.Plan,
  Min.Stocks.Coverage = Min.Stocks.Coverage,
  Max.Stocks.Coverage = Max.Stocks.Coverage
)

We can look at the output, which provides a lot of useful information including projected inventories and coverages:

glimpse(Calculated_Inv)

## Rows: 7
## Columns: 14
## Groups: DFU [1]
## $ DFU                            <chr> "Item0001", "Item0001", "Item0001", "It…
## $ Period                         <chr> "2022-01-01", "2022-02-01", "2022-03-01…
## $ Demand                         <dbl> 100, 100, 100, 800, 100, 300, 100
## $ Opening.Inventories            <int> 1000, 0, 0, 0, 0, 0, 0
## $ Calculated.Coverage.in.Periods <dbl> 2.9, 1.9, 2.7, 1.7, 0.7, 0.0, 99.0
## $ Projected.Inventories.Qty      <dbl> 900, 800, 1100, 300, 200, -100, 200
## $ Supply.Plan                    <int> 0, 0, 400, 0, 0, 0, 400
## $ Min.Stocks.Coverage            <int> 0, 0, 0, 0, 0, 0, 0
## $ Max.Stocks.Coverage            <int> 0, 0, 0, 0, 0, 0, 0
## $ Safety.Stocks                  <dbl> 0, 0, 0, 0, 0, 0, NA
## $ Maximum.Stocks                 <dbl> 0, 0, 0, 0, 0, 0, NA
## $ PI.Index                       <chr> "OverStock", "OverStock", "OverStock", …
## $ Ratio.PI.vs.min                <dbl> 0, 0, 0, 0, 0, 0, NA
## $ Ratio.PI.vs.Max                <dbl> 0, 0, 0, 0, 0, 0, NA

Focusing on the Projected.Inventories.Qty column, we see that it matches our example calculation in Figure 1.

Calculated_Inv %>% 
  select(Projected.Inventories.Qty)

## # A tibble: 7 × 2
## # Groups:   DFU [1]
##   DFU      Projected.Inventories.Qty
##   <chr>                        <dbl>
## 1 Item0001                       900
## 2 Item0001                       800
## 3 Item0001                      1100
## 4 Item0001                       300
## 5 Item0001                       200
## 6 Item0001                      -100
## 7 Item0001                       200

We can also take a look at projected coverage. It matches our example calculation in Figure 2: the opening coverage is 2.9 months.

Calculated_Inv %>% 
  select(Calculated.Coverage.in.Periods)

## # A tibble: 7 × 2
## # Groups:   DFU [1]
##   DFU      Calculated.Coverage.in.Periods
##   <chr>                             <dbl>
## 1 Item0001                            2.9
## 2 Item0001                            1.9
## 3 Item0001                            2.7
## 4 Item0001                            1.7
## 5 Item0001                            0.7
## 6 Item0001                            0  
## 7 Item0001                           99

Very easy to calculate!

The proj_inv() and light_proj_inv() functions can also be used and combined to perform more complex tasks. I’ve described several use cases in the appendix.

Moving forward, these functions form the basis for the classic DRP (Distribution Requirement Planning) calculation, where, based on some parameters (usually minimum and maximum levels of stock, a reorder quantity, and a frozen horizon), we calculate a Replenishment Plan.

Conclusion

Thank you for reading this introduction of projected inventories and coverages in Demand & Supply Planning! I hope that you enjoyed reading how this methodology translate into R.

ASCM (formerly APICS) guidelines

In the beginning of 2019, the Association for Supply Chain Management (ASCM) published an article about the usage of R (and Python) in Supply Chain Planning, and more precisely for the Sales & Operations Planning (S&OP) process, which is related to Demand and Supply Planning.

Figure 3: An extract from the ASCM article regarding the S&OP and Digital Supply Chain. It shows how R and Python are becoming more and more used for demand & supply planning and are great tools to run a S&OP process.

In the example above, we can see how R helps build the digital environment useful to run the S&OP process, which involves a lot of data processing. The planr package aims to support this process by providing functions that calculate projected inventories.

Stay tuned for more on projected calculations using R

Thank you for reading the introduction on how to use R for projected inventory calculations in Demand & Supply Planning! I hope you enjoyed this introduction to the planr package.

My series of posts will continue with:

Analyzing Projected Calculations Using R (using a demo dataset)
Visualizing Projected Calculations with reactable and Shiny (or, what your boss wants to see)

In the meantime, check out these useful links:

planr package GitHub repository: https://github.com/nguyennico/planr
URL for a Shiny app showing a demo of the proj_inv() and light_proj_inv() functions: Demo: app proj_inv() function (shinyapps.io)

Appendix: Use cases and examples

The proj_inv() and light_proj_inv() functions can easily be used and combined to perform more complex tasks, such as:

Modeling of a Supply Chain Network
Calculation of projected inventories from Raw Materials to Finished Goods
A multi-echelon distribution network: from a National Distribution Center to Regional Wholesalers to Retailers

Becoming a useful tool:

To build an End-to-End Supply Chain monitoring process
To support the S&OP (Sales and Operations Planning) process, allowing us to run some scenarios quickly:
- Change of Sales plan
- Change of Supply (Production) plan
- Change of stock level parameters
- Change of Transit Time
- Etc.

Here are some detailed use cases for the functions.

Third-party distributors

We sometimes work with third-party distributors to distribute our products. A common question is: how much stock do our partners hold?

If we have access to their opening inventories and Sales IN & OUT Forecasts, we can quickly calculate the projected inventories by applying the proj_inv() or light_proj_inv() functions. Then, we anticipate any risks of shortages or overstocks, and create a collaborative workflow.

Figure 4: Illustration of a SiSo (Sales IN Sales OUT) situation. We have some stocks held at a storage location, for example, a third-party distributor, and know what will be sold out of this location (the sales out) and what will be replenished to it (the sales in), as well as the opening inventory. The aim is to calculate the projected inventories and coverages at this location.

From raw materials to finished goods

In the example below, we produce olive oil (but it could be shampoo, liquor, etc.).

We start with a raw material, the olive oil, that we use to fill up different sizes of bottles (35cl, 50cl, etc…), on which we then apply (stick) a label (and back label). There are different labels, depending on the languages (markets where the products are sold).

Once we have a labelled bottle, we put it inside an outer box, ready to be shipped and sold. There are different dimensions of outer boxes, where we can put, for examplem 4, 6 or 12 bottles. They are the Finished Goods.

We have two groups of products here:

Finished Goods
Semi-Finished: at different steps, filled bottle (not yet labelled) or labelled bottle

We might be interested in looking at the projected inventories at different levels / steps of the manufacturing process:

Finished Goods
Raw Materials: naked bottles, labels, or liquid (olive oil)
Semi-Finished Products

For this, we can apply the proj_inv() or light_proj_inv() functions on each level of analysis.

Figure 5: Illustration of a production flow, starting from raw material (liquid of olive oil) which fills bottles (of different countenances) and then labelled with different stickers. The labelled bottles are placed inside different outer boxes, which are the finished goods (called SKU -Stock Keeping Unit - ) .

Multi-echelon networks

In the example below, we are looking at the distribution network within a country.

The products are stored at different locations:

At a Central Stock: managed by the company selling the products, for example, a National Distribution Center.
At some third parties:
- Who buy those products to distribute and sell them in more specific areas or regions
- A network of Wholesalers and Sub-Wholesalers

If we have access to the data of the 3rd parties, such as [Opening Inventories] and [Sales IN & OUT Forecasts], we could apply the proj_inv() or light_proj_inv() functions on each level and visualize the complete projected inventories within the Distribution Network.

Figure 6: Illustration of a distribution network, within a country, with different levels. The first level (level 1) is the main storage location, from which the products are shipped to the wholesalers (level 2), which then supply their local partners, identified here as Sub-Wholesalers (level 3). The aim is to visualize (and manage) the inventories (also Demand & Supply) throughout the whole network.

From production capacity to sales

In the example below, we have one or several Factories (within the box “Production Capacities”) which supply a [Global Stocks], which is then used to supply markets and a [Regional Stocks] directly. The [Regional Stocks] is then supplying other markets and a Third Party Distributor.

We want to manage the Monthly S&OP process:

Linking Sales & Manufacturing Operations through simple modeling.
To balance Demand & Supply over the medium-long term horizon.
Visualizing some impacts through the whole distribution network

And make some simulations and drive some decisions:

Change of Sales Plan
Change of Supply Plan
Change of Inventories Level
Change of Transit Lead Time
Change of Product Mix
Etc.

We could apply the proj_inv() or light_proj_inv() functions on each level and visualize the complete projected inventories within the Distribution Network.

Figure 7: Illustration of a distribution network with a wider geographical presence. The first level (level 1) is the global storage location; it receives products from some factories and holds stocks that are used to supply several markets in the world, and also a regional stock, which is level 2. Then the regional stock is used to supply other markets, usually closer geographically, which is level 3. As described in the picture, the markets can be affiliates (where the stocks belong to the company) and third-party distributors.

Money, Money, Money: RConcillation

Wed, 05 Oct 2022 00:00:00 +0000

Dr. Maria Prokofieva is a member of the R / Business working group which is promoting the use of R in accounting, auditing, and actuarial work. She is also a professor at the Victoria University Business School in Australia and works with CPA Australia.

We continue with our series for “nerdy” accountants who want to diverge from Excel and master the power and beauty of R automation by looking at one of the most important areas of ANY business! Cash!

Cash management is a really critical issue for both business owners and people like me who are trying not to look at recent interest rates jumps.

Cash management includes cash collection, handling, and usage of cash (spending!). It is essential to have enough cash to cover immediate expenses, fund business growth and have working capital. Or in simple terms, you need to have enough cash to pay for your coffee, cover your mortgage repayment and invest in that Tesla Model 3

Cash analysis is an important step to assess companies short-term liquidity, evaluate working capital and make decisions about investments.

Today, we are going to have a look at the step that comes before cash flow visualization. Much much earlier…. Before we are able to put cash flow items on a nice graph, we need to obtain those cash flow items “somehow”.

Accountants don’t have cash flow data by default, and there is no magic way to get it. Rather, it is necessary to go transaction by transaction, classify items, group them, collate them, and double-check that they actually occurred! We need to make sure that we are not double-charged as well as we are not underpaying or omitting any of our payments and they are all included in the list.

We start backwards from this very list and we dig into doing bank reconciliation and in particular, looking at our (business) bank statement. This is indeed a very useful exercise, not only in regards to your business but also for your own expense management.

For this post, we will work through a very simple example, just looking at a bank statement and poking around. It is a “personal” bank statement that comes from Kaggle

cf<-read_csv("bank_st.csv")

## Rows: 107 Columns: 7
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (4): Date, Day, Type, Category
## dbl (3): Debit Amount, Credit Amount, Closing Balance
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

cf%>%head()

## # A tibble: 6 × 7
##   Date     Day    Type  Category `Debit Amount` `Credit Amount` `Closing Balan…`
##   <chr>    <chr>  <chr> <chr>             <dbl>           <dbl>            <dbl>
## 1 1/8/2018 Wedne… Debit Shopping          2500                0          174656.
## 2 1/8/2018 Wedne… Debit Shopping           324                0          174332.
## 3 2/8/2018 Thurs… None  None                 0                0          174332.
## 4 3/8/2018 Friday Debit Shopping           404.               0          173928.
## 5 4/8/2018 Satur… Debit Shopping           100                0          173828.
## 6 4/8/2018 Satur… Debit Shopping          1395                0          172433.

This is a typical bank statement you can view in your bank account where each row is a transaction for a particular reporting period (e.g. month). We do not have the name of the second party for the transactions (e.g. the name of the store or the company that credited/debited the account), but all transactions have been classified - which can be seen under Category.

The dataset has Debit Amount, which is what you were charged, and Credit Amount, which is what you were paid. The Closing Balance is a running balance that shows the amount of cash in your account after the transaction. The most important parts of that Closing Balance are the initial and final numbers and they are used to reconcile (= match) balances in your own “books” (accounting books!= accounting records). If those number do not match, we investigate individual closing balances for the transactions to identify where we were overpaid or underpaid.

Let’s look closer at the data: it is not messy, but not ideal…

Column names have blanks and they do not play well in functions, so let’s use clean_names() from janitor package to make them more R friendly

cf<-cf%>%
  clean_names()

cf%>%head()

## # A tibble: 6 × 7
##   date     day       type  category debit_amount credit_amount closing_balance
##   <chr>    <chr>     <chr> <chr>           <dbl>         <dbl>           <dbl>
## 1 1/8/2018 Wednesday Debit Shopping        2500              0         174656.
## 2 1/8/2018 Wednesday Debit Shopping         324              0         174332.
## 3 2/8/2018 Thursday  None  None               0              0         174332.
## 4 3/8/2018 Friday    Debit Shopping         404.             0         173928.
## 5 4/8/2018 Saturday  Debit Shopping         100              0         173828.
## 6 4/8/2018 Saturday  Debit Shopping        1395              0         172433.

That’s better! so now all variables are in small letters and have snake_case!

names(cf)

## [1] "date"            "day"             "type"            "category"       
## [5] "debit_amount"    "credit_amount"   "closing_balance"

Let’s explore the data and do some simple counting - yes, we love to count!

First, what is the closing balance and how does it change during the month? But before we do so, let’s have a close look at the date column. In the first twenty rows you can see there are a few issues as some dates include single vs. double for days and two-digit vs. four-digit for year. It is also in a string format.

class(cf$date)

## [1] "character"

cf$date[1:20]

##  [1] "1/8/2018"  "1/8/2018"  "2/8/2018"  "3/8/2018"  "4/8/2018"  "4/8/2018" 
##  [7] "4/8/2018"  "4/8/2018"  "4/8/2018"  "5/8/2018"  "6/8/2018"  "6/8/2018" 
## [13] "7/8/2018"  "8/8/2018"  "9/8/2018"  "10/8/2018" "10/8/2018" "11/8/2018"
## [19] "11/8/2018" "11/8/2018"

To fix this, let’s convert to the date type and fix the formatting with lubridate package.

cf$date<-dmy(cf$date)

Now, let’s see the spend per each billing date. We exclude the days with no spend:

cf%>%
  group_by(date)%>%
  summarise(spend=sum(debit_amount))%>%
  filter(spend!=0)%>%
  ggplot(aes(date, spend))+
  geom_line()

Now, let’s see the type of categories we have.

cf%>%count(category, sort=TRUE)

## # A tibble: 10 × 2
##    category          n
##    <chr>         <int>
##  1 Shopping         46
##  2 None             21
##  3 ATM               9
##  4 Interest          8
##  5 Entertainment     7
##  6 Medical           5
##  7 Travel            4
##  8 Restaurant        3
##  9 Rent              2
## 10 Salary            2

This None category does not look right…. What is there?

cf%>% filter(category=="None")%>%
  head()

## # A tibble: 6 × 7
##   date       day       type  category debit_amount credit_amount closing_balance
##   <date>     <chr>     <chr> <chr>           <dbl>         <dbl>           <dbl>
## 1 2018-08-02 Thursday  None  None                0             0         174332.
## 2 2018-08-05 Sunday    None  None                0             0         162098.
## 3 2018-08-08 Wednesday None  None                0             0         158597.
## 4 2018-08-21 Tuesday   None  None                0             0          91343.
## 5 2018-08-24 Friday    None  None                0             0          61755.
## 6 2018-08-26 Sunday    None  None                0             0          38441.

It looks like the majority of these entries are not really transactions, but a closing balance. Do we need to include them? Probably not. Let’s confirm that they are not transactions and have debit_amount and credit_amount as zero

cf%>% filter(category=="None")%>%
  filter(debit_amount!=0 | credit_amount!=0)

## # A tibble: 0 × 7
## # … with 7 variables: date <date>, day <chr>, type <chr>, category <chr>,
## #   debit_amount <dbl>, credit_amount <dbl>, closing_balance <dbl>

and it is a good idea to exclude them

cf<-cf%>%filter(category!="None")

Let’s see which day has the most number of transactions and which category is the most used one (what is the money drainer!):

cf%>%count(day, sort=TRUE)

## # A tibble: 7 × 2
##   day           n
##   <chr>     <int>
## 1 Saturday     36
## 2 Friday       11
## 3 Thursday     10
## 4 Sunday        9
## 5 Wednesday     8
## 6 Monday        7
## 7 Tuesday       5

cf%>%count(category, sort=TRUE)

## # A tibble: 9 × 2
##   category          n
##   <chr>         <int>
## 1 Shopping         46
## 2 ATM               9
## 3 Interest          8
## 4 Entertainment     7
## 5 Medical           5
## 6 Travel            4
## 7 Restaurant        3
## 8 Rent              2
## 9 Salary            2

Well, good, but does not look nice? So let’s “paint it”. (We look at spending where credited amount is $0 per category.)

plot4<-cf%>%filter(credit_amount==0)%>%
  group_by(day)%>%
  summarise(day_spend=sum(debit_amount),
            n=n())%>%
  ggplot(aes(x=fct_reorder(day, desc(day_spend)),
             y=day_spend))+
  geom_col()+ 
  labs(x = "Days", y = "$ value",
title ="Cash across days")+
  theme(
  panel.border = element_blank(),
  panel.grid.major = element_blank(),
  panel.grid.minor = element_blank(),
  axis.line = element_line(colour = "black"),
  axis.text.x = element_text(angle = 90),
plot.title = element_textbox(hjust = 0.5,
                                 width = unit(0.5, "npc"),
                                 margin = margin(b = 15))  )

(plot1|plot2)/(plot3|plot4)

For a real business, this amount of Saturday transactions would raise a red flag, but this data is from personal records, so it looks like someone is having a blast after a busy week :)

Also, with category that None does not sound right…. it is the second highest so… I would really investigate what sort of None is that None…

Well, what are out total earn and which days we are paid and what for?

cf%>%filter(credit_amount>0)%>%
  count(category)

## # A tibble: 2 × 2
##   category     n
##   <chr>    <int>
## 1 Interest     8
## 2 Salary       2

It looks like we have only two major category - interest and salary. Let’s see what brings more money

cf%>%filter(credit_amount>0)%>%
  group_by(category)%>%
  summarise(category_total=sum(credit_amount))

## # A tibble: 2 × 2
##   category category_total
##   <chr>             <dbl>
## 1 Interest          4050.
## 2 Salary          500508

Well, it is still salary! but would be sooo good if it is our passive income that drives the cash flows!

Let’s see the balance for the month…

balance<-sum(cf$credit_amount)-sum(cf$debit_amount)

balance

## [1] 268715

Woohoo! Our balance is positive, so we managed to grow our wealth!

Indeed, it is a very simple example, but a good foundation to start your R experience in accounting!

Done for today: we are ready for bank RConciliation. Stay tuned! ….

References

https://www.kaggle.com/datasets/sandhaya4u/august-bank-statement-sandhaya

August 2022: "Top 40" New CRAN Packages

Mon, 26 Sep 2022 00:00:00 +0000

One hundred ninety-four new package made it to CRAN in August. Here are my “Top 40” picks in thirteen categories: “Computational Methods, Data, Epidemiology, Genomics, Insurance, Machine Learning, Mathematics, Medicine, Pharmaceutical Applications, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

brassica v1.0.1: Executes BASIC programs from the 1970s for historical and educational purposes. This enables famous examples of early machine learning, artificial intelligence, natural language processing, cellular automata, and so on, to be run in their original form. See BASIC.

MultistatM v1.1.0: Provides algorithms to build set partitions and commutator matrices used in the construction of multivariate d-variate Hermite polynomials. See the vignette.

Data

hubeau v0.3.1: functions to retrieve data from Hub’Eau the free and public French National APIs on water. See the vignette.

tinytiger v0.0.4: Download geographic shapes from the United States Census Bureau TIGER/Line Shapefiles. See the vignette.

Epidemiology

disbayes v1.0.0: Provides functions to estimate incidence and case fatality for a chronic disease, given partial information, using a multi-state model. See Jackson et al. (2021) for a description of the methods and the vignette for examples.

insectDiseases v1.2.1: Provides the Ecological Database of the Worlds Insect Pathogens created by David Onstad. See the vignette.

nncc v1.0.0: Provides nearest-neighbors matching and analysis of case-control data. See Cui et al. (2022) for background and the vignette for examples.

Genomics

epitopR v0.1.2: Offers a suite of tools to predict peptide MHC (major histocompatibility complex) presentation in the context of both human and mouse. Results are based on half maximal inhibitory concentration as queried through the immune epitope database API. See Vita et al. (2018) for background and the vignette for an example.

geneHapR v1.1.1: Provides functions to import genome variants data and perform gene haplotype Statistics, visualization and phenotype association. There is an Introduction, and there are vignettes on Data, and Workflow.

geniePBC v1.0.0: Linked to The American Association Research Project Genomics Evidence Neoplasia Information Exchange (GENIE) BioPharma Collaborative, this package provides an interface to the data corresponding to each release from Synapse and to prepare datasets for analysis. There are several vignettes including drug regimen sunburst and pull data synapse.

slendr v0.3.0: Implements a framework for simulating spatially explicit genomic data which leverages real cartographic information for programmatic and visual encoding of spatiotemporal population dynamics on real geographic landscapes. Population genetic models are then automatically executed using a custom built-in simulation ‘SLiM’ script as described in Haller et al. (2019). There are several vignettes including A Basic Tutorial, Programming Dispersion Dynamics, and Tree-sequence processing.

Insurance

actxps v0.2.0: Supports actuarial experience studies with functions to prepare data, create studies and perform exposure calculations as described in: Atkinson & McGarry (2016). Look here for an example.

Machine Learning

aorsf v0.0.2: Provides functions to fit, interpret, and predict with oblique random survival forests. See Jaeger et al. (2022) for methods to accelerate and interpret oblique random survival forest models. There is an Introduction and there are vignettes on Predictions and PD and ICE Curves.

calibrationband v0.2.1: Provides functions to assess the calibration of probabilistic classifiers using confidence bands for monotonic functions, and also facilitate constructing inverted goodness-of-fit tests whose rejection allows for a sought-after conclusion of a sufficiently well-calibrated model. See Dimitriadis et al. (2022) for details and README for an example.

kernelshap v0.2.0: Multidimensional version of the iterative Kernel SHAP algorithm from game theory that has become popular for interpreting Machine Learning Models. Covert & Lee (2021) for details and README for an example.

mlim v0.0.9: Uses automated machine learning techniques to fine-tune a Elastic Net, Gradient Boosting, Random Forest, Deep Learning, Extreme Gradient Boosting, or Stacked Ensemble machine learning model for imputing the missing observations of each variable. See README for examples.

tidytags v1.0.2: Facilitates the analysis of Twitter data by coordinates the simplicity of collecting tweets over time with a Twitter Archiving Google Sheet and the utility of the rtweet package for processing and preparing Twitter metadata. There is a Getting Started Guide and a vignette on using conference hashtags.

Mathematics

ambit v0.1.2: Provides tools to simulate and estimate various types of ambit processes, including trawl processes and weighted trawl processes. There is a vignette on Simulating trawl processes and another on estimating the Trawl function.

rTensor2 v0.1.1: Provides a set of tools for basic tensor operators including eigenvalue decomposition, the QR decomposition and LU decompositions, the inverse of a tensor, and the transpose of a symmetric tensor. (A tensor in this context is a multidimensional array.) See Kernfeld et al. (2015) for background.

Medicine

RIbench v1.0.1: Provides a benchmark suite of tools for indirect methods of reference interval estimation. See the vignette.

spiro v0.1.1: Provides functions to import, process, summarize and visualize raw data from metabolic carts. See Robergs, Dwyer, and Astorino (2010) for details and the vignettes Importing & Processing and Summarizing & Plotting.

Pharmaceutical Applications

gtreg v0.1.1: Provides functions that leverage the gtsummary package to creates tables suitable for regulatory agency submissions. See the vignette on adverse event counting methods.

tidyCDISC v0.1.0: Implements a drag and drop Shiny application to allow users to construct [ADaM](https://www.cdisc.org/adam-course#:~:text=The%20Analysis%20Data%20Model%20(ADaM,wide%20variety%20of%20analysis%20methods.) and CDISC compliant tables and plots for studying population and patient level data. See the vignette for more information and look here to run a simulation.

Statistics

cassowaryr v2.0.0: Computes a range of scatterplot diagnostics (scagnostics) on pairs of numerical variables in a data set including the graph and association-based scagnostics described in Wilkinson & Graham (2008) doi:10.1198/106186008X320465 and the association-based scagnostics described by Grimm (2016). See the vignette.

contsurvplot v0.1.1: Provides tools to visualize the causal effect of a continuous variable on a time-to-event outcome including survival area plots, survival contour plots, survival quantile plots and 3D surface plots. See Denz & Timmesfeld (2022) for details and the vignette for examples.

copre v0.1.0: Provides functions for Bayesian nonparametric density estimation using Martingale posterior distributions and includes the Copula Resampling (CopRe) algorithm, a Gibbs sampler for the marginal Mixture of Dirichlet Process (MDP) model, and an extension for full uncertainty quantification via a new Polya completion algorithm. See Moya & Walker (2022), Fong et al. (2021), and Escobar & West (1995) for background and README for an example .

InteractionPoweR v0.1.1: Provides functions for power analysis of regression models which tests the interaction of two independent variables on a single dependent variable. See the tutorial and README to get started.

singR v0.1.1: Implements the SING algorithm to extract joint and individual non-Gaussian components from two datasets. See Risk & Gaynanova (2021) for the theory and the vignette for examples.

SIRthresholded v1.0.0: Implements a version of the Sliced Inverse Regression method which may be used for variable selection. See the vignette.

sparseR v0.2.0: Implements ranked sparsity methods, including penalized regression methods such as the sparsity-ranked lasso, its non-convex alternatives, and elastic net, as well as the sparsity-ranked Bayesian Information Criterion. See Peterson and Cavanaugh (2022) for background and the vignette for an overview.

spmodel v0.1.0: Provides functions to fit, summarize, and predict a variety of spatial statistical models. Modeling features include anisotropy, random effects, partition factors and big data approaches. There is an Overview, a Detailed Guide and a vignette on Technical Details.

TRexSelector v0.0.1: Provides functions to perform fast variable selection in high-dimensional settings while controlling the false discovery rate at a user-defined target level. See Machkour et al. (2021) for background and the vignette for an overview.

VAJointSurv v0.1.0: Provides functions to estimate joint marker (longitudinal) and survival (time-to-event) outcomes using variational approximations which allow for correlated error terms and multiple types of survival outcomes which may be left-truncated, right-censored, and recurrent. See the vignette for some theory and examples.

Time Series

gmwmx v1.0.2: Implements the Generalized Method of Wavelet Moments with Exogenous Inputs estimator (GMWMX) presented in Cucci et al. (2022). There are vignettes on Estimating and comparing models, Removing extreme values in time series, Generating data, and Simulation.

Utilities

countdown v0.4.0: Provides a simple countdown timer for slides and HTML documents written in RMarkdown or Quarto and with Shiny. See README for an example.

D4TAlink.light v2.2.9: Provides workload management tools to facilitate structuring R&D activities to comply with FAIR principles as discussed by Jacobsen et al. (2017) and with ALCOA+ principles as proposed by the FDA. See the vignette.

tinkr v0.1.0: Provides functions to convert Markdown and RMarkdown files to XML and back to allow their editing with xml2 (XPath) instead of numerous complicated regular expressions. See the vignette.

typetracer v0.1.1: The R language includes a set of defined types, but has been described as being “absurdly dynamic” (Turcotte & Vitek (2019), and lacks tools to specify which types are expected by an expression. typetracerprovides functions to extract detailed information on the properties of parameters passed to R functions. See the vignette.

Visualization

atime v2022.9.16: Provides functions for computing and visualizing comparative asymptotic timings of different algorithms and code versions. Also includes functionality for comparing empirical timings with expected references such as linear or quadratic. There are several vignettes including Comparing git versions, Optimal segmentation examples, and Regular Expression examples.

ggsurvfit v0.1.0: Extends ggplot2 to ease the creation of publication ready survival plots. See the vignettes Gallery and Themes.

oblicubes v0.1.2: Extends the isocubes package to provide three-dimensional rendering for grid and ggplot2 graphics using cubes and cuboids drawn with an oblique projection. See the vignette.

ordr v0.1.0: Provides a tidyverse extension for ordinations and biplots. Ordination comprises several multivariate techniques including principal components analysis which rely on eigen-decompositions or singular value decompositions of pre-processed numeric matrix data. The overlay of the resulting shared coordinates on a scatterplot is called a biplot. See Roux & Rouanet (2005) for background. See the vignettes on Covariance Data and Ordination.

Automate a Twitter bot with the rtweet package and RStudio Connect

Tue, 13 Sep 2022 00:00:00 +0000

When developing the promotion plan for the many exciting talks at rstudio::conf(2022), we thought of using the rtweet package by Michael W. Kearney to create a Twitter bot to automate Tweet announcements when a presentation was about to start. As it turned out, we did not end up using this process, but we thought it would be helpful to share how this can be done with rtweet v1.0.2 and RStudio Connect.

retweet allows users to use Twitter from R, and the function rtweet::post_tweet() allows users to share status updates on their Twitter feed without leaving RStudio (or the IDE of our choice). Once you have the Tweets you would like to post, the question is: how can you automate it so you don’t have to watch the clock anytime you want to make an announcement? One option is to use RStudio Connect, RStudio’s publishing platform which hosts data science content such as Shiny apps, Jupyter notebooks, APIs, etc. that is created in either R or Python. For this application, RStudio Connect’s ability to rerun R Markdown documents on a schedule is crucial.

The major steps in our workflow are:

(Note the files are also available on GitHub).

Create a spreadsheet with the talk schedule

First, we create a spreadsheet with the relevant information on the talks. Here is an example using dummy data:

day	start	title	speaker	twitter_name	image
July 27	10:00AM	30 tips to improve your datascience	André Cardoso Azevedo	@fake_twitter1	thumbnail.jpg
July 27	11:00AM	5 drinks to combine with data science	Marie Wolf	@fake_twitter2	thumbnail.jpg
July 27	11:30AM	Data science techniques that changed my life forever	Jasna Šahbaz	@fake_twitter3	thumbnail.jpg
July 27	12:00PM	10 videos about data science that will make you laugh	Dennis R. Everhart	@fake_twitter4	thumbnail.jpg

Using this starting-off point:

We want to merge the day and start columns to create a date-time variable that specifies the time zone. This is important because we want to publish the Tweet in the time zone of the conference, which may differ from the time zone in which we’re creating this project!
We want to automate as much of the tweet-creation process as possible using R. This will help us avoid manual work if we want to add rows to our table later on.

Clean table and automate Tweet creation

Let’s begin with the following packages:

library(rtweet)
library(dplyr)
library(lubridate)

Now, we can load our data:

dat <-
  data.frame(
  stringsAsFactors = FALSE,
               day = c("July 27", "July 27", "July 27", "July 27"),
             start = c("10:00:00", "11:00:00", "11:30:00", "12:00:00"),
             title = c("30 tips to improve your datascience",
                       "5 drinks to combine with data science",
                       "Data science techniques that changed my life forever",
                       "10 videos about data science that will make you laugh"),
           speaker = c("André Cardoso Azevedo",
                       "Marie Wolf",
                       "Jasna Šahbaz",
                       "Dennis R. Everhart"),
      twitter_name = c("@fake_twitter1",
                       "@fake_twitter2",
                       "@fake_twitter3",
                       "@fake_twitter4"),
             image = c("thumbnail.jpg",
                       "thumbnail.jpg",
                       "thumbnail.jpg",
                       "thumbnail.jpg")
)

As mentioned above, we might be in a place with a different time zone than the conference location. We want to ensure that our Twitter bot posts status updates at the right time in the appropriate time zone.

Let’s specify the time zone in our date-time variable. To find out what it is called, we can see the complete list of time zone names in R using OlsonNames():

# Check timezone names
OlsonNames() %>%
  as_tibble() %>%
  dplyr::filter(stringr::str_detect(value, "US/")) # filtering for US-only

## # A tibble: 12 × 1
##    value            
##    <chr>            
##  1 US/Alaska        
##  2 US/Aleutian      
##  3 US/Arizona       
##  4 US/Central       
##  5 US/East-Indiana  
##  6 US/Eastern       
##  7 US/Hawaii        
##  8 US/Indiana-Starke
##  9 US/Michigan      
## 10 US/Mountain      
## 11 US/Pacific       
## 12 US/Samoa

We create an object s2 that prints the current time in our desired time zone:

s <- Sys.time()
s2 <- format(s, format = "%F %R %Z", tz = "US/Eastern")

s2

## [1] "2022-09-13 13:30 EDT"

Now, we create a new table cleaned_dat with the day and start columns merged into a single date-time object. We set the tz argument to use the appropriate time zone.

cleaned_dat <-
  dat %>%
  dplyr::mutate(date = lubridate::mdy_hms(paste0(day, ", 2022 ",
                                                 start, " EDT"),
                                          tz = "US/Eastern"))

Let’s add on to our mutate() pipeline to automate the Tweet message, which should read:

Happening now! [title]: [speaker] [twitter_name] 

Stream here: url goes here.

The \n creates a line break.

cleaned_dat <-
  cleaned_dat %>%
  dplyr::mutate(
    script =
      paste0(
        "Happening now! ",
        title,
        ": ",
        speaker,
        " ",
        twitter_name,
        "\n",
        "\n",
        "Stream here: ",
        "url goes here",
        "\n",
        "\n"
      ),
    .keep = "unused"
  )

This table contains all of the talks. Let’s create a table that filters to show only talks happening this minute.

filtered_dat <-
  cleaned_dat %>%
  filter(date(date) == date(s2) &
           hour(date) == hour(s2) & 
            minute(date) == minute(s2))

Authenticate R and Twitter

To use the API, all rtweet users must authenticate their Twitter accounts. The rtweet documentation has a comprehensive vignette on authentication. The high-level points are:

Apply for “Elevated” access to the API on the Twitter Developer Portal
Generate an “Access token and secret”
Save the API key, API secret key, access key, and access secret key in your R environment

Once we’ve finished these steps, we can authenticate a bot that takes action on our behalf. We reference the keys that we’ve saved in our R environment:

rbot_token <- rtweet::rtweet_bot(
  api_key = Sys.getenv("RBOT_TWITTER_API_KEY"),
  api_secret = Sys.getenv("RBOT_TWITTER_API_SECRET"),
  access_token = Sys.getenv("RBOT_TWITTER_ACCESS_KEY"),
  access_secret = Sys.getenv("RBOT_TWITTER_ACCESS_SECRET")
)

As our last step, we use rtweet::post_tweet() to post updates to our authenticated Twitter account. In addition to text, post_tweet() can also upload images to Twitter. We can specify them using the media argument.

We want this only to occur when there’s a talk starting at the current time. To accomplish this, we can create an if statement that checks to see if the table is empty. If it is not empty, then it posts the Tweet.

if (dim(filtered_dat)[1] > 0) {
  for (i in 1:dim(dat3)[1]) {
    rtweet::post_tweet(
      status = dat3$script[i],
      media = dat3$image[i],
      token = rbot_token
    )
  }
}

Schedule R Markdown documents on RStudio Connect

We can save the code above in a single R Markdown document and deploy it to RStudio Connect:

On RStudio Connect, we can schedule an R Markdown document to rerun at specific intervals. For this example, we can set it to run every minute. The R Markdown document reruns, filters the table based on the time, and posts a status update if there is a talk happening in the current minute.

With that, our Twitter bot is set up, authenticated, and has the information that it needs to post status updates based on our talk schedule.

Learn more

As mentioned above, there are various ways of automating a Twitter bot with R. RStudio Connect is a convenient choice as it deploys R Markdown documents and all associated files. It’s easy to set up and provides precise control over the schedule.

Another option is to use GitHub Actions:

Check out the GitHub repository for this blog post for an example of the R Markdown file. Happy Tweeting!

Beneath and Beyond the Cox Model

Tue, 06 Sep 2022 00:00:00 +0000

The Cox Proportional Hazards model has so dominated survival analysis over the past forty years that I imagine quite a few people who regularly analyze survival data might assume that the Cox model, along with the Kaplan-Meier estimator and a few standard parametric models, encompass just about everything there is to say about the subject. It would not be surprising if this were true because it is certainly the case that these tools have dominated the teaching of survival analysis. Very few introductory textbooks look beyond the Cox Model and the handful of parametric models built around Gompertz, Weibull and logistic functions. But why do Cox models work so well? What is the underlying theory? How do all the pieces of the standard survival tool kit fit together?

As it turns out, Kaplan-Meier estimators, the Cox Proportional Hazards model, Aalen-Johansen estimators, parametric models, multistate models, competing risk models, frailty models and almost every other survival analysis technique implemented in the vast array of R packages comprising the CRAN Survival Task View, are supported by an elegant mathematical theory that formulates time-to-event analyses as stochastic counting models. The theory is about thirty years old. It was initiated by Odd Aalen in his 1975 Berkeley PhD dissertation, developed over the following twenty years largely by Scandinavian statisticians and their collaborators, and set down in more or less complete form in two complementary textbooks [5 and 9] by 1993. Unfortunately, because of its dependency on measure theory, martingales, stochastic integrals and other notions from advanced mathematics, it does not appear that the counting process theory of survival analysis has filtered down in a form that is readily accessible by practitioners.

In this rest of this post, I would like to suggest a path for getting a working knowledge of this theory by introducing two very readable papers, which taken together, provide an excellent overview of the relationship of counting processes to some familiar aspects of survival analysis.

The first paper, The History of applications of martingales in survival analysis by Aalen, Andersen, Borgan, Gill, and Keiding [4] is a beautiful historical exposition of the counting process theory by master statisticians who developed a good bit of the theory themselves. Read through this paper in an hour of so and you will have an overview of the theory, see elementary explanations for some of the mathematics involved, and gain a working idea of how the major pieces of the theory fit together how they came together.

The second paper, Who needs the Cox model anyway? [7], is actually a teaching note put together by Bendix Carstensen. It is a lesson with an attitude and the R code to back it up. Carstensen demonstrates the equivalence of the Cox model to a particular Poisson regression model. Working through this note is like seeing a magic trick and then learning how it works.

The following reproduces a portion of Carstensen’s note. I provide some commentary and fill in a few elementary details in the hope that I can persuade you that it is worth the trouble to spend some time with it yourself.

Carstensen use the North Central Cancer Treatment Group lung cancer survival data set which is included in the survival package for his examples.

## # A tibble: 228 × 10
##     inst  time status   age   sex ph.ecog ph.karno pat.karno meal.cal wt.loss
##    <dbl> <dbl>  <dbl> <dbl> <dbl>   <dbl>    <dbl>     <dbl>    <dbl>   <dbl>
##  1     3   306      2    74     1       1       90       100     1175      NA
##  2     3   455      2    68     1       0       90        90     1225      15
##  3     3  1010      1    56     1       0       90        90       NA      15
##  4     5   210      2    57     1       1       90        60     1150      11
##  5     1   883      2    60     1       0      100        90       NA       0
##  6    12  1022      1    74     1       1       50        80      513       0
##  7     7   310      2    68     2       2       70        60      384      10
##  8    11   361      2    71     2       2       60        80      538       1
##  9     1   218      2    53     1       1       70        80      825      16
## 10     7   166      2    61     1       2       70        70      271      34
## # … with 218 more rows

It may not be obvious at first because there is no subject ID column, but this data frame contains one row for each of 228 subjects. The first column is an institution code. time is the time of death or censoring. status is the censoring indicator. The remaining columns are covariates. I select time, status, sex and age, drop the others from the our working data frame, and then replicate Carstensen’s preprocessing in a tidy way. The second line of mutate() adds a small number to each event time to avoid ties.

set.seed(1952)
lung <- lung %>% select(time, status, age, sex) %>% 
                  mutate(sex = factor(sex,labels=c("M","F")),
                         time = time + round(runif(nrow(lung),-3,3),2))

To get a feel for the data we fit a Kaplan-Meier Curve stratified by sex.

surv.obj <- with(lung, Surv(time, status == 2))
fit.by_sex <- survfit(surv.obj ~ sex, data = lung, conf.type = "log-log")
autoplot(fit.by_sex,
          xlab = "Survival Time (Days) ", 
          ylab = "Survival Probabilities",
         main = "Kaplan-Meier plot of lung data by sex") +  
 theme(plot.title = element_text(hjust = 0.5))

Next, following Carstensen, I fit the baseline Cox model to be used in the model comparisons below.

m0.cox <- coxph( Surv( time, status==2 ) ~ age + sex, data=lung )
summary( m0.cox )

## Call:
## coxph(formula = Surv(time, status == 2) ~ age + sex, data = lung)
## 
##   n= 228, number of events= 165 
## 
##          coef exp(coef) se(coef)     z Pr(>|z|)   
## age   0.01705   1.01720  0.00922  1.85   0.0643 . 
## sexF -0.52033   0.59433  0.16751 -3.11   0.0019 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
##      exp(coef) exp(-coef) lower .95 upper .95
## age      1.017      0.983     0.999     1.036
## sexF     0.594      1.683     0.428     0.825
## 
## Concordance= 0.603  (se = 0.025 )
## Likelihood ratio test= 14.4  on 2 df,   p=7e-04
## Wald test            = 13.8  on 2 df,   p=0.001
## Score (logrank) test = 14  on 2 df,   p=9e-04

The hazard ratios for age and sexF are given in the output column labeled exp(coef). As Carstensen points out mortality increases by 1.7% per year of age at diagnosis and that women have 40% lower mortality than men.

Carstensen next shows that this model can be exactly replicated by a particular and somewhat peculiar Poisson model. Doing this requires a shift in how time is conceived. In the Kaplan-Meier estimator and the Cox model, time is part of the response vector. In the counting process formulation, time is a covariate. Time is divided into many small intervals of length h in which an individuals “exit status” , d is recorded. d will be 1 if death occurred or 0 otherwise. The h intervals represent an individual’s risk time. The pair (d, h) are used to calculate an empirical rate for the process which corresponds to the theoretical rate:

λ(t) = lim_h→0 P[event in (t, t + h)| at risk at time t]/h (*)

The first step in formulating a Poisson model is to set up a data structure that will allow for this more nuanced treatment of time. The function Lexis from the Epi package creates an object of class Lexis, a data frame with columns that will be used to distinguish event time (death or censoring time) from the time intervals in which subjects are at risk for the event. Collectively, these intervals span period from when the first until the last recorded time.

Lung <- Epi::Lexis( exit = list( tfe=time ),
  exit.status = factor( status, labels=c("Alive","Dead") ),
  data = lung )

## NOTE: entry.status has been set to "Alive" for all.
## NOTE: entry is assumed to be 0 on the tfe timescale.

head(Lung)

##  lex.id tfe lex.dur lex.Cst lex.Xst   time status age sex
##       1   0   308.7   Alive    Dead  308.7      2  74   M
##       2   0   457.4   Alive    Dead  457.4      2  68   M
##       3   0  1008.6   Alive   Alive 1008.6      1  56   M
##       4   0   212.1   Alive    Dead  212.1      2  57   M
##       5   0   885.5   Alive    Dead  885.5      2  60   M
##       6   0  1023.7   Alive   Alive 1023.7      1  74   M

The new variables are:

lex.id - Subject ID
tfe - Time from entry at the beginning of the follow-up interval
lex.dur - Duration of the follow-up interval
lex.Cst - Entry status (Alive in our case)
lex.Xst - Exit status at the end of the follow-up interval: tfe + lex.dur (Either Alive or Dead in our case)

Next, the time variable is sorted to produce a vector of endpoints for the at risk intervals and a new time-split Lexis data frame is created using the splitMulti() function from the popEpi package.

Lung.s <- splitMulti( Lung, tfe=c(0,sort(unique(Lung$time))) )
head(Lung.s)

##  lex.id   tfe lex.dur lex.Cst lex.Xst  time status age sex
##       1  0.00    7.67   Alive   Alive 308.7      2  74   M
##       1  7.67    1.88   Alive   Alive 308.7      2  74   M
##       1  9.55    0.23   Alive   Alive 308.7      2  74   M
##       1  9.78    0.57   Alive   Alive 308.7      2  74   M
##       1 10.35    2.25   Alive   Alive 308.7      2  74   M
##       1 12.60    0.45   Alive   Alive 308.7      2  74   M

summary( Lung.s )

##        
## Transitions:
##      To
## From    Alive Dead  Records:  Events: Risk time:  Persons:
##   Alive 25941  165     26106      165      69632       228

tfe tracks the time from entry into the study. This is calendar time.

Carstensen then fits a Cox model to both the Lexis data set and the time-split Lexis data set and notes that the results match the original baseline Cox model. This is as one would expect since the three different data frames contain the same information. Nevertheless, it is a pleasant surprise that the coxph() and Surv() functions are flexible enough to assimilate the three different input data formats.

mL.cox <- coxph( Surv( tfe, tfe+lex.dur, lex.Xst=="Dead" ) ~ age + sex, eps=10^-11, iter.max=25, data=Lung )
mLs.cox <- coxph( Surv( tfe, tfe+lex.dur, lex.Xst=="Dead" ) ~ age + sex, eps=10^-11, iter.max=25, data=Lung.s )
round( cbind( ci.exp(m0.cox), ci.exp(mL.cox), ci.exp(mLs.cox) ), 6 )

##      exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5%
## age     1.0172 0.999 1.0357    1.0172 0.999 1.0357    1.0172 0.999 1.0357
## sexF    0.5943 0.428 0.8253    0.5943 0.428 0.8253    0.5943 0.428 0.8253

Now, Carstensen executes what would seem to be a very strange modeling maneuver. He turns calender time, tfe into a factor and fits a Cox model with tfe as a covariate.

mLs.pois.fc <- glm( cbind(lex.Xst=="Dead",lex.dur) ~ 0 + factor(tfe) + age + sex, family=poisreg, data=Lung.s )

An important technical point is that the time intervals in equation (*) above do not satisfy the independence assumption for a Poisson regression model. Nevertheless, the standard glm() machinery can be used to fit the model because, as Carstensen demonstrates, the likelihood function for the conditional probabilities is proportional to the partial likelihood function of the Cox model.

cbind( ci.exp(mLs.cox),ci.exp( mLs.pois.fc, subset=c("age","sex") ) )

##      exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5%
## age     1.0172 0.999 1.0357    1.0172 0.999 1.0357
## sexF    0.5943 0.428 0.8253    0.5943 0.428 0.8253

Carstensen concludes that this demonstrates that the Cox model is equivalent to a specific Poisson model which has one rate parameter for each time internal, and emphasizes that this is not a new result. He notes that the equivalence was demonstrated some time ago, theoretically by Theodore Holford [10], and in practice, by John Whitehead [14]. Also, in a vignette [15] for the survival package Zhong et al. state that this trick may be used to approximate a Cox model.

Carstensen then demonstrates that more practical Poisson models can be fit by using splines to decrease the number of at risk intervals. The first uses a spline basis with arbitrary knot locations and the second fits a penalized spline gam model.

splines

t.kn <- c(0,25,100,500,1000) 
mLs.pois.sp <- glm( cbind(lex.Xst=="Dead",lex.dur) ~ Ns(tfe,knots=t.kn) + age + sex, family=poisreg, data=Lung.s )

Penalized splines

mLs.pois.ps <- mgcv::gam( cbind(lex.Xst=="Dead",lex.dur) ~ s(tfe) + age + sex, family=poisreg, data=Lung.s )

Carstensen finishes up this portion of his analysis by noting the similarity of the estimates of age and sex effects from the different models.

ests <-
 rbind( ci.exp(m0.cox),
 ci.exp(mLs.cox),
 ci.exp(mLs.pois.fc,subset=c("age","sex")),
 ci.exp(mLs.pois.sp,subset=c("age","sex")),
 ci.exp(mLs.pois.ps,subset=c("age","sex")) )

cmp <- cbind( ests[c(1,3,5,7,9) ,],
 ests[c(1,3,5,7,9)+1,] )

rownames( cmp ) <-
 c("Cox","Cox-split","Poisson-factor","Poisson-spline","Poisson-penSpl")

 colnames( cmp )[c(1,4)] <- c("age","sex")
round( cmp,5 )

##                  age   2.5% 97.5%    sex   2.5%  97.5%
## Cox            1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Cox-split      1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Poisson-factor 1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Poisson-spline 1.016 0.9980 1.035 0.5993 0.4316 0.8322
## Poisson-penSpl 1.016 0.9983 1.035 0.6021 0.4338 0.8358

This demonstration provides some convincing evidence that both parametric and non-parametric models are part of a single underlying theory! When you think about it, this is an astonishing idea. To further explore the counting process theory of survival models, I provide a definition of Aalen’s multiplicative intensity model and a list of references below that I hope you will find helpful.

Finally, there is much more to Carstensen’s note than I have presented. He goes on to provide a fairly complete analysis of the lung data while looking at cumulative rates, survival, practical time splitting, time varying coefficients and more ideas along the way.

Appendix: Multiplicative Intensity Model

For some direct insight into how the Cox Proportional Hazards model fits into the counting process theory have a look at Odd Aalen’s definition of the multiplicative intensity model. Aalan begins his landmark 1978 paper Nonparametric Inference For a Family of Counting Processes] [1] by defining the fundamental components of his multiplicative intensity model.

Let N = (N₁, . . . N_k) be a multivariate counting process which is a collection of univariate counting processes on the interval [0,t] each of which counts events in [0,t]. The N_i may depend on each other. Let σ(F_t) be the sigma algebra which represents the collection of all events that can be determined to have happened by time, t. Let α = α₁, . . . α be an unknown, non-negative function and let Y = (Y₁, . . . Y_k) be a process observable over [0,t].

Define Λ_i(t) = lim_h→0E(N_i(t + h) - N_i(t) | F_t )/h i = 1, … k, to be the the intensity process of the counting process N.

Then the multiplicative intensity model is defined to be: Λ_i(t) = α_i(t)Y_i(t).

This last line certainly looks like the Cox model, and it is not to difficult to confirm that this is indeed the case. You can find the gory details in Fleming and Harrington [9 p 126] or comprehensive monograph by Andersen et al. [5 p481].

References

I believe that the following references (annotated with a few comments) comprise a reasonable basis from gaining familiarity with the counting process approach to survival modeling.

Aalen (1978) Odd O. Aalen Nonparametric Inference For a Family of Counting Processes. The Annals of Statistics 1978, vol 6, no 4, 701-726 This is the ‘Ur’ paper for the multiplicative intensity process. At least the first half should be approachable with some knowledge of measure theory and conditional expectation.
Aalen & Johansen (1978) Odd O. Allen and Soren Johansen. An Empirical Transition Matrix for Non-homogenous Markov Chains Based on Censored Observations. Scand J Statistics 5: 141-150, 1978. This is the source of the Aalen-Johansen estimator. The etm package provides an implementation.
Aalen et al. (2008) Odd O. Aalen, Ørnulf Borgan and Håkon K. Gjessing. Survival and Event History Analysis; A Process Point of View Springer Verlag 2008. This is definitely the text to read first. It is comprehensive, takes a modern point of view, is well written, and introduces the difficult mathematics without all of the technical details that often slow down the process of learning some new mathematics.
Aalen et al. (2009). Odd O. Aalen, Per Kragh Andersen, Ørnulf Borgan, Richard D. Gill and Niels Keiding. History of applications of martingales in survival analysis vol 5, no 1, June 2009
Andersen et al. (1993) Per Kragh Andersen, Ørnulf Borgan, Richard D. Gill, Niels Keiding. Statistical Models Based on Counting Processes. Springer-Verlag, 1993 This text presents numerous examples along with a discussion of the theory and emphasizes parametric models.
Borgan (1997). Ørnulf Borgan Three contributions to the Encyclopedia of Biostatistics: The Nelson-Aalen, Kaplan-Meier, and Aalen-Johansen estimators - Very clear summaries.
Carstensen (2019) Bendex Carstensen. Who needs the Cox model anyway?
Cox (1972) Regression Models and Life-Tables JRSS Vol. 34, No.2, 187-200
Fleming & Harrington (1991). Thomas R. Fleming and David P. Harrington. Counting Processes & Survival Analysis, John Wiley & Sons, Inc. 1991 This text book develops all of the math needed and goes on to study non-parametric models including the Cox model.
Holford. Life table with concomitant information. Biometrics, 32:587{597, 1976.
Mohammed (2019). Introduction to Survival Analysis using R
Threneau (2022). The survival package (vignette)
Therneau et al. (1990). Martingale based residuals for survival models. Biometrika 77, 147-160. Martingale residuals appear to be the one use case where martingales openly surface in the survival calculations.
Whitehead (1980). Fitting Cox’s regression model to survival data using GLIM. Applied Statistics, 29(3):268{275, 1980.
Zhong et al. (2019). Approximating a Cox Model. This is a survival package vignette.

Looking at cash flows

Fri, 02 Sep 2022 00:00:00 +0000

Let’s talk about cash flows!

We continue our series of actuarial and accounting posts where we strive to interest the data science community in using their R skills for the actuarial and accounting applications. We also hope to help some actuaries and accountants with their R skills.

Cash flow is a concept that is important to every business and household. It is difficult to over estimate the importance of disclosing and properly communicating the cash flow as an essential of running any enterprise. This is especially true in times of recession when cash flow is the elephant in the room. Nobody really wants to talk about it, but making cash flow analysis easier to understand and interpret would help almost everyone. To get the maximum impact from a cash flow analysis it is not only important to demonstrate that it has been truly and fairly reported (in accounting terminology), but also to help consumers of cash flow reports intuitively grasp the the meaning of what the numbers are showing.

So what is the problem with the way we normally communicate information about cash flow? Well, look at the image below and check your pulse. How excited did you get? Can you make any sense of it at all with just a flash look?

Actually, it’s incredibly insightful if you know what to look for, but for an non-accounting person (with powers to make decisions), it may not be.

Now look at this example$^1$ of a personal cash flow!

These numbers start talking to me in one insightful flow showing in a second where all the money goes! We all know that a picture is worth a thousand words, but if its millions of dollars, it’s truly priceless.

This type of chart is called a sankey chart. Sankey charts are used to show material, energy and cost flows. You can view some awesome visualizations made with the networkD3 package here. So, showing cash flow is a perfect example of what they were designed for. It is not all common in the actuarial or accounting worlds, but, honestly, it is not that common to see any visualizations in financial reports, - well, apart from happy customers in annual reports which, as we all know, is a marketing thing.

Let’s take a look at how the flow diagram above might be reproduced in R. There are several ways to do this and several packages (for example: networkD3, and diagrammerR) available to help. We will use a very simple dataset, but it could be easily extended to include additional transactional data: different sources of income, types of income, etc. We could also look at inflows and outflows, classify them into types of business activities and automate this tedious, manual task to present a nice chart to our stakeholders.

But. let’s start with … this simple one! And let’s use the networkD3

tidyverse.quiet = TRUE
networkD3.quiet = TRUE
library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──

## ✔ ggplot2 3.3.6     ✔ purrr   0.3.4
## ✔ tibble  3.1.7     ✔ dplyr   1.0.9
## ✔ tidyr   1.2.0     ✔ stringr 1.4.0
## ✔ readr   2.1.2     ✔ forcats 0.5.1

## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

library(networkD3)

Sankey diagrams visualize flows as connections from one node to another. In our case, the flows are distinct types of cash income that will flow into expenses.

For our example, we will use a simple data set with data that has already been classified into source and target.

data <- read_csv("cf_up.csv", show_col_types = FALSE)
head(data)

## # A tibble: 6 × 3
##   source target        value
##   <chr>  <chr>         <dbl>
## 1 Salary Earned Income   494
## 2 Salary Earned Income   677
## 3 Salary Earned Income   758
## 4 Salary Earned Income   933
## 5 Salary Earned Income   649
## 6 Salary Earned Income   825

sources are nodes that initiate the flow and targets are the nodes that finish the flow.

In this particular example, the amount of cash receipts have been matched with a particular cash spending type. In real life, you would not expect this data matching to have already been done for you as most likely your cash flow records just show type of cash flow (cash receipt vs. cash payment) and the amount. We will save the data munging required to transform a typical cash flow dataset into something that can be used to build the Sankey diagram for a future post.

data is a links dataframe with source, target and value columns. For example, $494 of salary was used to pay tax: salary is the earned income and payment of tax is a deduction. In this case the source is the Salary item, the target is Earned income and the value is 494. Earned incomeis also a source node that has Income (i.e. a broader category) target. Income is then distributed to expenses, i.e target category Deduction which is the later becomes a source node for Income tax target.

Here are the unique sources in our data:

unique(data$source)

##  [1] "Salary"                 "Credit Card Reward"     "Dividends"             
##  [4] "Interest"               "Earned Income"          "Passive Income"        
##  [7] "Passive income"         "Income"                 "Deduction"             
## [10] "Core Expenses"          "Financial Independence" "Disposable Income"

And, here are the unique targets.

unique(data$target)

##  [1] "Earned Income"          "Passive Income"         "Passive income"        
##  [4] "Income"                 "Deduction"              "Core Expenses"         
##  [7] "Financial Independence" "Disposable Income"      "Income Tax"            
## [10] "Social Justice"         "Bill Expenses"          "Food"                  
## [13] "Personal Care"          "Transportation"         "Pension"               
## [16] "Investment"             "Real Estate"            "Emergency Fund"        
## [19] "Leisure"

To create a cash flow diagram with the sankeyNetwork() function from the networkD3 package we also need an additional data frame, nodes which contains the list of receipts, types of receipts, types of spending and cash payments.

nodes <- data.frame(
  name=c(as.character(data$source), 
  as.character(data$target)) %>% unique()
)
head(nodes)

##                 name
## 1             Salary
## 2 Credit Card Reward
## 3          Dividends
## 4           Interest
## 5      Earned Income
## 6     Passive Income

Also, because we cannot use nodes names in the function, we need to replace our connections with id, i.e. IDsource and IDtarget

data$IDsource <- match(data$source, nodes$name)-1 
data$IDtarget <- match(data$target, nodes$name)-1

Now, we can draw our cash flow diagram.

p <- sankeyNetwork(Links = data, 
                   Nodes = nodes,
                   Source = "IDsource", 
                   Target = "IDtarget",
                   Value = "value", 
                   NodeID = "name",
                   units = '$',
                   sinksRight=TRUE,
  #                 LinkGroup="source"
                   fontSize = 10, 
                   nodeWidth = 30
                   
                   )

## Links is a tbl_df. Converting to a plain data frame.

There it is! In just a few lines of code you can turn a table that only an expert can read into a visualization that might capture the imagination of someone forced to read a financial document.

References

PPCexpo. How to Create a Cash Flow Chart? Easy to Follow Steps. Available at https://ppcexpo.com/blog/cash-flow-chart , Aug 31, 2022

July 2022: "Top 40" New CRAN Packages

Mon, 29 Aug 2022 00:00:00 +0000

Ninety-four new packages stuck to CRAN in July. Here are my “Top 40” selections in thirteen categories: Climate Modeling, Computational Methods, Data, Ecology, Genomics, Machine Learning, Mathematics, Medicine, Networks, Proteomics, Statistics, Utilities, and Visualization.

Climate Modeling

climetrics v1.0-5: Provides a framework that facilitates the spatio-temporal analysis of climate dynamics through exploring and measuring different dimensions of climate change in space and time. See the vignette.

HurreconR v1.0: Implements the HURRECON model which estimates wind speed, wind direction, enhanced Fujita scale wind damage, and duration of EF0 to EF5 winds as a function of hurricane location and maximum sustained wind speed. See Boose et al. (2001) and Boose et al. (2004) for background, and the vignette for an overview.

TrenchR v0.1.0: Provides tools for translating environmental change into organism response. The biophysical modeling tools include both general models for heat flows and specific models to predict body temperatures for a variety of ectothermic taxa. See Gates (1980) for background. There are vignettes on Allometries and conversions, Estimating microclimates, and Estimating body temperatures.

Computational Methods

elastes v0.1.6: Provides functions to compute functional elastic shape means over sets of open planar curves using a novel approach where planar curves are treated as complex functions. Also, a full Procrustes mean is estimated from the corresponding smoothed Hermitian covariance surface. See Steyer et al. (2022) and Stöcker et. al. (2022) for background and the vignette for examples.

ino v0.1.0: Provides a comprehensive toolbox for comparing different initialization strategies when performing numerical optimization. There is an Introduction and a vignette with an HMM example.

Data

giedata v0.1.0: Provides functions to access the API for GIE, Europe’s Gas Infrastructure database. See README to get started.

Ecology

rgrass v0.3-3: Implements a new interface to the GRASS geographical information system that supports both starting R from within the GRASS environment, or running a free-standing R session in a temporary GRASS location. There is an Introduction and a vignette on object formats.

riverconn v0.3.22: Provides functions to calculate indices for river network connectivity. See Jumani et al. (2021) for a review of the incdices and Baldan et al. (2022) for a list of package capabilities and architecture. Have a look at the vignette for examples.

Genomics

dcifer v1.1.1: Implements Dcifer (Distance for complex infections: fast estimation of relatedness), an identity by descent based method to calculate genetic relatedness between polyclonal infections from biallelic and multiallelic data. See Gerlovina et al. (2022) for the details, and the vignette for an example.

qPCRtools v0.1.1: Provides methods to calculate the amplification efficiency of genes, a crucial step in the qPCR process. See the vignette.

scAnnotate v0.0.4: Implements a data-driven cell type annotation tool for single-cell RNA sequencing data. See Ji et al. (2022) for details and the vignette for an introduction.

Machine Learning

cito v1.0.0: Provides functions for building and training custom neural networks using torch. See the vignette.

etree v0.1.0: Provides functions to implement Energy Trees, statistical models that perform classification and regression with structured and mixed-type data. See Giubilei et al. (2022) for a description of Energy Trees and the vignettes eforest and etree for examples.

LDABiplots v0.1.2: Provides tools to extract, explore, analyze, and visualize news published on the web by digital newspapers using Latent Dirichlet and machine learning algorithms. See Ble et al. (2003) and Gabriel (1971) for background. There are English and Spanish versions of the vignette.

mildsvm v0.4.0: Implements a support vector machine classifier for weakly supervised, multiple instance data. See Kent and Yu (2022) and Muandet et al. (2012) for the theory and README for an example.

nestedcv v0.2.3: Provides functions to perform nested cross-validation as described in Stone (1977) for lasso and elastic-net regularized linear models for glmnet. See the vignette.

Mathematics

delaunay v1.1.0: Provides functions to construct and visualize 2D and 3D Delaunay triangulations. See README for some visual examples.

Medicine

activeAnalyzer v1.0.4: Provides a tool to analyse Actigraphy accelerometer data using PROactive Physical Activity in COPD (chronic obstructive pulmonary disease) instruments. See the vignette for examples.

pedbp v1.0.0: Provides data and utilities for estimating pediatric blood pressure percentiles by sex, age, and height. See Lo et al. (2013) and Martin et al. (2022) for background, and the vignette for an overview.

tipmap v 0.1.7: Implements tipping point analysis for clinical trials using Bayesian dynamic borrowing via robust meta-analytic predictive priors as described in Best et al. (2021).

Networks

ergm.multi v0.1.0: Provides a set of extensions for the ergm package to fit multilayer, multiplex, and multirelational networks as well as samples of multiple networks. See Krivitsky et al. (2020) and Krivitsky et, al. (2022) for the details and the vignette for an extended example.

Families v1.0.1: Provides tools to study kinship networks, grand parenthood, and double burden (presence of children and oldest old parents) in virtual population produced by VirtualPop. See the vignette for examples.

Proteomics

promor v0.1.0: Implements a comprehensive set of tools for label-free proteomics data analysis and machine learning modeling including differential expression analysis, predictive modeling and performance assessment. Data from MaxQuant may be used. See the vignette for an introduction.

Statistics

GPCERF v0.1.0: Provides a non-parametric Bayesian framework based on Gaussian process priors for estimating causal effects of a continuous exposure and detecting change points in the causal exposure response curves using observational data. See Ren et al. (2021). Have a look at the Introduction and the vignettes Full Gaussian Process and Nearest-neighbor Gaussian Process.

lnmCluster v0.3.1: Extends the logistic normal multinomial (LNM) clustering model proposed by Fang and Subedi (2020) to provide LNM clustering for compositional data. Details of model assumptions and interpretation can be found in the papers Tu & Subedi (2021) and Tu & Subedi (2022). See the vignette for an example.

tlars v0.0.1: Provides functions to compute the solution path of the Terminating-LARS (T-LARS) algorithm. See Machkour et al. 2022, Efron et al. (2004) and Tibshirani (1996) for the theory, and the vignette for examples.

Utilities

codebookr v0.1.5: Provides functions to create code books (i.e. data dictionaries) directly from a data frame. See README for examples.

datadictionary v0.1.0: Provides tools to creates a data dictionary from any dataset in an R environment. It includes functions to add variable labels and write to Excel.

dtrackr v0.2.4: Provides tools to track and document dplyr data pipelines. As you filter, mutate, and join your way through a data set, functions seamlessly track data flow and generate publication ready documentation of the data pipeline. See the vignettes Consort statment example, Configuration example, Basic Operation, and Joining data pipelines.

howler v0.2.0: Enables audio interactivity within shiny applications using howler.js. See the vignette.

ixplorer v0.2.2: Provides tools to create and view tickets in gitea, a self-hosted git service, using an RStudio addin. It includes helper functions to publish documentation and use git. There are several vignettes including ixplorer basics and Credential management.

nextGenShinyApps v1.5: Provides responsive tools for designing and developing Shiny dashboards and applications. The scripts and style sheets are based on jQuery and Bootstrap. See the vignette for examples.

nplyr v0.1.0: Provides functions for manipulating nested data frames in a list-column using dplyr without first having to unnest()them. See the vignette.

renderthis v0.1.0: Provides tools to render slides to different formats, including html, pdf, png, gif, pptx, and mp4, as well as tool to make a png file of the first slide of a presentation that is re-sized for sharing on social media. There is an Overview and a vignette on basic usage.

video video: 0.1.0: Enables video interactivity within shiny applications using video.js. See the vignette.

Visualization

ensModelVis v0.1.0: Provides a function to display model fits for multiple models and their ensembles. There are vignettes for Classification and Regression.

ggpackets v0.2.0: Provides tools to create groups of ggplot2 layers that can be easily migrated from one plot to another, reducing redundant code and improving the ability to format many plots that draw from the same source. See the Getting Started Guide and the vignettes Composing Templates & Functions and Misc Examples.

ggseqplot v0.7.2: Provides wrappers to render TraMinR sequence plots in ggplot2. See the vignette.

RcppColors v0.1.1: Provides C++ header files to deal with color conversion from color spaces to hexadecimal with Rcpp, and exports some color mapping functions to R. Look here for examples.

xadmix 1.0.0: Provides functions that provide a quick way of subsetting genomic admixture data and generating customizable stacked barplots. See the vignette.

June 2022: "Top 40" New CRAN Packages

Fri, 29 Jul 2022 00:00:00 +0000

One hundred eighty-nine new packages made it to CRAN in June. Here are my “Top 40” selections in eleven categories: Computational Methods, Data, Ecology, Genomics, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Utilities, and Visualizations.

Computational Methods

itp v1.2.0: Implements the interpolate, truncate, project root-finding algorithm developed by Oliveira & Takahashi (2021). The vignette provides an overview.

QR v0..1.3: Provides a function to perform QR factorization without pivoting to a real or complex matrix. It is based on LAPACK. See the vignette.

qsplines v1.0.0: Provides functions to create quaterion splines. See Barry & Goldman (1988) and Kochanek & Bartels (1984) for the details and look here for an example.

VMDecomp v1.0.1: Implements the variational mode decomposition and two-dimensional variational mode decomposition algorithm. See Dragomiretskiy & Zosso (2014) for background and the vignette for examples.

Data

cmch v0.2.0: Implements a wrapper around the Canadian Mortgage and Housing Corporation web interface and enables programmatic and reproducible access to a wide variety of housing data. See the vignette for examples.

EDIutils v1.0.1: Implements a client for the Environmental Data Initiative repository REST API and provides access to ecological data and metadata. There are five short vignettes: Evaluate & upload, Citation Metrics, Download Metrics, Search andaccess, and Tests.

globaltrends v0.0.12: Provides functions to access global search volumes from the Google Trends portal. This working paper outlines the package’s methodological foundations and potential applications. See the vignette to get started.

kaigiroku v0.5: Allows users to search and download data from the API for Japanese Diet proceedings. Look here for examples.

NasdaqDataLink v1.0.0: Provides functions to interact directly with the Nasdaq Data Link API and obtain data in a number of formats. Look here for API documentation and here for package information.

stortingscrape v0.1.1: Provides functions for retrieving data from the Norwegian Parliament, through the Norwegian Parliament API. See the vingette for an introduction.

Ecology

PointedSDMs v1.0.6: Provides tools to build integrated species distribution models and includes tools to run spatial cross-validation and plotting. See Issac et al. (2020) for and introduction to the methods. There is a Setophaga Example and an example for the Solitary Tinamou.

restoptr v1.0.1: Implements a flexible framework for ecological restoration planning that aims to identify priority areas for restoration efforts using optimization algorithms described in Justeau-Allaire et al. 2021. See the vignette to get started.

Genomics

scapGNN v0.1.1: Implements a single cell active pathway analysis tool based on the graph neural network algorithm described in Scarselli et al. (2009) and Kipf & Welling (2017). This may be used to construct a gene-cell association network, infer pathway activity scores from different single cell modalities data and more. See the vignette for an overview and examples.

SRTsim v0.99.2: Implements an independent, reproducible, and flexible Spatially Resolved Transcriptomics simulation framework that can be used to facilitate the development analytical methods and for a wide variety of SRT-specific analyses. See the vignette.

xQTLbiolinks v1.1.1: Implements tools to query, download, and visualize of molecular quantitative trait locus and gene expression data from public resources through the GTEx API. There is a Quick Start Guide and vignettes on Colocalization, Specivicity, and Visualization.

Machine Learning

agua v0.0.1: Enables users to specify h2o as an engine for several tidymodels modeling methods. See README for examples.

MagmaClustR V1.0.0: Implements two main algorithms, called Magma (Leroy et al. (2022) and MagmaClust (Leroy et al. (2020)), using a multi-task Gaussian processes (GP) model to perform predictions for supervised learning problems. See README for examples.

openai v0.1.0: Provides a wrapper for OpenAI API endpoints including engines, completions, edits, files, fine-tunes, embeddings and legacy searches, classifications, and answers endpoints. See README to get started.

sketching v0.1.0: Provides functions to construct sketches of data via random subspace embeddings. See Lee & Ng (2022) for the theory and the vignette for examples.

webmorphR v0..1.1: Provides functions to create reproducible image stimuli, specialised for face images with psychomorph or webmorph templates. See README to get started.

Mathematics

GeneralizedWendland v0.5-2: Implements the fully parameterized generalized Wendland covariance function for use in Gaussian process models, as well as multiple methods for approximating it via covariance interpolation. The available methods are linear interpolation, polynomial interpolation, and cubic spline interpolation. See Bevilacqua et al. (2022) and the vignette for examples.

jacobi v2.0.0: Evaluates Jacobi theta functions and related functions including the Weierstrass elliptic function, the Weierstrass sigma function, the Weierstrass zeta function, the Klein j-function, the Dedekind eta function, the lambda modular function, Jacobi elliptic functions, Neville theta functions, and the Eisenstein series for real and complex variables. Look here for some images.

Medicine

clinicalsignificance v1.0.0: Implements the clinical significance algorithm proposed by Jacobson et al. (1984) to determine if an intervention has a meaningful practical effect. There is a Getting Started Guide and vignettes on Cutoffs and Plots.

PlatformDesign v1.0.1: Provides functions to calculate design parameters for an optimal two-period, multi-arm platform design allowing pre-planned deferred arms to be added during the trial. See Dunnett (1955) for background and the vignette for some theory and examples.

Statistics

bayesassurance v0.1.0: Provides functions to compute Bayesian assurance under various settings characterized by different assumptions and objectives, including precision-based conditions, credible intervals, and goal functions. See Pan & Banerjee (2021) for the theory. There are vignettes for using closed form solutions, the conjugate linear model, and precision based conditions.

DSSP v0.1.1: Provides functions to draw samples from the direct sampling spatial prior model as described in White, Sun, & Speckman (2019). See the vignette for examples.

edibble v0.1.0: Implements a system to facilitate designing comparative experiments using the grammar of experimental designs. See the edibble-book for documentation.

mixgb v0.1.0: Implements a method for multiple imputation using XGBoost, bootstrapping and predictive mean matching as described in Deng and Lumley (2021). There is an Introduction and a vignette on Imputing new data with a saved imputer.

outerbase v0.1.0: Implements in new method for high-dimensional regression using outer product models. See Plumlee (2014) and Plumlee et al. (2021) for background. There is a Getting started guide, a Base walkthrough, and vignettes on Learning from data and Speeding up inference.

PFIM v5.0: Provides functions to evaluate or optimize designs for nonlinear mixed effects models using the Fisher Information matrix. See Malle & Baccar D (1997) and Retout et al. (2007) for background and the vignettes Design evaluation and optimixation (01), Design evaluation and optimixation (02), and Library of models for examples.

VirtualPop v1.0.2: Provides functions to generate lifespans and fertility histories in continuous time using individual-level state transition (multi-state) models and data. See the vignettes on Simulation of life histories, Sampling from waiting time distributions, Simulation of individual fertility careers, and Validation.

Time Series

kssa v0.0.1: Implements the known sub-sequence algorithm described in Benavides et al. (2022), which helps to automatically identify and validate the best method for missing data imputation in a time series. Look here for examples.

ts2net v0.1.0: Implements methods to transform time series into networks, a technique which may be useful for complex systems modeling, time series data mining, or time series analysis using networks. For an introduction to the topic and descriptions of the methods see Mitchell (2006), Silva & Zhao (2016), and Silva et al. (2021). See README to get started.

Utilities

cppchedkR Allows users to run Cppcheck on C/C++ files as an R command or an RStudio addin. See README. .

gtExtras v0.4.1: Provides additional functions for creating tables with gt. See README for examples.

Visualization

ggpie v0.2.2: Provides functions for creating pie, donut and rose pie plots with ggplot2. See the vignette.

ggtrace v0.2.0: Provides ggplot2 geoms that allow groups of data points to be outlined or highlighted for emphasis. See the vignettes Trace lines and Trace points.

Morphoscape v1.0.0: Implements adaptive landscape methods first described by Polly et al. (2016) for the integration, analysis and visualization of biological trait data on a phenotypic morphospace which are typically defined by shape metrics. See the vignette.

r3js v0.0.1: Provides R and JavaScript functions to allow WebGL-based 3D plotting using the three.js library. See the vignettes: Getting Started, Creating a plot from scratch, and Grouping plot elements.

rgl2gltf v1.0.0: Provides functions to work with glTF files which are used to describe 3D models. See the vignette for examples..

shapviz v0.2.0: Provides functions to visualize SHapley Additive exPlanations (SHAP), such as waterfall plots, force plots, various types of importance plots, and dependence plots. See Lundberg & Lee (2017) for background and the vignette for examples.

Shiny showcase at rstudio::conf(2022)

Wed, 20 Jul 2022 00:00:00 +0000

Beginning with the invitation-only 2016 Shiny Developer Conference, Shiny has played a prominent part in all RStudio conferences, and rstudio::conf(2022) is no exception. Two workshops and eighteen talks showcase Shiny’s multiple, ever-increasing capabilities. What started out as a way to introduce R’s interactive statistical computations to the web has grown into a production-grade tool that supports serious data science workflows and facilitates the communication of data-generated insights throughout large organizations in both industry and government.

Here is your Shiny guide to rstudio::conf(2022). But before you attend the show or see the movie, you may want to have a look at the book.

Keynotes and talks will be livestreamed on the rstudio::conf(2022) website, free and open to all. No registration is required. If you would like, you can sign up to get access to our Discord server to meet and chat with attendees during conf.

On July 27-28th, head to the conference website to watch the livestreams and ask questions alongside other attendees.

Keynotes

The Past and Future of Shiny

Workshops

Talks

R is for actuaRies

Tue, 12 Jul 2022 00:00:00 +0000

Dr Maria Prokofieva is a member of the R / Business working group which is promoting the use of R in accounting, auditing, and actuarial work. She is also a professor at the Victoria University Business School in Australia and works with CPA Australia.

What is actuarial science

Actuarial data science lies at the intersection of math and business studies, combining statistical knowledge and methods from insurance and finance areas. Compared to data scientists, actuaries focus more on finance and business knowledge, while still collecting and analyzing data.

The profession is in high demand, and according to the Bureau of Labor Statistics (BLS), it is expected that actuary jobs will a enjoy 24% increase from 2020-30. This is much faster than the average for all occupations. Moreover, the median salary for an actuary is estimated to be over $100,000.

The focus of the field is on assessing the likelihood of future events, particularly in business settings (especially finance and insurance) to plan for outcomes and mitigate risks. With this in mind, probability analysis and statistics are applied to very many areas, such as predicting the number of children for a health insurance or the payout of the life insurance policy. Some common tasks for actuaries include calculating premium rates for mortality and morbidity products, assessing likelihood of financial loss or return, business risk consulting, pension and retirement planning and many more. Basically, actuaries perform any tasks that include risk modeling, be that in insurance, financial planning or energy and environment. Later in this post, we will go through some examples of those!

Reviewing particular applications across areas, we can mention:

Insurance in such areas as life insurance, credit and mortgage insurance, key person insurance for small business, long term care insurance, health savings accounts. The focus here is on analysis of mortality, production of life tables, calculation of compound interest for life insurance, annuities and endowment contingencies.

# Using imaginator package for individual claim simulation
# extract from the vignette 
set.seed(12345)
tbl_policy <- policies_simulate(2, 2001:2005)
tbl_claim_transaction <- claims_by_wait_time(
  tbl_policy,
  claim_frequency = 2,
  payment_frequency = 3,
  occurrence_wait = 10,
  report_wait = 5,
  pay_wait = 5,
  pay_severity = 50)
kableExtra::kbl(tbl_claim_transaction[1:8,], 
      caption = "Wait-time modelling with policies simulation") %>%
  kableExtra::kable_classic(full_width = F, html_font = "Cambria")

Table 1: Wait-time modelling with policies simulation
policy_effective_date	policy_expiration_date	exposure	policyholder_id	claim_id	occurrence_date	report_date	number_of_payments	payment_date	payment_amount
2001-05-23	2002-05-22	1	1	1	2001-06-02	2001-06-07	3	2001-06-12	50
2001-05-23	2002-05-22	1	1	1	2001-06-02	2001-06-07	3	2001-06-17	50
2001-05-23	2002-05-22	1	1	1	2001-06-02	2001-06-07	3	2001-06-22	50
2001-05-23	2002-05-22	1	1	2	2001-06-02	2001-06-07	3	2001-06-12	50
2001-05-23	2002-05-22	1	1	2	2001-06-02	2001-06-07	3	2001-06-17	50
2001-05-23	2002-05-22	1	1	2	2001-06-02	2001-06-07	3	2001-06-22	50
2001-02-21	2002-02-20	1	2	3	2001-03-03	2001-03-08	3	2001-03-13	50
2001-02-21	2002-02-20	1	2	3	2001-03-03	2001-03-08	3	2001-03-18	50

# Using MortalityTables package - Mortality tables
# extract from the vignette 
mortalityTables.load("Austria_Annuities")
mortalityTables.load("Austria_Census")
plotMortalityTables(
  title="Using dimensional information for mortality tables",
  mort.AT.census[c("m", "w"), c("1951", "1991", "2001", "2011")]) + 
  aes(color = as.factor(year), linetype = sex) + labs(color = "Period", linetype = "Sex")+
  scale_fill_brewer(palette="PiYG")

Life insurance and social insurance. Main tasks in this area include analysis of rates of disability, mordibity, mortality, fertility, etc., analysis of factors (such as georgaphy and consumer characteristics) on usage of medical services and procedures.

# Using MortalityTables package
# extract from the vignette 
mortalityTables.load("Austria_Annuities")
# Get the cohort death probabilities for Austrian Annuitants born in 1977:
qx.coh1977 = deathProbabilities(AVOe2005R.male, YOB = 1977)
# Get the period death probabilities for Austrian Annuitants observed in the year 2020:
qx.per2020 = periodDeathProbabilities(AVOe2005R.male, Period = 2020)
# Get the cohort death probabilities for Austrian Annuitants born in 1977 as a mortalityTable.period object:
table.coh1977 = getCohortTable(AVOe2005R.male, YOB = 1977)
# Get the period death probabilities for Austrian Annuitants observed in the year 2020:
table.per2020 = getPeriodTable(AVOe2005R.male, Period = 2020)
# Compare those two in a plot:
plot(table.coh1977, table.per2020, title = "Comparison of cohort 1977 with period 2020", legend.position = c(1,0))+
  scale_fill_brewer(palette="PiYG")

Pension: design, funding, accounting, administration, and maintenance or redesign of pension plans with valuation and modelling for various factors that may influence the calculation and payout

# Using lifecontingencies package
# Extract from vignette
# Calculation example for the benefit reserve for a deferred annuity-due on a policyholder aged 25 when the annuity is deferred until age 65.
yearlyRate <- 12000
irate <- 0.02
APV <- yearlyRate*axn(soa08Act, x=25, i=irate,m=65-25,k=12)
levelPremium <- APV/axn(soa08Act, x=25,n=65-25,k=12)
annuityReserve<-function(t) {
  out<-NULL
  if(t<65-25) out <- yearlyRate*axn(soa08Act, x=25+t,
                                    i=irate, m=65-(25+t),k=12)-levelPremium*axn(soa08Act,
                                                                                x=25+t, n=65-(25+t),k=12) else {
                                                                                  out <- yearlyRate*axn(soa08Act, x=25+t, i=irate,k=12)
                                                                                  }
  return(out)
  }
years <- seq(from=0, to=getOmega(soa08Act)-25-1,by=1)
annuityRes <- numeric(length(years))
for(i in years) annuityRes[i+1] <- annuityReserve(i)
dataAnnuityRes <- data.frame(years=years, reserve=annuityRes)
dataAnnuityRes%>%ggplot(aes(years, reserve))+
  geom_line(color="blue")+
  labs(x = "Years", y = "Amount",
       title ="Deferred annuity benefit reserve")

and many more! Just look at those examples!

Historically, actuarial science comes from the mathematical area (no surprise here!) and stems back to 17th century when developments in mathematics in Europe coincided with the growth in demands to have more precise estimation in risks in burial, life insurance and annuities calculations. This was the start of developing techniques for life tables (hail to John Graunt, the father of demography) and discounting values for present value calculation which is one of the keystone concepts in today’s finance and accounting. The field was developing very quickly but the computational side became more and more complex: without computers, this was a tedious task. But…

Why acturial science needs R

Long gone those days when all calculations were done by hand, table and god blessing… What is in use currently in the field?

Apparently… Excel still has its dominance (if not reign?) …

In March 2022 the Casualty Actuarial Society (CAS) released results and analysis from its first survey on technology used by actuaries. With responses from over 1,200 participants, the results suggest that more than 94.3% of respondents use Excel, and at least once a day, but! it is more than just one tool! With this in mind there is a strong demand for new skills- in R (47.2%), Python (39.1%) and SQL (30.8%).

To support this, the recent results (2020) of the “wasted time” survey show a striking outcome that actuaries waste most of their time…

waiting for excel to process the results
copying data from a model
recreating prior model values
waiting for vendor support

and quite a few of other tasks that could be easily resolved with open source and replicable modeling approach..

survey<-read_csv("survey_actuaries.csv")
survey%>%pivot_longer(-Tasks, names_to="Time", values_to="Score")%>%
   group_by(Tasks) %>% 
   mutate(sum_response = sum(Score))%>%
  ungroup()%>%
  ggplot(aes(x=fct_reorder(Tasks, sum_response), Score, fill=Time))+
  geom_col(show.legend = TRUE)+
  coord_flip()+
  scale_fill_brewer(palette="PiYG")+
  labs(y = "Responses", x = "Tasks",
title ="Data and model management tasks - wasted time")

What are the barriers? TIME! (80.5% of actuaries indicated so).. but what else is missing?

Learning resources

Learning resources is always a problem to move people to use available tools. Actuarial (data) science is not the exception. Good news is that there are some to start using R for actuarial data as well as there are some materials available to build further resources.

The available learning resources start with Introduction to R. This is a general category that includes the resources to get familiar with R, RStudio and main R packages (e.g. tidyverse).

R for Data Science

RStudio.cloud Primers

A well known list built by the enthusiastic and supportive R community: RStudio Education education.rstudio.com

In the meantime, there is a growing treasure trove of R resources developed by enthusiastic R Actuaries that address specific issues.

The work of the “Data Science” group of the Swiss Association of Actuaries (SAA) is amazing - lots of resources has been prepared, including lectures and hands-on tutorials in R.

Common and specific R packages used

Apart from educational resources, specialized packages are available to the community, including:

Such packages are scattered across CRAN and github as there is not currently a task view for actuarial data science at CRAN (hint!)…

There is a great potential in the actuarial data science and R / Business would love to invite the R passionate community to start a dialogue on promoting and developing R resources in this area!

May 2022: "Top 40" New CRAN Packages

Tue, 28 Jun 2022 00:00:00 +0000

One hundred seventy-nine new packages made it to CRAN in May. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Ecology, Epidemiology, Finance, Machine Learning, Networks, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

graDiEnt v1.0.1: Implements the derivative-free, optim-style Stochastic Quasi-Gradient Differential Evolution optimization algorithm published in Sala, Baldanzini, and Pierini (2018) that uses population members to build stochastic gradient estimates. See README for an example.

rxode2 V2.0.7: Provides facilities for running simulations from ordinary differential equation models, such as pharmacometrics and other compartmental models, but requires both C and Fortran compilers. See the vignette for an example.

ScaleSpikeSlab v1.0: Provides a scalable Gibbs sampling implementation for high dimensional Bayesian regression with the continuous spike-and-slab prior described in Biswas, Mackey & Meng (2022). See README for an example.

Data

bluebike v0.0.3: Provides functions that facilitate importing and working with the Boston Blue Bike Data Set including functions to compute trip distances and map the locations of stations within a given radius. See the vignette for examples.

eurodata v1.4.2: Implements an interface to Eurostat’s Bulk Download Facility with fast data.table based import of data, labels, and metadata along with data search and data description and comparison functions. See README to get started.

gbifdh v0.1.2: Implements a high performance interface to the Global Biodiversity Information Facility that supports large-scale analyses using SQL or dplyr operations on complete tables. See the vignette for examples.

getwiki v0.9.0: Implements a simple wrapper for Wikipedia data to retrieve text in a tidy format that can be used for Natural Language Processing. See the vignette.

Ecology

FIESTA v3.4.1: Implements an estimation tool for analysts that work with sample-based inventory data from the U.S. Department of Agriculture, Forest Service, Forest Inventory and Analysis Program. There are nine vignettes including manuals for Module Estimates and Population Data, and Small Area Estimators and Spatial Tools.

soiltestcorr v2.1.2: Provides functions designed to assist users on the correlation analysis of crop yield and soil test values including functions to estimate crop response patterns to soil nutrient availability and critical soil test values using various approaches. See Correndo et al. (2017), Cate & Nelson (1971), Anderson & Nelson (1975), Bullock & Bullock (1994) and Melsted & Peck (1977) for background. There are seven vignettes including an Introduction and Quadratic-plateau response.

sspm v0.9.1: Implement a gam-based spatial surplus production model, aimed at modeling northern shrimp population in Atlantic Canada but potentially to any stock in any location. There is a vignette on Package and Workflow design and another that provides an example with simulated data.

Epidemiology

EpiInvert v0.1.1: Inverts a renewal equation to estimate time-varying reproduction numbers and restored incidence curves with festive days and weekly biases corrected as described in Alvarez et al. (2021) and Alvarez et al. (2022). See README for examples.

linelist v0.0.1: Provides tools to help storing and handling case line list data by adding a tagging system to classical data.frame objects to identify key epidemiological data. See the vignette for examples.

Finance

markets v1.0.3: Provides functions to estimate markets in equilibrium and disequilibrium based on full information maximum likelihood techniques given in Maddala and Nelson (1974) and implemented using the analytic derivative expressions calculated in Karapanagiotis (2020). There is an Overview providing theory and code and vignettes on Model initializion details, Market-clearing assessment, and Use cases.

portvine v1.01: Provides portfolio level risk estimates including value at Risk and Expected Shortfall following the approach described in Sommer (2022) by modeling each asset with an ARMA-GARCH model and then modeling their cross dependency via a Vine Copula in a rolling window fashion. See the vignette to get started.

usincometaxes v0.4.0: Implements a wrapper to the NBER’s TAXSIM 35 tax simulator. TAXSIM 35 to calculate federal and state income taxes. There are vignettes on Uploading Data, Input Columns, Output Columns, and Calculating Taxes.

Genomics

MixviR v3.3.5: Implements tools for exploring DNA and amino acid variation and inferring the presence of target lineages from microbial high-throughput genomic DNA samples that potentially contain mixtures of variants/lineages. MixviR was originally created to help analyze environmental SARS-CoV-2/Covid-19 samples from environmental sources such as waste water or dust, but can be applied to any microbial group. See DePristo et al. (2011) and Danecek et al. (2021) for background, and the vignette for examples.

simer v0.9.0.0: Implements a data simulator including genotype, phenotype, pedigree, selection and reproduction for animals and plants and provides data for genomic gelection, genome-wide association, and breeding studies. See Kao and Zeng (2002) and Ripley (1987) for background. Look here for extensive documentation.

Machine Learning

fastTopics v0.6-135: Implements fast, scalable optimization algorithms for fitting topic models and non-negative matrix factorization for count data. The methods exploit the special relationship between the multinomial topic model (probabilistic latent semantic indexing) and Poisson non-negative matrix factorization. See the vignettes: Relationship between NMF and topic modeling and Topic mideling vs. clustering.

metrica v1.2.3: Provides functions to evaluate prediction performance of point-forecast models accounting for different aspects of the agreement between predicted and observed values including error metrics, model efficiencies, indices of agreement, goodness of fit, concordance correlation, and error decomposition, and plots the visualized agreement. See the vignettes on Available Metrics and Model Assessment.

SparseVFC v0.1.0: Implements The sparse vector field consensus algorithm described in Ma et al. (2013) for robust vector field learning. See the vignette.

Networks

Rwclust v0.0.1: Implements the random walk clustering algorithm for weighted graphs as found in Harel and Koren (2001). See the vignette.

Science

EvoPhylo v0.1: Provides functions to support automated morphological character partitioning for phylogenetic analyses, and analyses of macroevolutionary parameter outputs. See Simões and Pierce (2021) for background. There is a vignette on Theoretical Background, and there are others on Character Partitioning, FBD parameters, and Evolutionary Rates.

sits v1.0.0: Provides an end-to-end toolkit for land use and land cover classification using big Earth observation data, based on machine learning methods applied to satellite image data cubes, as described in Simoes et al (2021). Builds regular data cubes from collections in AWS, Microsoft Planetary Computer, Brazil Data Cube, and Digital Earth Africa using the STAC protocol and the gdalcubes package. An eBook provides extensive documentation.

Statistics

biosensors.usc v1.0: Provides a framework for using distributional representations of biosensor data such as ECG, medical imaging or fMRI data in various statistical modeling tasks: regression models, hypothesis testing, cluster analysis, visualization, and descriptive analysis. See Matabuena et al. (2021) for background and the vignette for an introduction.

CopSens v0.1.0: Implements the copula-based sensitivity analysis method for observational causal inference discussed in Zheng et al. (2022). See README for examples.

GeoModels v1.0.1: Provides functions to analyze Gaussian and Non Gaussian (bivariate) spatial and spatio-temporal data and simulate random fields using likelihood methods. See Bevilacqua and Gaetan (2015) and Vallejos et al. (2020) for background, and look here for examples.

nlmixr2 v2.0.6: Fit and compare nonlinear mixed-effects models with flexible dosing information commonly seen in pharmacokinetics and pharmacodynamics using differential equations solved by compiled C code provided in the rxode2 package. See Almquist et al. (2015) and Wang et al. (2015) for background, and the vignette for examples.

stdmod v0..1.7.1: Provides functions for computing a standardized moderation effect in moderated regression and forming its confidence interval by nonparametric bootstrapping as proposed in Cheung et al. (2002). There are six vignettes including a Quick Start Guide and a vignette on Standardized Moderation.

Time Series

forceR v1.0.15: Initially written and optimized to deal with insect bite force measurements, the functions in this package can be used to clean and analyze any time series. They provide a workflow to load, plot and crop data, correct amplifier and baseline drifts, identify individual peak shapes, rescale (normalize) peak curves, and find best polynomial fits to describe and analyze force curve shapes. See the vignette for examples.

ZINARp v0.1.0: Provides functions for simulation, exploratory data analysis and Bayesian analysis of p-order integer-valued autoregressive, INAR(p), and zero-inflated p-order integer-valued autoregressive, ZINAR(p), processes, as described in Garay et al. (2020).

Utilities

async v0.2.1: Provides functions for writing sequential-looking code that pauses and resumes similarly to generator and async constructs from Python or JavaScript. Objects produced are compatible with the iterators and promises packages. See the vignette for an example.

chronicler v0.2.0: Provides tools to decorate a function so that it returns its along with a log detailing when the function was run, what were its inputs, what were the errors (if the function failed to run) and other useful information. See the vignettes: A non-mathematician’s introduction to monads, The Maybe monad, and A real world example.

jpgrid v0.2.0: Provides functions to generate Japanese grid square codes from longitude, latitude and geometries. See README for examples.

partialised v0.1.0: Provides a partialised class that extends the partialising function of purrr by making it easier to change the arguments. This is similar to the function-like object in `Julia’. See README for an example.

shinybrowser v1.0.0: Provides information about shiny app users including browser name and version, device type (mobile or desktop), operating system and version, and browser dimensions. See README for more information.

webshot2 v0.1.0: Takes screenshots of web pages, including Shiny applications and R Markdown documents using a headless Chrome or Chromium browser as the browser back-end. See README for examples.

Visualization

accrualPlotv1.0.1: Implements accrual plots for clinical trials. See the vignette.

ggbraid v0.2.2: Implements stat_braid(), that extends the functionality of geom_ribbon() to correctly fill the area between two alternating lines (or steps) with two different colors, and geom_braid(). Three vignettes, US Supreme Court, NBA Finals Game, and Average Daily Temperatures provide examples.

ggisotonic v0.1.2: Provides stat_isotonic() to add weighted univariate isotonic regression curves. See README for an example.

UpSetVP v1.0.0: Uses the ideas of variance partitioning and hierarchical partitioning as described in Lai et al. (2022) to visualize the unique, common, or individual contribution of each predictor (or matrix of predictors) towards explaining variation. Look here for examples.

Frank's R Workflow

Fri, 17 Jun 2022 00:00:00 +0000

Frank Harrell’s new eBook, R Workflow, which aims to: “to foster best practices in reproducible data documentation and manipulation, statistical analysis, graphics, and reporting” is an ambitious document that is notable on multiple levels.

To begin with, the workflow itself is much more than a simple progression of logical steps.

This workflow is clearly the result of a process forged through trial and error by a master statistician over many years. As the diagram indicates, the document takes a holistic viewpoint of a statistical analysis covering document preparation, data manipulation, statistical practice computational concerns, and more.

Then, there is the synthesis of a wide range of content into a succinct, very readable exposition that dips in to some very deep topics. Frank’s examples are streamlined presentations of analyses and code that are both sophisticated an practical. The missing value section suggests a whole array of analyses through a careful presentation of plots, and the section on data checking introduces a level of automation beyond what is commonly done.

Frank’s writing style is clear, informal and from the perspective of a teacher who wants to show you some cool things along with the basics. For example, don’t miss the if Trick in section 2.4.3.

I should mention that Frank’s eBook is not a tidyverse presentation. The code examples are built around base R, Frank’s Hmisc and rms packages and an eclectic mix of packages that include data.table. plotly and tidyverse packages haven and ggplot2. In a way, this selection of packages reflects the evolution of R itself. For example, as with many popular R packages, Hmisc most likely started out as Frank’s personal tool kit. However, after many years of Frank’s deep commitment to using R and contributing R tools, which includes seventy versions of Hmisc in nineteen years, the package has become a fundamental resource. (Have a look at the reverse depends, imports, and suggests.) Also, the mix of packages with different design philosophies underlying R Workflow reflects the flexibility of the R language and the organic growth of the R ecosystem.

Perhaps the most striking aspect of the eBook is the way Frank uses Quarto, knitr and Hmisc to build an elegant reproducible document about building reproducible documents. For example, Quarto permits the effective placement of plots in the right margins of the document, and the Quarto callouts in Section 3.4 enable the mini tutorials that include Special Considerations for Latex/pdf and Using Tooltips with Mermaid to be embedded in the document without interrupting its flow. Moreover, along with functions like Hmisc::getHdata() and Hmisc::getRs(), Quarto enables the document to achieve a high level of reproducibility by pulling data and code directly from GitHub repositories.

Not only can Frank’s R Workflow teach you some serious statistics, but studying its construction will take you a long way towards building aesthetically pleasing reproducible documents.

Frank Harrell will be delivering a keynote address on August 26th at the upcoming R/Medicine conference.

April: "Top 40" New CRAN Packages

Mon, 30 May 2022 00:00:00 +0000

Computational Methods

hmer v1.0.1: Provides objects and functions for Bayes Linear emulation and history matching, including functions for automated training of emulators, diagnostic functions to ensure suitability, and a variety of methods for generating waves of points. There is a vignette Demo of the package, an Emulation and History matching Handbook, and vignettes on Examples, and Stochastic Emulation.

rminqa v0.1.1: Implements a wrapper for the C++ function bobyqa to perform derivative-free optimization algorithms in R.

rolog v0.9.4: Embeds SWI-Prolog, so that R can send deterministic and non-deterministic queries to Prolog. See README for examples.

torchopt v0..1.1: Implements optimizers for the torch deep learning that are not among the optimizers offered in torch. These include:adabelief by Zhuang et al (2020), adabound by Luo et al.(2019), adamw by Loshchilov & Hutter (2019), madgrad by Defazio and Jelassi (2021), nadam by Dozat (2019), qhadam by Ma and Yarats (2019), radam by Liu et al. (2019), swats by Shekar and Sochee (2018), and yogi by Zaheer et al.(2019)

villager v1.1.1: Provides a set of base classes with core functionality to allow users to create and run Agent Based Models. There are vignettes on Extending Agents and Extending Resources.

Data

baseballr v1.2.0: Provides numerous utilities for acquiring and analyzing baseball data from online sources such as Baseball Reference, FanGraphs, and the MLB Stats API. See the Getting Started Guide and the vignettes NCAA Scraping and Plotting Statcast data.

toRvik v1.0.2: Provides a suite of functions to quickly scrape and tidy advanced metrics, detailed player and game statistics, team and coach histories, and more from Barttorvik. See the vignette.

ustfd v0.1.0: Lets users make requests from the US Treasury Fiscal Data API endpoints. See README for an example.

valet v0.9.0: Implements a client for the recently updated Bank of Canada Valet API. See README for examples.

Ecology

OBIC v2.0.1: Provides functions to calculate the Open Boden Index method used in the Netherlands to evaluate the quality of soils of agricultural fields and evaluate the sustainability of the current agricultural practices. There are vignettes on the Open soil index, Score aggregation, and Workability.

timbeR v2.0.1: Provides functions to estimate wood volumes, for example, number of logs, diameters along the stem and heights at which certain diameters occur. See Weiskittel, A. (2021) for background and the vignette for an introduction.

Finance

AssetAllocation v1.0.0: Provides functions to implement customizable asset allocation strategies and automatically download data from Yahoo Finance. See the vignette.

multilateral v1.0.0: Implements multilateral price index calculations focused on time product dummy regression and GEKS variations, and allows for extension of the methods through automatic window splicing. See Krsinich (2016) for information on window splicing and the vignette for examples.

Machine Learning

clusterHD Provides tools for clustering high dimensional data as described in Raymaekers and Zamar (2020) and Raymaekers and Zamar (2020).

tglkmeans v0.3.4: Efficiently implements the Kmeans algorithm. See Arthur and Vassilvitskii (2007) and Ostrovsky et al. (2013) for background and the vignette to get started.

Mathematics

hilbert v0.2.1: Provides utilities for encoding and decoding coordinates to and from Hilbert curves based on the iterative encoding implementation described in Chen et al. (2006). See the vignette to get started.

gellipsoid v0.7.2: provides functions to represent degenerate and unbounded generalized geometric ellipsoids together with methods for linear and duality transformations, and for plotting. The ideas are described in Friendly, Monette & Fox (2013). See README for examples.

Medicine

crossnma v1.0.1: Provides functions for cross-design and cross-format Network Meta-Analysis Regression as described in Hamza et al. 2022. See the vignette.

exact.n v1.0.0: Allows the user to determine minimum sample sizes that achieve target size and power at a specified alternative. See Lloyd & Ripamonti (2021) for the theory and README for examples.

stoppingrules v0.1.1: Provides functions for creating, displaying, and evaluating stopping rules for safety monitoring in clinical studies including stopping rule methods described in Goldman (1987), Geller et al. (2003), Ivanova, Qaqish, & Schell (2005), and Kulldorff et al. (2011). See README for an example.

Networks

incidentally v0.9.0: Provides functions to generate random incidence matrices and bipartite graphs under different constraints or using different generative models. See the vignette.

networkscaleup v0.1-1: Provides a variety of network scale-up models to analyze aggregated relational data, including models from Laga et al. (2021) Zheng et al. (2006), Killworth et al. (1998), and Killworth et al. (1998). See the vignette.

pald v0.0.1: Implements the partitioned local depths algorithm described in Berenhaut, Moore, & Melvin (2022) which may be helpful in determining both local and global structure in data. Look here to get started.

Statistics

bmstdr v0.1.4: Provides functions to fit, validate, and compares a number of Bayesian models for spatial and space-time point referenced and areal unit data. See the vignette for theory and examples.

cubble v0.1.0: Implements a spatiotemperal data object in a relational data structure to separate the recording of time variant and invariant variables. See the vignettes: aggregation, design, cubble, import, and matching.

incubate v1.1.8: Fits parametric models to time-to-event data that show an initial incubation period, i.e., a variable phase where the hazard is zero. The delayed Weibull distribution serves as the foundational data model. Look here for an example.

pspatreg v1.0.2: Provides functions to estimate and analyze spatial and spatio-temporal semiparametric models including spatial or spatio-temporal non-parametric trends, parametric and non-parametric covariates with a possible spatial lag for the dependent variable, and temporal correlation in the noise. See Basile et al. (2014), Rodriguez-Alvarez et al. (2015), and especially Minguez et al. (2020) for background. There is an Introduction, and vignettes on Cross-sectional data and Spatial panel data.

sfdep v0.1.0: Provides and interface to spdep to integrate sf objects and the tidyverse. There are vignettes on The Basics, Conditional Permutations, and spdep and pysal.

smile v1.0.4.1: Provides functions to estimate, predict, and interpolate areal data. For estimation and prediction, areal data are assumed to be an average of an underlying continuous spatial process as in Moraga et al. (2017), Johnson et al. (2020), and Wilson and Wakefield (2020). There are vignettes on Fitting Models, Areal Interpolation, Converting to spm, Spatial Covariance, and Method.

SpatialPOP v0.1.0: Provides functions to generate a spatial population from a spatially varying regression model under the assumption that observations are collected from a uniform two-dimensional grid with unit distance between any two neighboring points. See Chao et al. (2018) for method details and the vignette for an example.

Time Series

lite v1.0.0: Performs likelihood-based inference for stationary time series extremes following the approach of Fawcett and Walshaw (2012). Marginal extreme value inferences are adjusted for cluster dependence using the methodology of Chandler and Bate (2007). See the vignette.

spooky v1.1.0: Uses the Discrete Fast Fourier Transformation to extrapolate time features beyond their boundaries. Look here for examples and references.

Utilities

chromote v0.1.0: Implements the Chrome DevTools Protocol for controlling a headless Chrome web browser. Look here for examples.

chkptstanr v0.1.1: Implements a framework to checkpoint Bayesian models fit with Stan and brms. The MCMC sampler can be stopped and then restarted where it left off. There is a vignette for brms and another for Stan.

ivs v0.1.0: Implements a new interval vector class for generic interval manipulations including locating various kinds of relationships between two interval vectors, merging overlaps within a single interval vector, splitting an interval vector on its overlapping endpoints, and applying set theoretical operations on interval vectors. The package was inspired by Allen (1983). See the vignette for an introduction.

listr v0.0.2: Pools for common operations on lists such as selecting and merging data stored in lists which can be used with pipes. See the vignette.

shinyGizmo v0.1: Provides UI components and input widgets for Shiny applications to apply non-standard operations and address performance issues. See README for examples.

shinytest2 v0.1.0: Provides automated unit testing of Shiny applications through a headless Chromium browser. There is a Getting Started Guide and seven additional vignettes including Testing in depth, Robust testing, and Monkey testing.

thaipdf v0.1.2: Provides R Markdown templates and aLaTeX preamble to create PDFs from R Markdown documents in the Thai language. See the vignette for examples.

Visualization

ggtrendline v1.0.3: Enhances ggplot2 with tools to add a trendline with a confidence interval for linear or nonlinear regression models and show the equation. See Ritz and Streibig (2008) and Greenwell and Schubert Kabban (2014) for background and look here for examples.

Skimming #rstats on Twitter

Fri, 13 May 2022 00:00:00 +0000

Even when filtering by the relatively sober #rstats hashtag, I find twitter to be the stream of consciousness of an undisciplined collective mind: disjoint and ephemeral. Nevertheless, on any given day some useful R resources float by, and it is frequently the case that interesting items disappear downstream before I can record them. Here are a few I did manage to fish out recently.

On May 12, @AedinCulhane announced that Bioconductor is seeking new members for its advisory board. Read about the positions, the process and make your nominations here.

@epiRhandbook celebrated the one year anniversary of the Epi R Handbook.

@Physacourses tweeted about the reactablefmtr package which allows for the creation of interactive data tables in R.

On May 10, @tobigerstenberg posted the online book of his notes along with slides for the graduate level Psych Stats class he teaches at Stanford.

On May 9, @mdsumner called out @carroll_jono’s post Where for (loop) ARt Thou?.

On May 8, @eddelbuettel announced the new r2u: R Binaries for Ubuntu repo and demo of a full brms installation in 13 seconds .

On May 7, @robinlovelace updated Geographic data in R, a book on how to get started with R for geographic research.

@RosanaFerrero pointed to Missing-data imputation from Gellman and Hill’s book and the FES 720 Introduction to R on the importance of missing values.

On May 5, @sharon000 pointed to @rlmcelreath’s Statistical Rethinking repo with links to free video lectures.

March: "Top 40" New CRAN Packages

Thu, 28 Apr 2022 00:00:00 +0000

Two hundred and six new packages stuck to CRAN in March. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Finance, Game Theory, Genomics, Machine Learning, Medicine, Networks, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

RCDT v1.1.0: Provides functions to perform 2D Delaunay triangulation, constrained or unconstrained, with the help of the CDT C++ library. See the vignette.

rlemon v0.2.0: Provides access to the LEMON C++ graph library. Look here for a list of algorithms.

Data

ag5Tools v0.0.1: Offers tools for downloading and extracting data from the Copernicus Agrometeorological indicators from 1979 to present derived from reanalysis (AgERAS) dataset. See ag5Tools to get started and the vignette on extracting data.

AirMonitor v0.2.2: Provides utilities for working with hourly air quality monitoring data with a focus on small particulates (PM2.5) along with algorithms to calculate NowCast and the associated Air Quality Index (AQI) as defined by the US Environmental Projection Agency AirNow program. There is an Introduction, a Developers Style Guide, and a Data Model.

BISdata v0.1-1: Provides access to data from the Bank for International Settlements in Basel. Look here for an example.

Finance

AFR v0.1.0: Provides tools for regression, prediction and forecast analysis of macroeconomic and credit data adapted for banking sector of Kazakhstan for bank analysts and non-statisticians. There are vignettes on Data transformatiom, Diagnostic Tests, and Regression.

fixedincome v0.0.1: Implements objects that abstract interest rates, compounding factors, day count rules, forward rates and term structure of interest rates to assist with calculations of interest rates and fixed income. Look here for examples.

Game Theory

socialranking v0.1.1: Offers a set of solutions to rank players based on a transitive ranking between coalitions, including CP-Majority, ordinal Banzhaf or lexicographic excellence solution summarized Allouche et al. (2020). See the vignette.

Genomics

AquaBPsim v0.0.1: Provides tools to simulate breeding programs including functions to simulate production and reproduction systems encountered in aquaculture. See the vignette for details.

coda4microbiome v0.1.1: Provides tools for microbiome data analysis that take into account its compositional nature including functions for variable selection for both, cross-sectional and longitudinal studies, and for binary and continuous outcomes. See the vignette

MitoHEAR v0.1.0: Provides functions for the estimation and downstream statistical analysis of the mitochondrial DNA Heteroplasmy calculated from single-cell datasets. See the vignette.

ZetaSuite v1.0.0: Provides functions to score hits from two-dimensional RNAi screens analyze single cell transcriptomics to differentiate rare cells from damaged ones. See Vento-Tormo et al. (2018) for background and the vignette for examples.

Machine Learning

RGAN v0.1.1: Implements the Generative Adversarial Nets algorithm described in Goodfellow et al. (2014). See README for examples.

sentiment.ai v0.1.1: Implements Sentiment Analysis via tensorflow deep learning and gradient boosting models and also allows users to create embedding vectors for text which can be used in other analyses. See the vignette for an example.

sentopics v0.6.2: Offers a framework that joins topic modeling and sentiment analysis of textual data, and implements a fast Gibbs sampling estimation of Latent Dirichlet Allocation. See Griffiths & Steyvers (2004) and the Joint Sentiment/Topic Model of Lin, Everson & Ruger (2012). There is a vignette on Bascic Usage and another on Topical time series.

transforEmotion v0.1.0: Provides access to sentiment analysis using the Python based huggingface transformer zero-shot classification model pipelines. The default pipeline is Cross-Encoder’s DistilRoBERTa trained on the Stanford Natural Language Inference and Multi-Genre Natural Language Inference datasets. There is a vignette on setting up Python.

Medicine

adaptr v1.0.0: Simulates adaptive clinical trials using adaptive stopping, adaptive arm dropping, and adaptive randomization. There is an Overview and vignettes on Basic and Advanced examples.

EVI v0.1.1-4: Implements the epidemic volatility index, a novel early warning tool for identifying new waves in an epidemic as described in Kostoulas et al. (2021). See the vignette.

rts2 v0.3: Provides functions to support modelling case data for real-time surveillance of infectious diseases including functions to generate a computational grid over an area of interest and approximate log-Gaussian Cox Process model. See Diggle et al. (2013) and Solin and Särkkä (2020) for background and look here for examples.

Networks

BCDAG v1.0.0: Provides functions for structure learning of causal networks and estimation of joint causal effects from observational Gaussian data. See Castelletti & Mascaro (2021) and Castelletti & Mascaro (2022) for background, and the vignettes Random Data Generation, Output of learn_DAG(), and MCMC scheme for posterior inference for details.

SEset v1.0.1: Implements tools to compute and analyze the set of statistically-equivalent Gaussian, linear path models which generate the input precision or (partial) correlation matrix. See README for examples.

Science

bdc v1.1.0: Provides functions for biodiversity data cleaning organized into five themes: Merging datasets, Pre-filtering, Taxonomy, Space (Flagging low precision coordinates), and Time (flagging inconsistent data collection dates). There are vignettes on Standardization, Pre-filter, Space, Taxonomy, and Time.

CSHShydRology v1.2.1: Offers a collection of user submitted functions to aid in the analysis of hydrological data. See the vignette.

Statistics

cheem v0.2.0: Provides functions to explore local explanations of non-linear models by first calculating the tree SHapley Additive exPlanation for every observation and for calculating a projection basis, and then changing the basis with a radial tour. See Lundberg et al. (2019), Spyrison & Cook (2020) and Cook and Buja (2012) for background and the vignette to get started.

CondCopulas v0.1.2: Provides functions for the estimation of conditional copulas models, various estimators of conditional Kendall’s tau statistic as proposed in Derumigny and Fermanian 2019a, 2019b and 2020. See the vignette for examples.

multilevelmod v0.1.0: Implements bindings for hierarchical regression models for use with the parsnip package. Models include longitudinal generalized linear models as described in Liang and Zeger (1986) and mixed-effect models as described in Pinheiro and Bates (2000). See the vignette for examples.

rbmi v1.1.3: Implements reference based multiple imputation allowing for the imputation of longitudinal datasets using predefined strategies. These include conventional MI methods, conditional mean imputation methods, and bootstrapped MI methods. See the Scope section of the Statistical Specifications vignette for more information on MI methods. There is a Quick Start Guide and an additional vignette on Advanced Functionality.

remiod v1.0.0: Implements reference-based multiple imputation of ordinal and binary responses under Bayesian framework, as described in Wang and Liu (2022). See the vignette.

rlcv v1.0.0: Provides functions to estimate likelihood cross-validation bandwidth for uni- and multi-variate kernel densities which are robust with respect to fat-tailed distributions and outliers. See Wu (2019) for the theory and the vignette for an example.

sftime v0.2-0: Provides classes and methods for spatial objects that have a registered time column, in particular for irregular spatiotemporal data. See the vignette.

workboots v0.1.1: Provides functions for generating bootstrap prediction intervals from a tidymodels workflow. There is a Getting Started Guide and a vignette on estimating linear prediction intervals.

Time Series

dtts v0.1.0: Provides high-frequency time-series support via nanotime and data.table. See README for examples.

ngboostForecast v0.0.2: Implements probabilistic time series forecasting via natural gradient boosting for probabilistic prediction. See README for an example.

svines v0.1.4: Provides functions to fit and simulate from stationary vine copula models for time series. See Nagler et al. (2022) for the theory and look here for examples.

Utilities

dyn.log v0.4.0: Implements dynamic, configuration driven logging. There are vignettes on Configuration, Formats, Layouts, and Levels.

formatters v0.2.0: Provides a framework for rendering complex tables to ASCII, and a set of formatters for transforming values or sets of values into ASCII-ready display strings. See the vignette for examples.

git4r v0.1.2: Implements an interactive git user interface from the R command line that includes tools to make commits, branches, remotes, and diffs an integrated part of R coding. See the vignette.

Visualization

ggmice v0.0.1: Provides functions to enhance a mice imputation workflow with visualizations for incomplete and imputed data including functions to inspect missing data, develop imputation models, evaluate algorithmic convergence, and compare observed versus imputed data. See the Getting Started Guide and the vignette Old Firends.

langevitour v0.2: Implements an HTML widget that uses Langevin dynamics to show random walks through 2D projections of numerical data. It can be used from within R, or included in a self-contained Rmarkdown document. See the vignette for an examples.

picker v0.2.6: Provides functions to zoom, pan, and pick points from a deck.gl scatterplot and includes tooltips, labels, a grid overlay, legends, and coupled interactions across multiple plots. See README for examples.

MLDataR - Real-world Datasets for Machine Learning Applications

Tue, 19 Apr 2022 00:00:00 +0000

This is a guest post from Gary Hutson, lead of Machine Learning at Crisp Thinking, a company that provides AI solutions to moderate and detect offensive and abusive content online. His website is available at https://hutsons-hacks.info/ and he can be reached through Twitter, @StatsGary.

MLDataR package motivation

I love all things Machine Learning. The MLDataR package was driven by the need to have example datasets across the healthcare system for machine learning problems. I have been a machine learning practitioner for over nine years; however, I still find it interesting to explore new examples and datasets related to supervised machine learning classification and regression.

Because the package contains clinical examples and examples from real hospital systems, it allows the potential machine learning engineer to practice all things related to supervised machine learning.

Despite the package initially being aimed at healthcare, I have expanded it to new territories and domains.

What does the package contain?

The package contains several datasets for modelling. This is just a start and I am working with the NHS-R community to build it out even further. It is a sort of a call to arms to equip the package with even more examples of excellent datasets that can be used for machine learning.

Diabetes disease prediction - This dataset contains key variables, gathered from hospital research and papers in the British Medical Journal to identify the drivers behind diabetes disease. This dataset is useful for working with supervised classification machine learning problems or statistical problems. It uses past historical patient information to train and classify a model, with the aim to classify if a patient will have diabetes when they first present to the service.

Diabetes early onset - Gathered by Asif Laldin, the package contributor from Gloucestershire Clinical Commissioning Group, this dataset contains information on the time between a prediabetes diagnosis and the onset of diabetes.

Failing care home prediction - Using measures from the NHS incident reporting databases, this dataset contains data to classify if a care home will fail based on results from retrospective inspections.

Heart disease prediction - This dataset is intended for supervised machine learning classification problems on which patients are likely to present with heart disease. This uses independent variables, such as resting blood pressure, maximum heart rate, history of angina, and other metrics.

Thyroid disease classification - This one has a personal effect on me, as I am a sufferer of this disease. This drove me to source this dataset from the Garavan Institute, based on a collection of studies this institute did around thyroid disease. This contains 28 independent or predictor variables of patients with or without the disease. It is also covered in the vignette supporting this package and the supporting YouTube tutorial using tidymodels with various ML techniques.

Counter Strike Global Offensive (CSGO) - This dataset was kindly contributed by Asif Laldin, and is a detraction from the healthcare datasets in the package. I never intended the package to be purely healthcare ML datasets, and I plan to include credit card fraud examples, tabular playground examples from Kaggle, and many more, so watch this space…

Run a tidymodels routine with heart disease dataset

Let’s explore the heart disease dataset contained in MLDataR using a logistic regression model.

# install.packages("MLDataR")
library(MLDataR)
library(dplyr)
library(tidyr)
library(tidymodels)
library(data.table)
library(ConfusionTableR)
library(OddsPlotty)
glimpse(MLDataR::heartdisease)

## Rows: 918
## Columns: 10
## $ Age              <dbl> 40, 49, 37, 48, 54, 39, 45, 54, 37, 48, 37, 58, 39, 4…
## $ Sex              <chr> "M", "F", "M", "F", "M", "M", "F", "M", "M", "F", "F"…
## $ RestingBP        <dbl> 140, 160, 130, 138, 150, 120, 130, 110, 140, 120, 130…
## $ Cholesterol      <dbl> 289, 180, 283, 214, 195, 339, 237, 208, 207, 284, 211…
## $ FastingBS        <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ RestingECG       <chr> "Normal", "Normal", "ST", "Normal", "Normal", "Normal…
## $ MaxHR            <dbl> 172, 156, 98, 108, 122, 170, 170, 142, 130, 120, 142,…
## $ Angina           <chr> "N", "N", "N", "Y", "N", "N", "N", "N", "Y", "N", "N"…
## $ HeartPeakReading <dbl> 0.0, 1.0, 0.0, 1.5, 0.0, 0.0, 0.0, 0.0, 1.5, 0.0, 0.0…
## $ HeartDisease     <dbl> 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0,…

A glimpse into the dataset gives us a quick overview of our dataset. We have 918 rows and 10 columns. We can see our outcome, heart disease, and our nine predictors.

There are a couple of things we want to clean up. Notice that our outcome variable HeartDisease loads as a double variable. We want to convert it into a factor variable for our machine learning model.

The variables Sex, RestingECG, and AnginaY are character variables. For creating models, it is better to encode characters as factors.

hd <- heartdisease %>%
  mutate(across(where(is.character), as.factor),
         HeartDisease = as.factor(HeartDisease)) %>% 
  # Remove any non complete cases
  na.omit()
is.factor(hd$HeartDisease)

## [1] TRUE

For Machine Learning models, it is generally recommended to split your data into a training and testing set, or if you are using hyperparameter tuning and updating your model, a training / test and validation set. Other methods are available, such a K-Fold Cross Validation; however, we will stick to a basic training and testing split for the purposes of this walkthrough.

To do this, and to make sure that the results are repeatable, we will use the set.seed(123) value - which essentially says when we are randomly splitting this data, make sure that the random pattern is the same as the walkthrough., i.e. give me the same split as this post:

set.seed(123)
split_prop <- 0.8
testing_prop <- 1 - split_prop
split <- rsample::initial_split(hd, prop = split_prop)
training <- rsample::training(split)
testing <- rsample::testing(split)
# Print a custom message to show the samples involved
training_message <- function() {
  message(
    cat(
      'The training set has: ',
      nrow(training),
      ' examples and the testing set has:',
      nrow(testing),
      '.\nThis split has ',
      paste0(format(100 * split_prop), '%'),
      ' in the training set and ',
      paste0(format(100 * testing_prop), '%'),
      ' in the testing set.',
      sep = ''
    )
  )
}
training_message()

## The training set has: 734 examples and the testing set has:184.
## This split has 80% in the training set and 20% in the testing set.

##

We can fit a parsnip model to the training set and then we can evaluate the performance.

lr_hd_fit <- logistic_reg() %>%
  set_engine("glm") %>% 
  set_mode("classification") %>% 
  fit(HeartDisease ~ ., data = training)

If we want to see the summary results of our model in a tidy way (i.e., a data frame with standard column names), we can use the tidy() function:

tidy(lr_hd_fit)

## # A tibble: 11 × 5
##    term             estimate std.error statistic  p.value
##    <chr>               <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)       0.925     1.29        0.718 4.73e- 1
##  2 Age               0.0175    0.0123      1.42  1.55e- 1
##  3 SexM              1.17      0.241       4.84  1.29e- 6
##  4 RestingBP        -0.00164   0.00544    -0.301 7.63e- 1
##  5 Cholesterol      -0.00335   0.00104    -3.23  1.25e- 3
##  6 FastingBS         0.944     0.248       3.81  1.39e- 4
##  7 RestingECGNormal -0.291     0.261      -1.12  2.64e- 1
##  8 RestingECGST     -0.383     0.343      -1.12  2.64e- 1
##  9 MaxHR            -0.0192    0.00450    -4.28  1.88e- 5
## 10 AnginaY           1.54      0.229       6.72  1.78e-11
## 11 HeartPeakReading  0.689     0.113       6.12  9.54e-10

We can see the statistically significant ones by pulling out those with p < 0.05: Male (SexM), cholesterol (Cholesterol), fasting blood sugar (FastingBS), maximum heart rate (MaxHR), having angina (AnginaY), and peak heart rate reading (HeartPeakReading).

tidy(lr_hd_fit) %>% 
  filter(p.value < 0.05) %>% 
  pull(term)

## [1] "SexM"             "Cholesterol"      "FastingBS"        "MaxHR"           
## [5] "AnginaY"          "HeartPeakReading"

Let’s convert the probabilities from the fitted GLM model into odds ratios (ORs). An OR measures the association between an exposure and an outcome.¹ In this case, the OR represents the odds of heart disease will occur given a particular condition, compared to the odds of heart disease occurring in the absence of that condition.

We can visualize the results using the OddsPlotty package and the fit list object from the model.

tidy_oddsplot <- OddsPlotty::odds_plot(
  lr_hd_fit$fit,
  title = "Heart Disease Odds Plot",
  point_col = "#6b95ff",
  h_line_color = "red"
)
tidy_oddsplot <- tidy_oddsplot$odds_plot +
  theme(legend.position = "none") +
  geom_text(
    label = round(tidy_oddsplot$odds_plot$data$OR, digits = 3),
    hjust = -0.5,
    vjust = 1,
    cex = 2.8
  )
tidy_oddsplot

We can also pull out the odds ratio data:

tidy_oddsplot$data

##                      OR  lower  upper             vars
## Age              1.0176 0.9935 1.0426              Age
## SexM             3.2156 2.0172 5.2026             SexM
## RestingBP        0.9984 0.9877 1.0090        RestingBP
## Cholesterol      0.9967 0.9946 0.9987      Cholesterol
## FastingBS        2.5710 1.5912 4.2114        FastingBS
## RestingECGNormal 0.7472 0.4471 1.2452 RestingECGNormal
## RestingECGST     0.6817 0.3474 1.3348     RestingECGST
## MaxHR            0.9809 0.9722 0.9895            MaxHR
## AnginaY          4.6768 2.9989 7.3842          AnginaY
## HeartPeakReading 1.9914 1.6056 2.4982 HeartPeakReading

In this example, odds ratios are used to compare the relative odds of the occurrence of heart disease, given exposure to the variable of interest. Looking at the results from odds_plot():

Having angina (AnginaY), peak heart rate reading (HeartPeakReading), Male (SexM), and fasting blood sugar (FastingBS) have an OR of greater than 1, implying that these conditions are associated with higher odds of heart disease. For example, people with Angina are 4.677 times more likely to get heart disease than those without this condition.
Normal resting electrocardiogram (RestingECGNormal) and normal ST segment of an electrocardiogram (RestingECGST) have an OR of less than 1, implying that these conditions are associated with lower odds of heart disease, however due to the length of the error bars these conditions do not have as significant effect on heart disease, as those outlined with odds greater than 1.

Note the confidence intervals, indicating the precision of the OR. The large CIs around RestingECGNormal and RestingECGST indicate a low level of precision of the OR. The small CIs around Age, RestingBP, Cholesterol, and MaxHRindicate a higher precision of the ORs. This is normally due to the representation of the encoded items within the model, i.e. the presence of an effect code.

I evaluate the outputs of logistic regression models by using the sampling probability values that the variables have been selected by chance (p-values) for the cut off, but then use the odds ratios of the effect of the predictor variable on my outcome, using the odds plots to make the final decision.

Evaluating with testing set

The final way we can evaluate how well our model fits, before pushing this into production, would be to use a confusion matrix to collect how well our testing partition performs, against our training model predictions. Here we are trying to get a sense of, if we pushed this into the wild, how well would it do on unseen observations i.e. those new items that we don’t have a label for, a label in this sense is whether someone has had heart disease, or not.

library(ConfusionTableR)
# Use our model to predict labels on to testing set
predictions <- cbind(predict(lr_hd_fit, new_data = testing),
                     testing)
# Create confusion matrix and output to record level for storage to monitor concept drift
cm <- ConfusionTableR::binary_class_cm(
  predictions$.pred_class,
  predictions$HeartDisease,
  mode = 'everything',
  positive = '1'
)

## [INFO] Building a record level confusion matrix to store in dataset

## [INFO] Build finished and to expose record level cm use the record_level_cm list item

The next step is to expose the confusion matrix to view how well our model did on estimating the labels on the testing set:

# Access the confusion matrix list object
cm$confusion_matrix

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  0  1
##          0 70  6
##          1 16 92
##                                         
##                Accuracy : 0.88          
##                  95% CI : (0.825, 0.924)
##     No Information Rate : 0.533         
##     P-Value [Acc > NIR] : <2e-16        
##                                         
##                   Kappa : 0.758         
##                                         
##  Mcnemar's Test P-Value : 0.055         
##                                         
##             Sensitivity : 0.939         
##             Specificity : 0.814         
##          Pos Pred Value : 0.852         
##          Neg Pred Value : 0.921         
##               Precision : 0.852         
##                  Recall : 0.939         
##                      F1 : 0.893         
##              Prevalence : 0.533         
##          Detection Rate : 0.500         
##    Detection Prevalence : 0.587         
##       Balanced Accuracy : 0.876         
##                                         
##        'Positive' Class : 1             
##

Our model did relatively well. Picking apart some of the metrics:

True positives (TP) we have 92 correctly classified instances of heart failure
True negatives(TN) we have 70 cases classified as not having heart disease
False negatives (FN) we have 6 cases were our model said the patient didn’t have heart disease and they did
False positives (FP) we have 16 cases were our model said a patient did have heart disease, but they actually didn’t
Recall (also called Sensitivity) is 0.9388 meaning from all the patients that had heart disease - how many did we predict correctly - this equation is Recall=TP / (TP + FN)
Precision (also called Positive Predictive Value) is 0.8519 meaning from all the classes we predicted as positive, how many were actually positive. The equation here is Precision=TP/(TP + FP).

As I run many ML experiments, I wanted a way to store this data into a record-level extract. I actually stored this in a model metrics table on Postgres SQL database, but could be stored in any proprietary system. To get these metrics easily in record level, the package I created helps you do that with ease:

# As we used the binary record level cm method, this stores the model data down
record_level_cm <- cm$record_level_cm %>%
  dplyr::mutate(user_name = Sys.getenv("USERNAME"))
glimpse(record_level_cm)

## Rows: 1
## Columns: 24
## $ Pred_0_Ref_0         <int> 70
## $ Pred_1_Ref_0         <int> 16
## $ Pred_0_Ref_1         <int> 6
## $ Pred_1_Ref_1         <int> 92
## $ Accuracy             <dbl> 0.8804
## $ Kappa                <dbl> 0.7581
## $ AccuracyLower        <dbl> 0.8246
## $ AccuracyUpper        <dbl> 0.9235
## $ AccuracyNull         <dbl> 0.5326
## $ AccuracyPValue       <dbl> 4.902e-24
## $ McnemarPValue        <dbl> 0.05501
## $ Sensitivity          <dbl> 0.9388
## $ Specificity          <dbl> 0.814
## $ Pos.Pred.Value       <dbl> 0.8519
## $ Neg.Pred.Value       <dbl> 0.9211
## $ Precision            <dbl> 0.8519
## $ Recall               <dbl> 0.9388
## $ F1                   <dbl> 0.8932
## $ Prevalence           <dbl> 0.5326
## $ Detection.Rate       <dbl> 0.5
## $ Detection.Prevalence <dbl> 0.587
## $ Balanced.Accuracy    <dbl> 0.8764
## $ cm_ts                <dttm> 2022-04-19 11:18:51
## $ user_name            <chr> ""

#Export to csv
data.table::fwrite(record_level_cm, file = 'heart_disease_cm_record_level.csv')

If you were happy with your model now, you could put this into production.

Can I contribute my own dataset to MLDataR?

The answer is you can, and I would greatly encourage it. To boot, you will become a package contributor. I am looking for ML datasets from across a wide range of industries and organisations.

Suitable datasets for machine learning applications:

Have sufficient predictive variables for feature engineering
Have a nominal outcome variable
May have missing values
Consist of interesting features

If you have an idea, please submit a pull request to the GitHub repository and add your dataset.

Final thoughts

I have really enjoyed putting this package together and I hope you can use it to:

Learn tidymodels or caret. I have put together a few tutorials on these in the past:

Building a tidymodels classification model from scratch: https://www.youtube.com/watch?v=hxRx7ozLNKw&t=2583s
Advanced modelling with caret for supervised machine learning: https://www.youtube.com/watch?v=rO40vvKXU-4&t=3085s
Reticulate - R and Python a happy union: https://www.youtube.com/watch?v=8WE-EU5k97Q&t=235s
Collapsing a caret confusion matrix with ConfusionTableR: https://youtu.be/9zcUlgLySZo

Put your models into production:

Deploying a caret machine learning model as an API with Plumber: https://youtu.be/WMCkV_J5a0s
Creating a microservice with Docker and serving as a restful API: https://youtu.be/JK6VLAKRjO4

Szumilas M. (2010). Explaining odds ratios. Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l’Academie canadienne de psychiatrie de l’enfant et de l’adolescent, 19(3), 227–229.↩︎

A Macroeconomics Dashboard on Turkey Inflation

Thu, 07 Apr 2022 00:00:00 +0000

Enes Gencer is a lead data scientist at Doktar and a graduate of Tilburg University (Research Master, Economics). Enes’ interests include data science problems in Economics, Finance, and Agriculture.

Introduction

Inflation is a hot topic for both globally and in Turkey nowadays. Inspired by Rami Krispin, I reverse engineered his U.S. Electricity dashboard and created a dashboard to explore a forecast for inflation in Turkey. In this post, I would like to begin to describe some of the technology that went into creating the dashboard.

The data and machine learning pipelines are automated via Docker, Github Actions, and R Markdown. The dashboard has too many dimensions to cover in a single blog post so in this blog post, I focus on the underlying economics and the forecasting method I choose, Elastic Net. In the coming blog posts, I might write on different aspects like flexdashboard, Docker, or a more interesting forecasting method like LSTM using Keras on R.

Getting Started

First, we load the required libraries and data set.

library(tidyverse)
library(lubridate) 
library(caret)
library(zoo)
library(ggthemes)
library(magick)

load("processed_data.Rdata")

Dataset

The data set consists of monthly macroeconomic variables. The dependent variable is the Y-o-Y change in CPI, and the independent variables are lags of

Price indexes
Tendency surveys of the economic agents
Production statistics
Balance of payments statistics
Exchange rate return
Month (to take into account seasonality)
CPI Forecast of TBATS model (inspired by the stacking methods)

First, I only included lags to avoid hindsight bias or the “knew-it-all-along” phenomenon. For instance, as of today, the model predicts March inflation. However, March statistics are not announced yet. If the model is constructed to use March statistics, then it has no value in real-life prediction.

Second, the analysis does not cover the effect of interest rates on inflation. That is only because I have a pre-written script to pull monthly variables. Therefore, I do not want to spend time adjusting the frequency. Also, the effect of the interest rate is known anyway. Central banks shift the price of money (or cost of borrowing) by changing interest rates, which is the primary tool to manage inflation. A rise in interest rate has two effects: first, it makes borrowing more costly, so people defer their consumption. Second, it might reduce the present value of financial assets. Hence, creating a negative wealth effect. Both impacts cause the inflationary pressures to ease.

We have the training data:

forecast_date = "2022-03-31"

train_data <- forecast_df %>% 
  filter(Date < forecast_date)

tail(train_data,3)

## # A tibble: 3 × 94
##   Date         CPI CPI_Lag1 CLI_Lag1 Domestic_PPI_Lag1 Inflation_Expectation_La…
##   <date>     <dbl>    <dbl>    <dbl>             <dbl>                     <dbl>
## 1 2021-12-31 0.361    0.213     102.             0.546                      15.6
## 2 2022-01-31 0.487    0.361     102.             0.799                      21.4
## 3 2022-02-28 0.544    0.487     101.             0.935                      25.4
## # … with 88 more variables: FS_Confidience_Lag1 <dbl>,
## #   RS_Confidience_Lag1 <dbl>, Production_Volume_Lag1 <dbl>,
## #   Export_Orders_Lag1 <dbl>, BoP_Lag1 <dbl>, Utilization_Rate_Lag1 <dbl>,
## #   Consumer_Confidience_Lag1 <dbl>, Import_Annual_Ret_Lag1 <dbl>,
## #   Import_Monthly_Ret_Lag1 <dbl>, UsdTry_Annual_Ret_Lag1 <dbl>,
## #   UsdTry_Monthly_Ret_Lag1 <dbl>, CPI_Lag2 <dbl>, CLI_Lag2 <dbl>,
## #   Domestic_PPI_Lag2 <dbl>, Inflation_Expectation_Lag2 <dbl>, …

And the test data:

test_data <- forecast_df %>% 
  filter(Date  == forecast_date)

head(test_data)

## # A tibble: 1 × 94
##   Date         CPI CPI_Lag1 CLI_Lag1 Domestic_PPI_Lag1 Inflation_Expectation_La…
##   <date>     <dbl>    <dbl>    <dbl>             <dbl>                     <dbl>
## 1 2022-03-31    NA    0.544     101.              1.05                      24.8
## # … with 88 more variables: FS_Confidience_Lag1 <dbl>,
## #   RS_Confidience_Lag1 <dbl>, Production_Volume_Lag1 <dbl>,
## #   Export_Orders_Lag1 <dbl>, BoP_Lag1 <dbl>, Utilization_Rate_Lag1 <dbl>,
## #   Consumer_Confidience_Lag1 <dbl>, Import_Annual_Ret_Lag1 <dbl>,
## #   Import_Monthly_Ret_Lag1 <dbl>, UsdTry_Annual_Ret_Lag1 <dbl>,
## #   UsdTry_Monthly_Ret_Lag1 <dbl>, CPI_Lag2 <dbl>, CLI_Lag2 <dbl>,
## #   Domestic_PPI_Lag2 <dbl>, Inflation_Expectation_Lag2 <dbl>, …

Modelling

I evaluated multiple machine learning algorithms: Elastic Net, generalized linear models, Random Forest, and Support Vector Machines. I decided that Elastic Net has crucial advantages. First, there is a very high correlation between the explanatory variables. Notice that I included multiple lags of a given variable. Also, two different variables might have associations through multiple macroeconomic channels. Hence, multi-collinearity has to be taken into account. Second, I prefer an interpretable model to validate the model results with stylized macroeconomic facts. Third, in this case, it has higher performance (in terms of RMSE and MAPE) compared to more complex models like Support Vector Machines or Random Forest. Therefore, Elastic Net provides me a sweet spot for interpretability, performance, and speed.

So, what is Elastic Net? It is a regularized regression technique that linearly combines L_1 and L_2 penalties. Basically, it is a combination of LASSO and Ridge regressions. In other words, it sets a group of the coefficient to zero and shrinks the remaining ones towards zero.

Mathematically, Elastic Net optimizes the following equation:

$\min_{\beta_0,\beta} \frac{1}{N} \sum_{i=1}^{N} w_i l(y_i,\beta_0+\beta^T x_i) + \lambda\left[(1-\alpha)\|\beta\|_2^2/2 + \alpha \|\beta\|_1\right]$

Visual comparison between LASSO, Ridge, and Elastic Net penalties:

magick::image_read(path = "elastic-net.png")

Source: Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net.

# Construct time slices for time-series cross validation

time_slices <- trainControl(
  method = "timeslice",
  initialWindow = 36,
  fixedWindow = TRUE,
  horizon = 3,
  savePredictions = TRUE,
  verboseIter = FALSE
)

# Train the model

elastic_net_fit <- train(
  CPI ~ .,
  data = train_data[,-1],
  na.action = "na.pass",
  method = "glmnet",
  preProcess = c("center", "scale"),
  trControl = time_slices
)

I use time-series cross-validation with a fixed window size to tune the hyper-parameters and evaluate the model performance. My idea is that the inflation dynamics might evolve. For instance, we have observed the increasing effect of the exchange rate on inflation in recent years. However, as a result of the commodity super-cycle, we might see the producer price index come into play. Therefore, having a more flexible model structure with respect to time makes sense.

In this case, the model is trained on the first 36 observations and evaluated on the 3 consecutive values on the test set. Then, the time slice shifts one month.

To learn more details on time-series cross-validation, please visit: https://robjhyndman.com/hyndsight/tscv/

Results

First, I get the best hyper parameters and collect the predictions of the selected model.

# Visualize results

lasso_preds %>% 
  rename(Actual = CPI,
         Forecasted = pred) %>% 
  gather(key = "Variable", value = "Value", Actual:Forecasted) %>% 
  ggplot(aes(x = Date, y = Value, color = Variable))+
  geom_line(size = 0.75, alpha = 0.5)+
  geom_point(size = 0.75, alpha = 0.75)+
  scale_color_manual(values = c("steelblue4", "darkred"))+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)+
  scale_x_date(breaks = seq(min(lasso_preds$Date), 
                            max(lasso_preds$Date), 
                            by="4 months"),
             date_labels = '%m-%y')+
  labs(x = "",
       y = "",
       subtitle = "Based on YoY CPI changes",
       title = "One-month ahead inflation forecast",
       color = "")+
  expand_limits(y = 0)+
  theme(plot.subtitle = element_text(face = "italic"))

The plot shows a one-month ahead inflation forecast and actual inflation based on the YoY CPI changes.

lasso_preds %>% 
  ggplot(aes(x = pred, y = CPI))+
  geom_point(size = 1.5, alpha = 0.8, color = "darkred")+
  geom_abline(intercept = 0, 
              slope = 1, 
              size = 0.3, 
              linetype = 3, 
              alpha = 0.8, 
              color = "steelblue4")+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)+
  scale_x_continuous(labels = scales::percent)+
  expand_limits(y = 0, x = 0)+
  labs(title = "Model Prediction & True Value",
       x = "Model Prediction",
       y = "True Value")

The second plot shows the predictions vs. the real values. You would expect to see the points are piled up around the 45-degree line. The model seems consistent in its predictions except for the three outliers, which will become sounder in the following plot.

lasso_preds %>% 
  mutate(Residual = CPI - pred) %>% 
  ggplot(aes(x = Date, y = Residual))+
  geom_point(size = 1.5, alpha = 0.8, color = "steelblue4")+
  geom_ribbon(stat='smooth', se=TRUE, alpha=0.05) +
  geom_line(stat='smooth', alpha=1, color = "darkred")+
  theme_bw() +
  labs(y = "Residuals",
       x = "",
       title = "Elastic Net Model Residuals",
       subtitle = "Over time")+
  scale_y_continuous(labels = scales::percent)

The third plot shows the model residuals over time. Notice that the model performance improves over time (until 2021), the smoothed residual curve gets closer to zero, and the frequency of outliers decreases. However, since 2021, the model performance has started to deteriorate caused by economic shocks. A different dynamic that cannot be taken into account comes into play.

var_names <- coef(elastic_net_fit$finalModel, 
                  elastic_net_fit$finalModel$lambdaOpt) %>% 
  rownames()

var_values <- coef(elastic_net_fit$finalModel, 
                   elastic_net_fit$finalModel$lambdaOpt) %>% 
  as.numeric()

model_result <- tibble(Variable_Names = var_names,
                       Coefficients = var_values)

model_result %>% 
  mutate(Variable = gsub('[[:digit:]]+', '', 
                         str_remove(model_result$Variable_Names, 
                                    "_Lag"))) %>% 
  group_by(Variable) %>% 
  summarise(Overall_Effect = sum(Coefficients)) %>% 
  filter(!Overall_Effect == 0) %>%
  filter(!Variable == "(Intercept)") %>% 
  mutate(If_Positive = ifelse(Overall_Effect >0, "Positive", "Negative")) %>% 
  mutate(Sum = sum(abs(Overall_Effect))) %>% 
  mutate(Percent_Effect = Overall_Effect / Sum) %>% 
  ggplot(aes(x = reorder(Variable, abs(Overall_Effect)), y = abs(Percent_Effect), fill = If_Positive))+
  geom_col(alpha = 3)+
  scale_y_continuous(labels = scales::percent)+
  expand_limits(y = c(0,0.45))+
  coord_flip() +
  labs(y = "",
       x = "",
       fill = "",
       title = "The weighted effects macroeconomic variables on Turkey's Inflation",
       subtitle = "Shows the overall effect through different lags", 
       caption = "Source: CBRT")+
  theme_bw()+
  scale_fill_wsj()

The final plot shows the weighted effect of macroeconomic variables on Turkey’s inflation. Notice two things. First, the values on the x-axis are not coefficients, but the share of each variable in total effect. Second, it shows the overall impact coming through multiple lags. Beyond the predictive abilities, the findings are consistent with macroeconomic facts. Remember that we want to interpret the model results. First, the highest share of CPI confirms the well-known “inertia’ phenomenon in inflation.”Inertia" refers to a situation in which prices in the whole economy adjust with the change in the price index. Therefore, it creates a self-sustaining loop.

Secondly, the effect of annual changes in import prices follows CPI. It is known as one of the most significant determinants of Turkey’s inflation. It is mainly because of the structure of the Turkish economy and production. Inflation expectations have the third-highest effect.

Theoretically, economic agents process all the information available constitutes expectation about the future and maximizes their utility. When the economic agents believe that future inflation rises, and behave accordingly, it turns into a self-fulfilling prophecy. Hence, it shows the importance of shaping the economic agents’ beliefs and expectations for a policy-maker.

Finally, the domestic producer price index is a variable with a relatively higher weight, which is consistent with the “pass-through effect”. The producer price index is more volatile due to its higher sensitivity towards shocks, however, it is expected that the producer and consumer price indexes move together in the long run. Therefore, in a relatively short-term analysis (which covers six months), we observe a partial “pass-through” effect.

Conclusion

To sum up, I focus on the underlying economics and machine learning algorithm used to forecast inflation. The elastic net algorithm is chosen considering the three factors: interpretability of model results, time efficiency, and model performance. The model results are consistent with the stylized facts of Turkey’s macroeconomics and underlying economic theory.

Beyond economics and theory, however, there is still a lot to uncover. The data and machine learning pipelines are automated via Github Actions and Docker. The dashboard is constructed with R Markdown and flexdashboard. Hopefully, I will discuss them in the coming posts with different dashboards. Till then, you can always reach me if you have any questions.

Note: for more information on the code and data used in this post, please see the author’s Github repository.

February 2022: "Top 40" New CRAN Packages

Mon, 28 Mar 2022 00:00:00 +0000

February was a good month for new R packages on CRAN. Here are my “Top 40” selections from the two hundred packages that arrived in thirteen categories: Computational Methods, “Data”, Genomics, Linguistics, Machine Learning, Mathematics, Medicine, Networks, Science, Sports, Statistics, Utilities, Visualization.

Computational Methods

fastadi v0.1.0: Implements the adaptive-impute matrix completion algorithm described in Cho, Kim & Rohe (2016). See README for details.

invertiforms v0.1.0: Provides composable invertible transforms for sparse matrices. See README for examples.

mirai v0.1.1: Provides a simple and lightweight method for concurrent and parallel code execution, built on NNG, Nanomsg Next Gen technology. See README for examples.

Data

amerifluxr v1.0.0: Provides a programmatic interface to the AmeriFlux database and includes query, download, and data summary tools. There are vignettes on Data Import and Site Selection.

finnishgrid v0.1.0: Implements an API client for Fingrid Open Data on the electricity market and the power system. See the vignette for an introduction.

fixtuRes v0.1.3: Provides functions to generate mock data in R using YAML configurations. See the vignette.

jpstat v0.2.0: Provides tools for using the e-Stat API, a portal site for Japanese government statistics, and includes functions for automatic query generation, data collection and formatting. See README for examples.

Gemomics

fcfdr v1.0.0: Provides functions to implement the Flexible cFDR (Hutchinson et al. (2021)) and Binary cFDR (Hutchinson et al. (2021)) methodologies to leverage auxiliary data from arbitrary distributions, for example functional genomic data, with GWAS p-values to generate re-weighted p-values. There is an Introducion and there are vignettes on LDAK and TID

simphony v1.0.0: Provides a tool for simulating rhythmic data: transcriptome data using Gaussian or negative binomial distributions, and behavioral activity data using Bernoulli or Poisson distributions. See Singer et al. (2019) for details, and the vignettes on simphony’s options and evaluating rhythm detection for examples.

Linguistics

glottospace v0.0.111: Provides streamlined workflows for geolinguistic analysis, including: accessing global linguistic and cultural databases, data import, data entry, data cleaning, data exploration, mapping, visualization and export. See README to get started.

Machine Learning

aum v2022.2.7: Uses a standard template library sort to implement an efficient algorithm for computing AUM, Area Under Min(FP, FN), and directional derivatives. See Hillman & Hocking (2021) for details and the vignettes Accuracy comparison and Speed comparison.

familiar v1.0.0: Provides an unified interface for end-to-end automated machine learning and model evaluation. There is an Introduction, and five additional vignettes including Evaluation and explanation and Feature selection methods.

OptHoldoutSize v0.1.0.0: Provides tools to estimate the size of a holdout set and the associated errors when updating predictive scores. See Haidar-Wehbe et al. (2022) for details and the vignettes Comparison of algorithms, ASPRE Example, and Simulated Example.

soundClass v0.0.9.1: Implements a sound classification workflow with functions to automatically classify sound events using convolutional neural networks. See Gibb et al. (2019), Mac Aodha et al. (2018), and Stowell et al. (2019) for background and the vignette for an example.

Mathematics

gmpoly v1.1.0: Provides symbolic calculation, addition, multiplication, and evaluation of multivariate polynomials with rational coefficients. See README for examples.

gyro v0.2.0: Implements functions for three dimensional hyperbolic geometry based on the theory found in Ungar (2005). The short vignette points to resources to get you started.

sumR v0.4.6: Implements functions based on theoretical results which ensure that the summation of an infinite discrete series is within an arbitrary margin of error of its true value. See Braden (1992) for background.

Medicine

admiral v0.6.3: Implements a toolbox for programming CDISC compliant Analysis Data Model ADaM datasets in R in accordance with the Analysis Data Model Implementation Guide. There are seven vignettes including Creating ADSL and Creating a BDS Exposure ADaM.

baker v1.0.0: Provides functions to specify, fit and visualize nested partially-latent class models for inference of population disease etiology and individual diagnosis. See Wu et al. (2015), Wu et al. (201). and Wu & Chen (2020) for background and the vignette for examples.

MicroMB v0.0.12: Implements a framework based on S3 dispatch for constructing models of mosquito-borne pathogen transmission which are constructed from submodels of various components. Consistent mathematical expressions for the distribution of bites on hosts enables stochastic and deterministic models to be coherently incorporated and updated over a discrete time step. There are nine vignettes including the Ross-Macdonale mosquito model, transmission model, and Blood feeding.

musclesyneRgies v1.1.3: Provides a framework to factorize electromyography data including tools for raw data pre-processing, non negative matrix factorization, classification of factorised data and plotting of obtained outcomes. There are vignettes on analysis, plots, pro tips, and workflow.

ravetools v0.0.3: Implements signal processing tools for analyzing electrophysiology data including a fast, memory-efficient Notch-filter, Welch-periodogram, and a discrete wavelet transform algorithm for hours of high-resolution signals. See the RAVE Project and Magnotti et al. (2020) for background and README for examples.

wildmeta v0.1.0: Implements single coefficient tests and multiple-contrast hypothesis tests of meta-regression models using cluster wild bootstrapping, based on the methods examined in Joshi et al. (2021). The vignette provides examples.

Networks

leidenbase v0.1.9: Implements an R to C/C++ interface that runs the Leiden community detection algorithm to find a basic partition. It includes the required source code files from the official leidenalg distribution and functions from the R igraph package. See the vignette for an example.

netropy v0.1.0: Provides functions to conduct a statistical entropy analysis of network data as introduced by Frank & Shafie (2016). There are vignettes on Joint Entropies, Prediction Power, Uni, Bi & Trivariate Entropies, and Data Editing.

Science

IceSat2R v1.0.1: Implements an interface to to the OpenAltimetry ICESat data through the API allowing users to download and process Global Geolocated Photon Data, Land Ice Height, Sea Ice Height, Land and Vegetation Height and more. There are vignettes on IceSat-2 Atlases, Mission Orbits, and Virtual File System Orbits.

panstarrs v0.1.0: Implements an interface to the API for Pan-STARRS1, a data archive of the PS1 wide-field astronomical survey which allows access to the PS1 catalog and to the PS1 images. See the vignettes Cat and Images.

ThermalSampleR v0.1.0: Implements a range of simulations to aid researchers in determining appropriate sample sizes when performing critical thermal limits studies including a number of wrapper functions are provided for plotting and summarizing outputs from these simulations. There is both a vignette and a Shiny App.

Sports

footBayes v0.1.0: Provides functions to estimate, visualize, and predict the most well-known football models: double Poisson, bivariate Poisson, Skellam, student_t. The package allows Hamiltonian Monte Carlo (HMC) estimation through the underlying Stan environment and Maximum Likelihood estimation (MLE, for ‘static’ models only). See Dixon & Coles (1997) Karlis & Ntzoufras (2003), and Pauli & Torelli (2018) for background and the vignette for and introduction to the package.

Statistics

fido v1.0.0: Provides methods for fitting and inspecting Bayesian Multinomial Logistic Normal Models using MAP estimation and Laplace approximation as developed in Silverman et. Al. (2022). There is an Introduction and vignettes on PCR Bias, Non-linear models, Joint Modeling, and Picking priors.

geostan v0.2.1: Provides Stan-based tools for Bayesian inference with spatial data, including exploratory analysis tools, multiple spatial model specifications, spatial model diagnostics, and special methods for inference with small area survey data. See Donegan et al. (2020) for background and the vignettes on Spatial autocorrolation and Survey data for examples.

Landmarking v1.0.0: Provides functions to perform Landmark survival analyses which allow survival predictions to be updated dynamically as new measurements from an individual are recorded. There is an Introduction and a vignette on how to use the package.

PUMP v1.0.0: Provides functions to estimate power, minimum detectable effect size and sample size requirements in the context of multilevel randomized experiments with multiple outcomes. See Hunter et al. (2021) for the details. There is a Package Demo vignette and vignettes on the Sampling method, and Simulating multi-level data.

safestats v0.8.6: Provides functions to design and apply tests which are anytime valid, can be used to design hypothesis tests in the prospective/randomized control trial setting or in the observational/retrospective setting, and remain valid under both optional stopping and optional continuation. For details on the theory of safe tests, see Grunwald et al. (2019). There is a vignette on Safestats and another on Contingency tables.

Utilities

audubon v0.1.1: Provides a collection of Japanese text processing tools for filling Japanese iteration marks, Japanese character type conversions, segmentation by phrase, and text normalization which is based on rules for the Sudachi morphological analyzer and the NEologd (Neologism dictionary for MeCab). See README to get started.

rconfig v0.1.1: Allows users to manage R configuration files and override configuration statements from the command line. Look here for details.

Visualization

ggchangepoing v0.1.0: R provides tools for changepoint analysis and uses ggplot2 to visualize changepoints. See the vignette.

ggdensity v0.0.1: Provides functions for visualizing contours of 2-d kernel density estimates and implements several additional density estimators as well as more interpretable visualizations based on highest density regions instead of the traditional height of the estimated density surface. Look here for examples.

gghdr v0.1.0: Provides a framework for visualizing Highest Density Regions in ggplot2. See the vignette.

ggpattern v0.4.2: Provides geoms filled with various patterns including patterned versions of every ggplot2 geom that has a region that can be filled along with a suite of aesthetics and scales for controlling pattern appearances. There four vignettes: Developing Patterns and three more that cover gradients, polygons, and crosshatching.

Some R Conferences for 2022

Tue, 15 Mar 2022 00:00:00 +0000

The 2022 R Conference season is already underway. Here is a list of upcoming conferences that we know about. If we have missed your conference, please write to us with the details. We will update our list as we receive more information.

April

(27 - 29) on-line and free - Appsilon Shiny Conference will bring together members from the global community of Shiny developers to learn, network, and collaborate.

May

(25 - 27) on-line and free - VI International Seminar on Statistics with R, recognized by the R Foundation for its pioneering work with R users in Latin America, SSR is a multidisciplinary event for professionals and students alike.

June

(3 - 4) Chicago - R/Finance, one of the longest running R conferences, is the primary meeting for academics and Quants using R for finance. This single track event is the place to talk time series, stochastic modeling, and meet some legendary R developers.

(4) on-line - IDEA, The R Consortium working group on Inclusion, Diversity, Equity and Accessibility, will be holding francophone satRdays event. More information should be available on the R Consortium Blog shortly.

(7 - 10) Pittsburgh, PA - SDSS, the Symposium on Data Science and Statistics, will offer R based short courses on spatial statistics and data visualization and likely feature R in a number of talks on computational statistics and data science.

(8 - 10) NYC - R Conference is an R Community institution: excellent talks, nice people, and Spring in NYC.

(15 - 17) Milan and on-line - Insurance Data Science Conference is the conference to attend for to learn about AI, data science, ML, and computational statistics in the insurance industry.

(20 - 23) on-line - useR! now in its 18th year is the official conference of the R Project for Statistical Computing. Keynotes include Sebastian Meyer, Amanda Cox, Julia Silge, Mine Dogucu and will showcase the afrimapr project supports the development of a community of practice in Africa around map-making in R.

July

(25 - 28) Washington D.C. - rstudio::conf 2022 is back and live and will probably be the biggest gathering in the R world this year. Come for the talks (Keynotes by Mine Çetinkaya-Rundel & Julia Stewart Lowndes, Julia Silge & Max Kuhn, and Jeff Leek), for the people, and to see what’s new at RStudio. Registration is open and there is still time to submit a talk.

(27 - 29) BioC 2022, the North American gathering for the Bioconductor project will feature keynotes by Jalees Rehman,Sandrine Dudoit, and Mine Çetinkaya-Rundel). The call for abstracts is still open. The venue has not yet been decided.

August

(6 - 11) Washington D.C. - JSM one of the largest statistical events in the world, will likely have several R-related talks. There is still time to participate in the program.

(23 - 26) on-line - R/Medicine, an international conference with a devoted following, explores the use of R based statistics and data science tools to improve clinical research and practice. J.J. Allaire and Frank Harrell will deliver keynotes.

September

(14 -16) - Heidelberg - EUROBIOC 2022 the European Bioconductor gathering will be live. The call for submissions will open soon.

(6 - 8) London - EARL Conference 2022 is a cross-sector conference focusing on the commercial use of the R programming language. Whether you’re coding, wrangling data, leading a team of R users, or making data-driven decisions, EARL will have something to offer you.

November

(8 - 10) on-line - R/Pharma, a small conference with a big impact, focuses on the user of R in the development of pharmaceuticals.

December

(26 - 30) Bangalore - IISA 2022 Conference, the annual conference of the International Indian Statistical Association is considering talks on Statistics with R programming, Precision Medicine, Bayesian Spatial Statistics, Statistical Machine Learning, Deep Learning, and Python Programming. A half day, hands-on R workshop is also being planned.

Data Challenges for R Users

Thu, 10 Mar 2022 00:00:00 +0000

Today, I want to write about “data challenges”, where participants partake in a series of prompts designed for a variety of skill sets and levels. These challenges are opportunities to practice programming skills, work on algorithmic efficiency, or learn something new. Often, participants share the code for their submissions, teaching others their processes and techniques.

The challenges test the participants’ tenacity with time-bound but flexible prompts. TidyTuesday was originally borne out of the R4DS Online Learning Community and the R for Data Science textbook. RStudio’s Tom Mock shares a dataset every Tuesday. The challenge is to explore the dataset and create a chart within seven days. Participants decide what variables to focus on and how to best visualize them.

Using a recent TidyTuesday dataset, Moriah Taylor visualizes alternative fueling stations in the U.S. The stacked tilted maps communicate relationships, density, and location while demonstrating beautiful design.

Moriah’s code introduced me to the layer package, which tilts maps and turns them into ggplot2 objects:

#plot tilted maps
plot <- plot_tiltedmaps(maps_list,
                layer = c("value", "value", "value", "value"),
                palette = c("magma", "magma", "magma", "magma"))

Data challenges can also be language-agnostic. Genuary encourages participants to create works of generative art with the programming language of their choice. R users apply their coding skills and tools in imaginative ways. On Day 6 of genuary, Ryan Timpe “destroys the square”:

Taking a look at the code, we learn how to plot a square in R with the tidyr and dplyr packages:

library(tidyverse)
set.seed(12238)

#The square

square_size = 400

the_square <- expand_grid(
  x=1:square_size,
  y=1:square_size
) %>%
  mutate(
    color = "#87ceeb"
  )

plot(the_square$x, the_square$y)

We can also see how Ryan inserts randomness to ‘destroy’ the square through the strategic use of functions like sample() and runif():

circ_origin_size = square_size * runif(1, 2/4, 3/4)

Interested in participating in a data challenge? The 30 Day Chart Challenge is coming up. The organizers offer different chart types for you to visualize in April. Showcase your R skills, test your perseverance, and use the hashtag #30DayChartChallenge to share your work with other participants.

We started to compile a list of other data challenges to join. Please leave a comment if you are aware of others.

Timeframe	Name	URL
January	genuary	http://genuary.art/
April	30 Day Chart Challenge	https://twitter.com/30DayChartChall
October	Hacktoberfest	https://hacktoberfest.digitalocean.com/
November	30 Day Map Challenge	https://github.com/tjukanovt/30DayMapChallenge
December	Advent of Code	https://adventofcode.com/
Monday	Makeover Monday	https://www.makeovermonday.co.uk/
Tuesday	Tidy Tuesday	https://github.com/rfordatascience/tidytuesday
Thursday, Summer Months	Recreation Thursday	https://github.com/sharlagelfand/RecreationThursday
Sunday	SportsViz Sundays	https://data.world/sportsvizsunday
Monthly	SWD	https://community.storytellingwithdata.com/
Ongoing	100 Days of Code	https://www.100daysofcode.com/

Thank you to Moriah Taylor and Ryan Timpe for sharing their visualizations.

January 2022: "Top 40" New CRAN Packages

Mon, 28 Feb 2022 00:00:00 +0000

Two Hundred and two new packages made it to CRAN in January. Several were in applications areas that don’t show up very often, and a few of these made it into my January “Top 40” picks. My selections are listed below in thirteen categories: Agriculture, Computational Methods, Data, Engineering, Finance, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities, and Visualization. Slightly expanding the number of categories helps to emphasize the various application areas, but makes it more difficult to classify packages which could fit in multiple categories. I hope this does not inconvenience readers.

Agriculture

ALUES v0.2.0: Provides functions for fuzzy modeling to evaluate land suitability for different crops production according to the methodology established by the Food and Agriculture Organization and the International Rice Research Institut. There is a Getting Started Guide along with seven vignettes including: Methodology and Visualizing with maps.

Computational Methods

CGNM v0.1.1: Implements the cluster Gauss-Newton method to find multiple solutions to nonlinear least squares problems. See Aoki et al. (2020) for background and the vignette to get started.

simpr v0.2.2: Implements a tidyverse friendly framework for simulation studies, design analysis, and power analysis. It enables users to generate data, fit models, and tidy up model results in a single pipeline. There is an Introduction and there are short vignettes on Optimization, Reproducing simulations, and Managing simulation errors.

Data

chessR v1.5.0: Provides functions to enable users to extract chess game data from popular chess sites, including Lichess and Chess.com and then perform analyses on game data. See the vignette.

dictionaRy v0.1.1: Implements an interface to the Free Dictionary API which allows users to retrieve dictionary definitions for English words, as well as additional information including phonetics, part of speech, origins, audio pronunciation, example usage, synonyms and antonyms which are returned in tidy format for ease of use.

flightsbr: v0.1.0: Provides functions to download flight and airport data from Brazil’s Civil Aviation Agency (ANAC) that includes detailed information on all aircraft, aerodromes, airports, and airports movements registered in ANAC, and on every international flight to and from Brazil, as well as domestic flights within the country. There is an Introduction and there are vignettes on Airport data and Flights data.

rGhanaCensus v0.1.0: Contains literacy and education data sets scrapped from the 2021 Ghana Population and Housing Census See the vignette.

Engineering

TesiproV v0.9.1: Provides functions to calculate the failure probability of civil engineering problems with Level I up to Level III Methods. See AU & BECK (2001) for background and the vignette for examples.

Finance

PDtoolkit v0.1.0: Provides functions for developing probability of default rating models including functions for imputations, binning of numeric and categorical risk factors, weights of evidence, information value calculations, and risk factor clustering as well as validation functions for testing homogeneity, heterogeneity, discriminatory and predictive power of the model. See README for examples.

ufRisk v1.0.2: Provides functions to calculate Value at Risk (VaR) and Expected Shortfall (ES) by means of various parametric and semiparametric GARCH-type models. The approaches implemented in this package are described in Feng et al. (2020) and Letmathe et al. (2021). See README to get started.

Genomics

aphylo v0.2-1: Implements a parsimonious evolutionary model to analyze and predict gene-functional annotations in phylogenetic trees as described in Vega Yon et al. (2021). See the vignette for an example.

edlibR v1.0.0: Implements an interface to the Edlib C/C++library for exact sequence alignment using Levenshtein distance. See the vignette to get started.

freqpcr v0.4.0: Implements functions for the interval estimation of the population allele frequency from qPCR analysis based on the restriction enzyme digestion (RED)-DeltaDeltaCq method as described in Osakabe et al. (2017). The vignette is in English and Japanese.

Machine Learning

longmixr v1.0.0: Adapts the consensus clustering approach from ConsensusClusterPlus for longitudinal data using flexmixflexible mixture models. See the vignette.

reclin2 v0.1.1: Provides functions to assist performing probabilistic record linkage and deduplication: generating pairs, comparing records, em-algorithm for estimating m- and u-probabilities. See Fellegi & Sunter (1969) for background. There is an Introduction and there are vignettes on Deduplication, Record lingage, and Using a cluster for parallel computing.

rego v1.3.4: Implements a machine learning algorithm for predicting and imputing time series along with a Bayesian stochastic search methodology for model selection. Written in C++, the authors claim that the package is suitable for problems with hundreds or thousands of dependent variables and problems in which the number of dependent variables is greater than the number of observations. Look here for documentation.

Mathematics

tesselation v1.0.0: Computes Delaunay and Voronoï tessellations and provides functions to plot the 2 and 3 dimensional tessellations. Delaunay tessellations are computed in C with the help of the Qhull. There is a vignette on GitHub.

weyl v0.0-1: Provides functions for working with Weyl Algebras. See the vignette.

Medicine

biodosetools v3.6.0: Implements a Shiny application to perform the various statistical tests and calculations needed by Biological Dosimetry Laboratories. There ar vignettes on Dicentrics dose estimation and dose-effect fitting and on Translocations dose estimation and dese-effect fitting.

rccola v1.0.2: Provides secure convenience functions for entering and handling API keys and pulling data directly into memory. By default, it will load from REDCap instances, but other sources are injectable via inversion of control. See README for documentation.

Science

datelife v0.6.1: Implements the functions underlying the DateLife web service to allow researchers and the general audience to obtain open scientific data on ages for their organisms of interest. Age data are extracted from dated phylogenetic trees (chronograms) that have been published and peer-reviewed in association with a scientific article in an indexed journal. There is a Getting Started Guide and a Case Study.

dynamAdes v2.0.2: Implements a model to study the population dynamics of invasive Aedes mosquitoes. See Da Re et al. (2021) for the model rationale, and Da Re et al. (2021) for the model framework. The vignette provides an example.

Statistics

autoReg v0.1.0: Provides functions to create summary tables for descriptive statistics and select explanatory variables automatically for various regression models including linear models, generalized linear models and cox-proportional hazard models. There is a Getting Started Guide and there are vignettes on Automatic Regression Modeling, Bootstrap Simulation, Statistical tests, and Survival Analysis.

conformalInference.multi v1.0.0: Provides functions to compute full conformal, split conformal and multi split conformal prediction regions when the response variable is multivariate (i.e. dimension is greater than one). For background see Lei et al. (2016), Diquigiovanni et al. (2021), and Solari & Djordjilovic (2021). Look here for examples.

gamselBayes v1.0-2: Provides functions to fit and select generalized additive models via approximate Bayesian inference according to the methodology described in He & Wand (2021). There is a vignette.

interpretCI v0.1.1: Provides functions for estimating confidence intervals for various statistics and plotting the results. There is an Introduction along with ten additional vignettes for the various statistics including CI for difference between proportions and Hypotheses test for difference between paired means.

lmls v0.1.0: Implements functions for Gaussian location-scale regression model (a multi-predictor model with explanatory variables for the mean (location) and the standard deviation (scale) of a response variable) using the algorithms described in Girolami & Calderhead (2011) and Nesterov (2009). See the vignette.

sandwichr v1.0.1: Implements the Spatial Stratified Heterogeneity (SSH) spatial interpolation algorithms described in Wang et al. (2013). See the vignette.

Utilities

gittargets v0.0.3: Provides functions to preserve historical output in targets workflows by capturing version-controlled snapshots of the data store. Each snapshot links to the underlying commit of the source code enabling users to recover contemporaneous data when rolling back to a previous commit. See the Tutorial.

httptest2 v0.1.0: For packages using httr2 this package enables testing all of the logic on the R side of the API without requiring access to the remote service. It also allows recording real API responses to use as test fixtures. There is an Introduction and there are vignettes on Modifying Recorded Requests, Writing Vignettes with API’s, and FAQs.

maybe v0.2.0: Implements a maybe type which represents the possibility of some value or nothing. This may be used instead of throwing an error or returning NULL. maybe and has the advantages of being composable and requiring the developer to explicitly acknowledge the potential absence of a value. Look here to get started.

nanonext v0.2.0: Implements an R binding for NNG (Nanomsg Next Gen), a socket library for implementing a high-performance cross-platform protocol standard for messaging and communications. It serves as a concurrency framework that can be used for building distributed applications. See README to get started.

powerjoin v0.0.1: Provides extensions of dplyr and fuzzyjoin join functions to preprocess data, apply various data checks, and deal with conflicting columns. See README for examples.

quickcheck v0.1.0: Builds on the framework provided by hedgehogto implement property based testing in R. It was inspired by QuickCheck and has been designed to seamlessly integrate with testthat. See README for an introduction.

Visualization

fisheye v0.1.0: Provides functions to transform base maps to focus on a specific location using an azimuthal logarithmic distance transformation. See README to get started.

forestplotter v0.1.1: Provides functions to create multi-column forest plots with confidence intervals that may be grouped the data. See the vignette.

geomtextpath v0.1.0: Implements a ggplot2 extension which allows text to follow curved paths. Curved text makes it easier to directly label paths or neatly annotate in polar co-ordinates. There is an Introduction and there are vignettes on Curved Text in Polar Plots and Aesthetics.

ggESDA v0.1.0: Implements an extension of ggplot2 to visualize symbolic data and also provides a function to transform classical data to symbolic data by both clustering algorithm and customized methods.

toastui v0.2.1: Implements an interface to the TOAST UI libraries for creating interactive tables and plots that can be integrated into Shiny applications. There is a vignette and a webpage with interactive examples.

tornado v0.1.1: Implements linear models, generalized linear models, survival regression models, and machine learning models using the caret package framework and draws tornado plots to visualize the range of outputs expected from a variety of inputs, or alternatively, the sensitivity of the output to the range of inputs. See the vignette.

CRAN "Golden Oldies" and "One Hit Wonders"

Wed, 16 Feb 2022 00:00:00 +0000

If you are a regular R Views reader, then you know that every month I post my Top 40 picks for new CRAN packages. In this post, I’ll borrow some additional terminology from the Top 40 AM radio of my youth, and talk about two new categories of CRAN packages: Golden Oldies and One Hit Wonders.

Golden Oldies

On Top 40 radio, Golden Oldies are tunes that continue to get a lot of air play over decades, and are generally thought to be classics of 50’s, 60’s, and 70’s pop culture. By a Golden Oldie R package I mean an R package that has been around for a long time, has gone through several version upgrades of bug fixes and enhancements, and is thought to be an indispensable package for some part of the R Community.

For example, the nlme and lme4 R packages for fitting mixed models would surely be on the Golden Oldie list for any biostatistician who works in R. nlme has been around since 1999 and has had 139 version bumps, while lme4 was introduced in 2003 and has had 118 versions so far. The download statistics indicate that lme4 has largely replaced nlme but nlme is still getting some serious play time.

d_stats_mm <- cran_stats(c("nlme","lme4"))
ggplot(d_stats_mm, aes(end, downloads, group=package, color=package)) + geom_line() +
  ggtitle("Downloads for Mixed Model Packages") + xlab("Month")

In addition to signalling usefulness, Golden Oldieness also serves as a proxy for quality. I believe that most R users would agree that packages which have been lovingly maintained over many years and still see frequent use are most likely to contain code that you can count on. This is the kind of risk mitigation metric that is driving the R Validation Hub effort.

The following is a list of ten packages that I count as Golden Oldies. They have all been around for at least 14 years and they all get considerable play.

10 Golden Oldies
pkg	date	num_pub
quantreg	1999-01-11	67
nlme	1999-11-23	139
survival	2001-06-22	94
lme4	2003-06-25	118
Hmisc	2003-07-17	70
zoo	2004-02-20	64
data.table	2006-04-15	56
ggplot2	2007-06-10	41
xts	2008-01-05	37
Rcpp	2008-11-06	91

One Hit Wonders

On Top 40 radio One Hit Wonders could also be Golden Oldies. The designation just meant that the recording artists had one big hit and then could not find their way back into the Top 40 again. (Here is a One Hit Wonder favorite of mine from 1969.)

For CRAN packages, I am using One Hit Wonder to mean a package that has been on CRAN for many years but has never had a version bump: no bug fixes, no upgrades, no faults - just the opposite of “Golden Oldies”. Finding them is not so easy though. As far as I can tell, there is no easy way to access the CRAN archive metadata. However, you can hunt for One Hit Wonders by searching through the long left tail of the publication date distribution of CRAN packages.

This code picks out the publication dates for CRAN packages.

c_db <- tools::CRAN_package_db()
cran_db <- clean_CRAN_db()
package_network <- cran_db %>% build_network(perspective = "package")

pkg <- package_network$nodes$package
pub <- package_network$nodes$published

published <- na.omit((data.frame(pkg,pub)))
clean_pub <- published %>% filter(pub > as.Date(2005-01-01, format = "%Y-%m-%d", origin = "1900-01-01"))

And, here is the histogram. The vertical lines mark the first two quantiles.

ggplot(clean_pub, aes(x=pub)) + geom_histogram(bins = 193) +
  geom_vline(aes(xintercept = median(pub)),col='grey',size=1, linetype="dotted") +
  geom_vline(aes(xintercept = quantile(pub,probs = .25, type = 1)), col='grey',size=1, linetype="dotted") + 
  xlab("CRAN Package Publication Date") + ylab("Number of packages") + ggtitle("Histogram of Publication Dates")

When I recently ran this code, the oldest current publication date of the 18,850 package on CRAN is March 15, 2006.

summary(clean_pub)

##      pkg                 pub            
##  Length:18970       Min.   :2006-03-15  
##  Class :character   1st Qu.:2018-06-19  
##  Mode  :character   Median :2020-08-07  
##                     Mean   :2019-09-22  
##                     3rd Qu.:2021-08-13  
##                     Max.   :2022-02-16

The following code extracts the packages in the first quantile and sorts them. Determining if the publication date is the original package publication date, however, requires verifying that the package has never been archived.

q1_pub <- quantile(clean_pub$pub, type = 1)[2]
q1_pkgs <- clean_pub %>% filter(pub <= q1_pub) %>% arrange(pub)
q1_pkgs[1:20,]

##              pkg        pub
## 1      coxrobust 2006-03-15
## 2  BayesValidate 2006-03-30
## 3       fuzzyFDR 2007-10-16
## 4         poilog 2008-04-29
## 5        SASPECT 2008-06-23
## 6            RM2 2008-08-13
## 7           pack 2008-09-08
## 8         expert 2008-10-02
## 9            kzs 2008-10-28
## 10           ETC 2009-01-30
## 11 CreditMetrics 2009-02-01
## 12   Reliability 2009-02-01
## 13           spe 2009-02-24
## 14          mcsm 2009-04-28
## 15    SEMModComp 2009-05-05
## 16   bootStepAIC 2009-06-04
## 17      HybridMC 2009-06-08
## 18    PearsonICA 2009-06-29
## 19    crantastic 2009-08-08
## 20         km.ci 2009-08-30

Eyeballing the package documentation on CRAN revealed that there are nine One Hit Wonders among the first twenty oldest packages in the left tail. Here are their download stats.

But what can you say about these One Hit Wonders? Except for fuzzyFDR, the first nine don’t seem to be getting much action. The small download numbers, however, do not themselves signal a lack quality or irrelevance. There may indeed be some treasures among One Hit Wonders that survive on CRAN because unlike the music business, as long as a package plays and does not break any other package, the CRAN D.J.s keep it in the playlist.

So, are you the kind of person who enjoys sorting through old records: 33s, 45s, and maybe even 78s. Would you visit a shop like Rough Trade, the AI Record Shop or Stranded Records on a trip to New York City? If so, maybe you would enjoy looking through CRAN for esoteric packages like BayesValidate which supports an interesting paper on validating Bayesian models. Wouldn’t it be special to find a One Hit Wonder out there on CRAN in mint condition, needing no bug fixes or enhancements, that does something really sweet?

SEM Stochastic Simulation and Optimal Control

Mon, 07 Feb 2022 00:00:00 +0000

Andrea Luciani is a Technical Advisor for the Directorate General for Economics, Statistics and Research at the Bank of Italy, and co-author of the bimets package.

Introduction

In this post, I show how to analyze the forecast error for a Structural Equation Model (SEM) by means of a stochastic simulation, and how to perform optimal control. In my previous post, I presented an approach to estimating and simulating SEMs in R which focused on R tools that allow users to forecast advanced simultaneous equations models having linear restrictions on coefficients, error autocorrelation on residuals, and conditional equation evaluation. I described how simulating these econometric models often requires using iterative algorithms in ways that are well beyond what the ts(), lm() and predict() functions were designed to do and worked through a forecasting exercise of a Klein model by using the deterministic simulation procedure available into the bimets package.

Stochastic Simulation

Deterministic algorithms, however, have significant limitations when applied to SEM forecasts which are subject to several sources of error such as: a random disturbance term of each stochastic equation, errors in estimated coefficients, and errors in forecasts of exogenous variables.

The forecast error depending on the structural disturbances can be analyzed by using a stochastic simulation in which the structural disturbances are given values with specified stochastic properties. The error terms of the estimated equations are appropriately perturbed. Technical identities and exogenous variables can also be perturbed by disturbances with specified stochastic properties. Programmatically, a straightforward approach to solve the problem is a Monte-Carlo method: the model is solved for each data set with different values for the disturbances. Finally, forecast statistics (e.g. mean, standard deviation, etc.) can be computed for each simulated endogenous variable.

Using the Klein model defined in my previous post, we can attempt to complete a US Gross National Product forecasting exercise, given a user-specified uncertainty on Consumption and Government Expenditure time series.

The Klein model will be perturbed by applying a normal disturbance to the endogenous Consumption behavioral cn in year 1942, and a uniform disturbance to the exogenous Government Expenditure time series g along all the forecast time range. The normal disturbance applied to the cn stochastic equation has a zero mean and a standard deviation equal to its regression standard error, thus roughly replicating the regression error during the current perturbation (not accounting for inter-equations cross-covariance).

#we want to perform a stochastic forecast of the GNP up to 1944
#we will add normal disturbances to endogenous Consumption 'cn' 
#in 1942 by using its regression standard error
#we will add uniform disturbance to exogenous Government Expenditure 'g'
#in whole TSRANGE

myStochStructure <- list(
  cn=list(
    TSRANGE=c(1942,1,1942,1),
    TYPE='NORM',
    PARS=c(0,kleinModel$behaviorals$cn$statistics$StandardErrorRegression)
  ),
  g=list(
    TSRANGE=TRUE,
    TYPE='UNIF',
    PARS=c(-1,1)
  )
)
 
#model stochastic forecast 
kleinModel <- STOCHSIMULATE(kleinModel
                            ,simType='FORECAST'
                            ,TSRANGE=c(1941,1,1944,1)
                            ,StochStructure=myStochStructure
                            ,StochSeed=123
                            ,quietly=TRUE
)

#print mean and standard deviation for the forecasted GNP
with(kleinModel$stochastic_simulation,TABIT(y$mean, y$sd))

## 
##       Date, Prd., y$mean      , y$sd        
## 
##       1941, 1   ,  125.5      ,  4.251      
##       1942, 1   ,  173.3      ,  9.263      
##       1943, 1   ,  186        ,  11.88      
##       1944, 1   ,  141.1      ,  11.7

Optimal Control

Another limitation of the deterministic simulation procedure relies on the fact that analysts often do not want to perform a mere forecasting exercise. Usually, a performance measure (i.e. objective-function) is assigned to an econometric model, depending on the value of forecasted endogenous variables; thus, analysts try to enhance this measure by fine-tuning exogenous variables (i.e. the instruments) of the model (e.g. policy analysis).

Generally speaking, this is an optimal control exercise: the optimization consists of maximizing an objective-function, depending on (forecasted) endogenous variables, given a set of user-constraints plus the constraints imposed by the econometric model equations.

In the following exercise, we will use the bimets OPTIMIZE() procedure which allows R users to perform optimal control exercises on simultaneous equations models. The exercise will be completed by using the following code on the previously defined Klein model, and we will assume that:

The objective-function definition is: $f(y, cn, g) = (y-110)+(cn-90)*|cn-90|-\sqrt{g-20}$ given $y$ as the simulated Gross National Product, $cn$ as the simulated Consumption and $g$ as the exogenous Government Expenditure. The basic idea is to maximize Consumption, and secondarily the Gross National Product, while reducing the Government Expenditure;
The instrument variables (i.e. INSTRUMENT in following code snippet) are the $cn$ Consumption “booster” (i.e. the add-factor, not to be confused with the simulated Consumption in the objective-function) and the $g$ Government Expenditure, defined over the following domains: $cn \in (-5,5)$, $g \in (15,25)$;
The following restrictions are applied to the INSTRUMENT: $g + cn^2/2 < 27 \wedge g + cn > 17$, given again $cn$ as the Consumption “booster” (i.e. the add-factor) and $g$ as the Government Expenditure;

The following code performs the optimization.

#we want to maximize the non-linear objective function:
#f()=(y-110)+(cn-90)*ABS(cn-90)-(g-20)^0.5
#in 1942 by using INSTRUMENT cn in range (-5,5) 
#(cn is endogenous so we use the add-factor)
#and g in range (15,25)
#we will also impose the following non-linear restriction:
#g+(cn^2)/2<27 & g+cn>17

#define INSTRUMENT and boundaries
myOptimizeBounds <- list(
    cn = list( TSRANGE = TRUE
            ,BOUNDS = c(-5,5)),
     g = list( TSRANGE = TRUE
            ,BOUNDS = c(15,25))
)

#define restrictions
myOptimizeRestrictions <- list(
    myRes1=list(
         TSRANGE = TRUE
        ,INEQUALITY = 'g+(cn^2)/2<27 & g+cn>17')
)

#define objective function
myOptimizeFunctions <- list(
    myFun1 = list(
         TSRANGE = TRUE
        ,FUNCTION = '(y-110)+(cn-90)*ABS(cn-90)-(g-20)^0.5')
)

#Monte-Carlo optimization by using 10000 stochastic realizations
#and 1E-4 convergence criterion 
kleinModel <- OPTIMIZE(kleinModel
                      ,simType = 'FORECAST'
                      ,TSRANGE=c(1942,1,1942,1)
                      ,simConvergence= 1E-4
                      ,simIterLimit  = 1000
                      ,StochReplica  = 10000
                      ,StochSeed = 123
                      ,OptimizeBounds = myOptimizeBounds
                      ,OptimizeRestrictions = myOptimizeRestrictions
                      ,OptimizeFunctions = myOptimizeFunctions
                      )

## OPTIMIZE(): optimization boundaries for the add-factor of endogenous variable "cn" are (-5,5) from year-period 1942-1 to 1942-1.
## OPTIMIZE(): optimization boundaries for the exogenous variable "g" are (15,25) from year-period 1942-1 to 1942-1.
## OPTIMIZE(): optimization restriction "myRes1" is active from year-period 1942-1 to 1942-1.
## OPTIMIZE(): optimization objective function "myFun1" is active from year-period 1942-1 to 1942-1.
## 
## 
Optimize:     100.00 %
## OPTIMIZE(): 2916 out of 10000 objective function realizations (29%) are finite and verify the provided restrictions.
## ...OPTIMIZE OK

#print local maximum
kleinModel$optimize$optFunMax

## [1] 210.6

#print INSTRUMENT that allow local maximum to be achieved
kleinModel$optimize$INSTRUMENT

## $cn
## Time Series:
## Start = 1942 
## End = 1942 
## Frequency = 1 
## [1] 2.032
## 
## $g
## Time Series:
## Start = 1942 
## End = 1942 
## Frequency = 1 
## [1] 24.9

The scatter plot of objective function surface is populated with 2916 objective-function stochastic realizations; the 210.58 local maximum is stored in the kleinModel$optimize$optFunMax variable, while the instruments that allow local maximum to be achieved are stored in the kleinModel$optimize$INSTRUMENT variable.

Summary

Forecast errors in Structural Equation Models can be analyzed by using a stochastic simulation in which the structural disturbances are given values with specified stochastic properties. Stochastic modeling forecasts the probability of various outcomes under different conditions, using random variables. Often, a performance measure (i.e. objective-function) is assigned to an econometric model, depending on the value of forecasted endogenous variables. The Monte Carlo simulation is one example of a stochastic model; it can simulate how an econometric model may perform based on the probability distributions of individual variables of the model, allowing analysts to enhance the objective-function by fine-tuning exogenous variables (i.e. the instruments) of the model (e.g. policy analysis) in the so-called optimal control procedure. Examples presented in this post show how to perform stochastic simulation and optimal control in R.

Disclaimer: The views and opinions expressed in this page are those of the author and do not necessarily reflect the official policy or position of the Bank of Italy. Examples of analysis performed within these pages are only examples. They should not be utilized in real-world analytic products as they are based only on very limited and dated open source information. Assumptions made within the analysis are not reflective of the position of the Bank of Italy.

December 2021: "Top 40" New CRAN Packages

Mon, 24 Jan 2022 00:00:00 +0000

One hundred thirty-four new packages made it to CRAN last December. Here are my “Top 40” picks in eight categories: Data, Genomics, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Data

aurin v0.5.1: Implements an API for AURIN, Australia’s largest resource for accessing clean, integrated, spatially enabled and research-ready data on issues surrounding health and well being, socio-economic metrics, transportation, and land-use. See README for examples.

fastRhockey v0.1.0: Implements a utility to scrape and load play-by-play data and statistics from the Premier Hockey Federation (PHF), formerly known as the National Women’s Hockey League (NWHL), and access the National Hockey League’s stats API. See README.

pedalfast.data v1.0.0: Provides data files and documentation for PEDiatric vALidation oF vAriableS in TBI (PEDALFAST) used in Bennett et al. (2016). There is a vignette describing the PEDALFAST Data and another on Functional Status Scale.

rasterbc v1.0.1: Provides access to a large data set hosted at the Federated Research Data Repository relevant to forest ecology in British Columbia, Canada. The collection includes: elevation, biogeoclimatic zone, wildfire, and cutblocks forest attributes from Hansen et al. (2013) and Beaudoin et al. (2017). See the vignette.

spectator v0.1.0: Provides an interface to the Spectator Earth API, mainly for obtaining the acquisition plans and satellite overpasses for Sentinel-1, Sentinel-2 and Landsat-8 satellites. See the vignette.

Genomics

CAMML v0.1.1: Provides functions to create multi-label cell-types for single-cell RNA-sequencing data based on weighted VAM scoring of cell-type specific gene sets. See Schiebout (2022). There is a Quick Start guide, a Figure vignette and an extended example with melanoma data.

hacksig v0.1.1: Provides a collection of cancer transcriptomics gene signatures as well as a simple, tidy interface to compute single sample enrichment scores either with the original procedure or with three alternatives: the combined z-score of Lee et al. (2008) the single sample GSEA of Barbie et al. (2009) and the singscore of Foroutan et al. (2018). See README for examples.

mispitools v0.1.5: Provides functions for computing likelihood ratios thresholds and error rates in DNA kinship testing. See Marsico et al. (2021) for background and README for examples.

paleobuddy v1.0.0: Provides functions to simulate species diversification, fossil records, and phylogenies along with environmental data from Morlon et al. (2016). See the vignette for an overview.

toolStability v0.1.1: Provides a collection of functions for describing the stability of a trait in terms of genotype and environment and also includes a data set from Casadebaig et al. (1966). Computed indices are from Döring & Reckling (2018), Eberhart & Russell (1966), Finlay & Wilkinson GN (1963), Hanson WD (1970), Lin & Binnsn(1988), Pinthus (1973) and others. Th vignette provides an overview of the the theory and examples.

twosigma v1.0.2: Implements the TWO-Component Single Cell Model-Based Association Method for gene-level differential expression analysis and DE-based gene set testing of single-cell RNA-sequencing datasets. See Van Buren et al. (2020) and Van Buren et al. (2021) for the theory and README for examples.

Machine Learning

brulee v0.0.1: Provides high-level modeling functions to define and train models using the torch R package. Models include linear, logistic, and multinomial regression as well as multilayer perceptrons. See README for an example.

imageseg v0.4.0: Implements a general-purpose workflow for image segmentation using TensorFlow models based on the U-Net architecture by Ronneberger et al. (2015) and provides pre-trained models for assessing canopy density and understory vegetation density from vegetation photos. See the vignette.

TrueSkillThroughTime v0.1.0: Provides methods to model the entire history of game activities using a single Bayesian network allowing the information to propagate correctly throughout the system. The core ideas implemented in this project were developed by Dangauthier et al. (2007). Look here for examples.

Medicine

aba v0.0.9: Offers a tool to fit clinical prediction models and plan clinical trials using biomarker data across multiple analysis factors (groups, outcomes, predictors). There is a Package Overview and an Introduction to aba models.

expirest v0.1.2: Provides functions to estimate the release limit and the associated shelf life for chemically derived medicines in accordance with the recommendations of The Australian Regulatory Guidelines for Prescription Medicines guidance on Stability testing for prescription medicines and the International Council for Harmonisation’s guidance Q1E Evaluation of stability data. See README for examples.

fca v0.1.0: Provides functions to perform various floating catchment area methods to calculate a spatial accessibility index for demand point data. See Bauer & Groneberg (2016) for background and the vignette for an example.

grpseq v1.0: Provides functions to help with the design of group sequential trials, including non-binding futility analysis at multiple time points. See Gallo, Mao, & Shih (2014) for background and the vignette for an example.

lemna v0.9.0: Implements the model equations and default parameters for the toxicokinetic-toxicodynamic model of the Lemna (duckweed) aquatic plant. Lemna is a standard test macrophyte used in ecotox effect studies. See Schmitt et al. (2013) for background. There is an introduction and a vignette on verification.

Science

AeroSampleR v0.1.12: Provides functions to estimate ideal efficiencies of aerosol sampling through sample lines. See Hogue et al. (2014) for background and README for examples.

crestr v1.0.0: Implements the CREST climate reconstruction method to reconstruct past climates using biological proxies. See Chevalier (2021) for background. There is a Get started guide and vignettes on Theory, Formatting, and Calibration.

renz v0.1.1: Provides utilities to analyze Michaelis-Menten Equation models of enzyme kinetics. See Aledo (2021) for background along with the vignettes: Enzyme Kinetic Parameters, M-M and the Lambert W function, Linearized M-M Equations, Fitting the M-M Model, and Integrated M-M Equation .

Statistics

brmsmargins v0.1.1: Provides functions to calculate Bayesian marginal effects and average marginal effects for models fit using the brms package including fixed effects, mixed effects, and location scale models. See Pavlou et al. (2015) for background and the vignettes on marginal effects for Fixed Effects, Location Scale, and Mixed models.

changepoints v1.0.0: Implements a series of offline and/or online change-point detection algorithms for univariate mean, univariate polynomials, univariate and multivariate nonparametric settings, high-dimensional covariances, high-dimensional networks with and without missing values, high-dimensional linear regression models, high-dimensional vector autoregressive models, high-dimensional self exciting point processes, dependent dynamic nonparametric random dot product graphs, and univariate mean against adversarial attacks. See README for references for all of these methods and the vignettes VAR and Univariate means for examples.

cmtest v0.1-1: Provides functions to perform conditional moments test, as proposed by Newey (1985) and Tauchen (1985), useful to detect specification violations for models estimated by maximum likelihood. See the vignette.

gplsim v0.9.1: Provides functions that employ penalized splines to estimate generalized partially linear single index models which extend the generalized linear models to include nonlinear effect for some predictors. See Yu et al. (2017) and Yu & Ruppert (2002) for the details and README for an example.

gremes v0.1.0: Provides tools for estimation of the tail dependence parameters in graphical models parameterized by family of Huesler-Reiss distributions. See Asenova et al. (2021) for background. There is an Introduction and fifteen small vignettes described here.

kfa v0.1.0: Provides functions to explore possible factor structures for a set of variables and helps identify plausible and replicable structures via k-fold cross validation. See Flora & Flake (2017) and MacCallum et al.(1996) for background and README for examples.

lrstat v0.1.1: Provides functions to perform power and sample size calculation for non-proportional hazards model using the Fleming-Harrington family of weighted log-rank tests. See Tsiatis (1982) for background. There are six vignettes including Sample Size Calculation with Fixed Follow-up and Power Calculation Using Max-Combo Tests.

nphPower v1.0.0: Provides tools to perform combination tests and sample size calculation for fixed designs with survival endpoints using combination tests under either proportional hazards or non-proportional hazards. See Brendel et al. (2014) and Cheng & He (2021) for background on projection and maximum weighted logrank tests, Schoenfeld (1981) and and Freedman (1982) on sample size calculation under the proportional hazards assumption and Lakatos (1988) for calculation under non-proportional hazards, and the [README]() for an example.

optedr v1.0.0: Provides functions to calculate D-optimal, Ds-optimal, A-optimal, and I-optimal designs for non-linear models, via an implementation of the cocktail algorithm described in Yu (2011). See README for examples.

quid v0.0.1: Provides functions to test whether equality and order constraints hold for all individuals simultaneously by comparing Bayesian mixed models through Bayes factors. See Haaf & Rouder (2017) and Haaf, Klaassen & Rouder (2019), and Rouder & Haaf (2021) for background and the Quick Start guide and Manual for examples.

Utilities

abbreviate v0.1: Provides functions to abbreviate strings to at least minlength characters, such that they remain unique (if they were) and are recognizable. See README for examples.

fledge v0.1.0: Offers functions to streamline the process of updating changelogs (NEWS.md) and versioning R packages developed in git repositories. There is Quick Start guide, a demo, and a short vignette on internals.

qlcal v0.0.1: Provides QuantLib bindings using Rcpp via an evolved version of the initial header-only Quantuccia project offering an subset of QuantLib. See README for examples.

jagstargets Builds on targets and JAGS to implement a pipeline toolkit tailored to Bayesian statistics making it easy to set up scalable JAGS pipelines that automatically parallelize the computation and skip expensive steps when the results are already up to date. There is an Introduction and a vignette that uses simulation to validate a Bayesian model.

RCLables v0.1.0: Provides functions to assist manipulation of matrix row and column labels for all types of matrix mathematics where row and column labels are to be respected. See the vignette.

trampoline v0.1.1: Implements a trampoline algorithm based on on the Python trampoline module that enables users to write theoretically infinite recursive functions. See the vignette and README for examples.

Visualization

recolorize v0.1.0: Offers automatic, semi-automatic, and manual functions for generating color maps from images. The idea is to simplify the colors of an image according to a metric that is meaningful to the user, using deterministic methods whenever possible. There is an Introduction and a vignette for each of the six steps involved in the process: Acquisition & Preparation, Loading & Processing, Clustering, Refinement, Tweaks & Edits, and Export & Visualization.

scImmuneGraph v1.1.3: Provides functions to compute statistics and visualize multiple distributions including diversity, composition of clonotypes, abundance and length of CDR3, the abundance of V and J genes, and the abundance of V-J gene pairs; all of which are basic for single-cell immune group analysis. See the vignette.

November 2021: "Top 40" New CRAN Packages

Tue, 21 Dec 2021 00:00:00 +0000

Two hundred eleven new packages made it to CRAN in November. Here are my “Top 40” picks in thirteen categories: Computational Methods, Data, Ecology, Finance, Genomics, Humanities, Machine Learning, Medicine, Networks, Statistics, Time Series, Utilities, and Visualization. It was gratifying to see multiple packages developed for applications in the Computational Humanities. R is helping to extend the reach of data literacy.

Computational Methods

bigQF v1.6: Implements a computationally efficient leading-eigenvalue approximation to tail probabilities and quantiles of large quadratic forms (e.g. in the sequence kernel association test) and also provides stochastic singular value decomposition for dense or sparse matrices. See Lumley et al. (2018) for background and the vignettes: Checking pQF vs SKAT, Matrix-free computations, SKAT, weights, and projections.

LMMsolver v1.0.0: Implements an efficient system to solve sparse, mixed model equations. See the vignette.

nimbleAPT v1.0.4: Provides functions for adaptive parallel tempering with nimble models. See Lacki & Miasojedow (2016) and the vignette to get started.

Data

binancer v1.2.0: Implements an R client to the Binance Public Rest API for data collection on cryptocurrencies, portfolio management and trading. See README for an example.

japanstat v0.1.0: Provides tools for using the API of e-Stat, a portal site for Japanese government statistics which include functions for automatic query generation, data collection and formatting. See README for examples.

openeo v1.1.0: Provides to access data and processing for openEO compliant back-ends. See the vignette to get started.

Ecology

mFD v1.0.1: Provides functions to compute functional traits-based distances between pairs of species for species gathered in assemblages. Chao et al. (2018) and Mouillot et al. (2013) along with the other references listed for background. There are five vignettes including General Workflow and Compute and Interpret Quality of Functional Spaces

popbayes v1.0: Implements a Bayesian framework to infer the trends of animal populations over time from series of counts by accounting for count precision, smoothing the population rate of increase over time, and accounting for the maximum demographic potential of species. There is a Get started vignette and another on Getting species information.

Finance

MultiATSM v0.0.1: Provides functions to model the affine term structure of interest rates based on the single-country unspanned macroeconomic risk framework of Joslin et al. (2014) and the multi country extensions of Jotikasthira et al.(2015), and Candelon and Moura (2021). See the vignette for an example.

Genomics

gwasrapidd v0.99.12: Provides easy access to the NHGRI-EBI GWAS Catalog via the REST API. There is a Getting Started Guide, a vignette on Variants associated with BMI, and an FAQ.

phylosamp v0.1.6: Implements tools to estimate the probability of true transmission between two cases given phylogenetic linkage and the expected number of true transmission links in a sample. The methods are described in Wohl et al. (2021). There are vignettes on estimating FDR from sample size, Sample size from FDR, Sensitivity and specificity, and Examples.

Rtropical v1.2.1: Provides functions to process phylogenetic trees with tropical support vector machines and principal component analysis defined with tropical geometry. See Tang et al. (2020) for details about tropical SVMs, Page et al. (2020) and Yoshida et al. (2019) for tropical PCA, and Ardila & Develin (2007) for some background on tropical mathematics. The vignette will get you started.

Humanities

litRiddle v0.4.1: Provides a dataset, functions to explore the quality of literary novels, the data of a reader survey about fiction in Dutch, a description of the novels the readers rated, and the results of stylistic measurements of the novels. There is a vignette.

kairos v1.0.0: Provides a toolkit for absolute dating and analysis of chronological patterns, including functions for chronological modeling and dating of archaeological assemblages from count data. There is a Manual, and a Bibliography.

Machine Learning

cuda.ml v0.3.1: Implements an R interface for RAPIDS cuML, a suite of GPU-accelerated machine learning libraries powered by CUDA. Look here for an example.

innsight v0.1.1: Implements methods to analyze the behavior and individual predictions of modern neural networks including Connection Weights as described by Olden et al. (2004), Layer-wise Relevance Propagation as described by Bach et al. (2015), Deep Learning Important Features as described by Shrikumar et al. (2017), and gradient-based methods like SmoothGrad described by Smilkov et al. (2017), and Gradient x Input described by Baehrens et al. (2009). There is an Introduction and a vignette on Custom Model Definition.

topicmodels.etm v0.1.0: Provides functions to find topics in texts which are semantically embedded using techniques like word2vec or GloVe. See Dieng et al. (2019) for the details, and look here for an example.

Medicine

clustra v0.1.5: Provides functions to cluster medical trajectories of unequally spaced and unequal length time series aligned by an intervention time. See the vignette for examples.

eSIR v0.4.2: Implements the extended state-space SIR models developed by Song Lab at UM school of Public Health which include capabilities to model time-varying transmission, time-dependent quarantine, and time-dependent antibody-immunization. See Wang et al. (2020) for background, and README for examples.

QHScrnomo v2.2.0: Provides functions for fitting and predicting competing risk models, creating nomograms, k-fold cross validation, calculating the discrimination metric, and drawing calibration curves. See the vignette for a short tutorial.

rnmamod v0.1.0: Implements a comprehensive suite of functions to perform and visualize pairwise and network meta-analyses. The package covers core Bayesian one-stage models implemented in a systematic review with multiple interventions, including fixed-effect and random-effects network meta-analysis. There are vignettes on Performing a network meta-analysis and Describing the network.

whomds v1.0.1: Provides functions for calculating and presenting the results from the WHO Model Disability Survey (MDA). See Andrich (2014) for background on Rasch measurement. There are English and Spanish versions of six vignettes including Background on disability measurement and Best practices with Rasch Analysis.

Networks

diffudist v1.0.0: Enables the evaluation of diffusion distances for complex single-layer networks by providing functions to evaluate the Laplacians, stochastic matrices, and the corresponding diffusion distance matrices. See De Domenico (2017) and Bertagnolli & De Domenico (2021) for the details and the vignette for some theory and examples.

NetFACS V0.3.0: Provides functions to analyze and visualize facial communication data, based on network theory and resampling methods and primarily targeted at datasets of facial expressions coded with the Facial Action Coding System. See Farine (2017) and Carsey & Harden (2014) for background, and the vignette for an introduction.

Statistics

adass v1.0.0: Implements the adaptive smoothing spline estimator for the function-on-function linear regression model described in Centofanti et al. (2020). Look here for an example.

OneSampleMR v0.1.0: Provides functions for one sample Mendelian randomization and instrumental variable analyses including implementations of the Sanderson & Windmeijer (2016) conditional F-statistic, the multiplicative structural mean model Hernán and Robins (2006), and two-stage predictor substitution and two-stage residual inclusion estimators explained by Terza et al. (2008). There are short vignettes on the Multiplicative structural mean model and the Coditional F-Statistic.

PhaseTypeR v1.0.1: Implements functions to model continuous and discrete phase-type distributions, both univariate and multivariate. See Navarro (2019) for background. There is an Introduction and vignettes on Population Genetics and The Site Frequency Spectrum.

stan4bart v0.0-2: Fits semiparametric linear and multilevel models with non-parametric additive Bayesian additive regression trees BART and Stan. Multilevel models can be expressed using lme4 syntax. Look here to get started.

zoid v1.0.0: Fits Dirichlet regression and zero-and-one inflated Dirichlet regression (also called trinomial mixture models) with Bayesian methods implemented in Stan. There are vignettes on Fitting Models, Simulating data, Priors, and Prior sensitivity for overdispersion.

Time Series

surveil v0.1.0: Fits time series models for routine disease surveillance tasks and returns probability distributions for a variety of quantities of interest, including measures of health inequality, period and cumulative percent change, and age-standardized rates. See the vignette for an introduction.

VisitorCounts v1.0.0: Provides functions for modeling and forecasting of park visitor counts using social media data and (partial) on-site visitor counts. See Wood et al. (2013) for background and the vignette for examples.

Utilities

dateutils v0.1.5: Provides utilities for mixed frequency data, in particular, to aggregate and normalize tabular mixed frequency data, index dates to end of period, and seasonally adjust tabular data. There is an Introduction.

datefixR v0.1.2: Provides functions to fix messy dates such as those entered via text boxes. Standardizes “/” or “-”, whitespace separation, month abbreviations, and more. See the vignette.

lambdar v1.1.0: Provides functions for serving containers that can execute R code on the AWS Lambda serverless compute service. There are five vignettes including api-gateway-invocations and lambda-runtime-in-container.

mtb v0.1.1: Provides functions to summarize time related data, generate axis transformation from data, and assist Markdown and Shiny file editing. There are vignettes on working with axes, color, Markdown, and time related data.

rocker v0.2.1: Provides an R6 class interface for handling database connections using the DBI package as backend which allows handling of connections to PostgreSQL, MariaDB, SQLite and other databases. There are seven short vignettes including DBI package objects and functions in R6 rocker class and Database transactions with R6 rocker class.

shinySelect v1.0.0: Provides a customizable, select control widget for Shiny to enable using HTML in the items and KaTeX to type mathematics. Look here for examples.

Visualization

gggrid v0.1-1: Extendsggplot2 to make it easy to add raw grid output, such as customised annotations, to a plot. See the vignette for examples.

NHSRplotthedots v0.1.0: Provides tools for drawing statistical process control charts with functions to draw XmR charts, use change points and apply rules with summary indicators for when rules are breached. There is an Introduction and vignettes Deviations from Excel defaults and Number of points required.

shinymodels v0.1.0: Allows users to launch a shiny application for tidymodels results. For classification or regression models, the app can be used to determine if there is lack of fit or poorly predicted points. Look here to get started.

The COVID19 package, an interface to the COVID-19 Data Hub

Wed, 08 Dec 2021 00:00:00 +0000

The COVID-19 Data Hub provides a daily summary of COVID-19 cases, deaths, recovered, tests, vaccinations, and hospitalizations for 230+ countries, 760+ regions, and 12000+ administrative divisions of lower level. It includes policy measures, mobility, and geospatial data. This post presents version 3.0.0 of the COVID19 package to seamlessly import the data in R.

Why another package for COVID-19 data

Many packages now exist to retrieve COVID-19 related data from within R. As an example, covid19br retrieves case data for Brazil, covid19sf for San Francisco, covid19us for United States, covid19india for India, covid19italy for Italy, covid19swiss for Switzerland, covid19france for France, and so on. There also other packages, such as coronavirus, that retrieve national-level statistics worldwide from the Center for Systems Science and Engineering at Johns Hopkins University (JHU CCSE). However, national counts only represent a small portion of the available governmental data, and having the information scattered across many packages and different interfaces makes international comparisons of large, detailed outbreak data difficult, and prevents inferences from such data to be effective.

COVID19 is the official package created around COVID-19 Data Hub: a unified database harmonizing open governmental data around the globe at fine-grained spatial resolution. Moreover, as epidemiological data alone are typically of limited use, the database includes a set of identifiers to match the epidemiological data with exogenous indicators and geospatial information. By unifying the access to the data, this database makes it possible to study the pandemic in its global scale with high resolution, taking into account within-country variations, non pharmaceutical interventions, and environmental and exogenous variables.

In particular, this package allows you to download a large set of epidemiological variables, exogenous indicators from World Bank, mobility data from Google and Apple mobility reports, and geospatial information from Eurostat for Europe or GADM worldwide, in, literally, one line of code.

What’s new in version 3.0.0

Version 3 is a major update of COVID-19 Data Hub, which includes a great improvement in the spatial coverage, new data on vaccines, and a new set of identifiers to enable geospatial analyses. The full changelog is available here.

The large amount of data that is now available (~2GB) has led to some breaking changes in the way the data are provided. Version 3 of the COVID19 package is designed to be compatible with the latest version of COVID-19 Data Hub, and process large amount of data at speed with low memory requirements. The documentation and a quick start guide is available here.

Data coverage

The figure shows the granularity and the spatial coverage of the data as of November 27, 2021.

What’s included?

library(COVID19)  # load the package
x <- covid19()    # download the data

Refer to the documentation for the description of each variable.

colnames(x)

##  [1] "id"                                  "date"                               
##  [3] "confirmed"                           "deaths"                             
##  [5] "recovered"                           "tests"                              
##  [7] "vaccines"                            "people_vaccinated"                  
##  [9] "people_fully_vaccinated"             "hosp"                               
## [11] "icu"                                 "vent"                               
## [13] "school_closing"                      "workplace_closing"                  
## [15] "cancel_events"                       "gatherings_restrictions"            
## [17] "transport_closing"                   "stay_home_restrictions"             
## [19] "internal_movement_restrictions"      "international_movement_restrictions"
## [21] "information_campaigns"               "testing_policy"                     
## [23] "contact_tracing"                     "facial_coverings"                   
## [25] "vaccination_policy"                  "elderly_people_protection"          
## [27] "government_response_index"           "stringency_index"                   
## [29] "containment_health_index"            "economic_support_index"             
## [31] "administrative_area_level"           "administrative_area_level_1"        
## [33] "administrative_area_level_2"         "administrative_area_level_3"        
## [35] "latitude"                            "longitude"                          
## [37] "population"                          "iso_alpha_3"                        
## [39] "iso_alpha_2"                         "iso_numeric"                        
## [41] "iso_currency"                        "key_local"                          
## [43] "key_google_mobility"                 "key_apple_mobility"                 
## [45] "key_jhu_csse"                        "key_nuts"                           
## [47] "key_gadm"

Data transparency

This package applies no pre-processing to the original data, that are provided as-is. The data acquisition pipeline is open source and all the original data providers are listed here.

As an example, the following code snippet plots the fraction of confirmed cases on a given day per number of tests performed on that day in U.S. Notice that around June 2021, the fraction becomes negative. This is a known issue due to decreasing cumulative counts in the original data provider. This package applies no cleaning procedure for this kind of issues, which are typically due to changes in the data collection methodology. If the provider corrects the data retroactively, the changes are reflected in this package.

library(xts)
library(dygraphs)
x <- covid19("USA", verbose = FALSE)  # download the data
ts <- xts(x[,c("confirmed", "tests")], order.by = x$date)  # convert to an xts object
ts$ratio <- diff(ts$confirmed) / diff(ts$tests)  # compute daily ratio
dygraph(ts$ratio, main = "Daily fraction confirmed/tests in U.S.")  # plot

World Bank data

Country-level covariates by World Bank Open Data can be easily added. Refer to the table at the bottom of this page for relevant indicators. The following code snippet shows e.g., how to download the number of hospital beds for each country. Refer to the quickstart guide for more details.

x <- covid19(wb = c("hosp_beds" = "SH.MED.BEDS.ZS"), verbose = FALSE)

Mobility data

Mobility data are obtained from Google and Apple mobility reports. The following example shows how to download the data by Google. Refer to the quickstart guide for Apple’s reports and for more details.

x <- covid19(gmr = TRUE, verbose = FALSE)
colnames(x[,48:53])

## [1] "retail_and_recreation_percent_change_from_baseline"
## [2] "grocery_and_pharmacy_percent_change_from_baseline" 
## [3] "parks_percent_change_from_baseline"                
## [4] "transit_stations_percent_change_from_baseline"     
## [5] "workplaces_percent_change_from_baseline"           
## [6] "residential_percent_change_from_baseline"

Spatial data

The dataset contains NUTS codes to match the Eurostat database for Europe, and GID codes to match the GADM worldwide database. The following example shows how to access spatial data using GADM for U.S. counties. Similar maps are available worldwide for most other countries at the various granularity levels.

First, download level 3 data for U.S.

x <- covid19("USA", level = 3, verbose = FALSE)

GADM data by country can be found here. Download the geopackage for U.S. using GADM version 3.6:

url <- "https://biogeo.ucdavis.edu/data/gadm3.6/gpkg/gadm36_USA_gpkg.zip"
zip <- tempfile()
download.file(url, destfile = zip)

Unzip the geopackage:

exdir <- tempfile()
unzip(zip, exdir = exdir)

Load the sf package and list the layers:

library(sf)
file <- paste0(exdir, "/gadm36_USA.gpkg")
st_layers(file)

## Driver: GPKG 
## Available layers:
##     layer_name geometry_type features fields
## 1 gadm36_USA_2 Multi Polygon     3148     13
## 2 gadm36_USA_1 Multi Polygon       51     10
## 3 gadm36_USA_0 Multi Polygon        1      2

Read layer 2 that corresponds to U.S. counties. Note: in general, there is not a perfect correspondence between GADM layers and the granularity level from this package. It is recommended to read all the layers, and match on the corresponding GID. Read more about how key_gadm from this package is mapped to the corresponding GID here.

g <- st_read(file, layer = "gadm36_USA_2")

## Reading layer `gadm36_USA_2' from data source 
##   `/private/var/folders/w0/skxpg0h51jg72m5b01y_8n_c0000gn/T/RtmpmUy7P6/file56232c611dfd/gadm36_USA.gpkg' 
##   using driver `GPKG'
## Simple feature collection with 3148 features and 13 fields
## Geometry type: MULTIPOLYGON
## Dimension:     XY
## Bounding box:  xmin: -179.2 ymin: 18.91 xmax: 179.8 ymax: 72.69
## Geodetic CRS:  WGS 84

Subset the data to extract only the counts as of, e.g., 15 November 2021. Select only the administrative divisions inside the following bounding box for better visualization.

x <- x[
  x$date == "2021-11-15" & 
  x$latitude > 24.9493 & x$latitude < 49.5904 &
  x$longitude > -125.0011 & x$longitude < -66.9326,]

Merge COVID-19 data with the spatial data:

library(dplyr)
gx <- right_join(g, x, by = c("GID_2" = "key_gadm"))

Plot e.g., the total number of confirmed cases:

plot(gx["confirmed"], logz = TRUE)

Academic publications

See the publications that use COVID-19 Data Hub.

Cite as

Guidotti, E., Ardia, D., (2020), “COVID-19 Data Hub”, Journal of Open Source Software 5(51):2376, doi: 10.21105/joss.02376.

A BibTeX entry for LaTeX users is

@Article{,
    title = {COVID-19 Data Hub},
    year = {2020},
    doi = {10.21105/joss.02376},
    author = {Emanuele Guidotti and David Ardia},
    journal = {Journal of Open Source Software},
    volume = {5},
    number = {51},
    pages = {2376}
}

October 2021: "Top 40" New CRAN Packages

Mon, 29 Nov 2021 00:00:00 +0000

One hundred forty-one new packages made it to CRAN in October. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Networks, Science, Social Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

gslnls v1.0.2: Implements an R interface to nonlinear least-squares optimization with the GNU Scientific Library (GSL). See M. Galassi et al. (2009).

rsvddpd v1.0.0: Implements a robust method for computing the singular value decomposition using power density divergence. See Roy et. al (2021) for background and the vignette for an example.

Data

blsR v0.2.1: Implements v2 of the Bureau of Labor Statistics API for requests of survey information and time series data. See README to get started.

insiderTrades v0.0.1: Provides functions to download insider trading transactions and insider holdings from a public NoSQL SEC database using keyword criteria and generate a relational dataframe. See the vignettes insiderTrades and pullAndScrape.

KingCountyHouses v0.1.0: Contains data on houses in and around Seattle WA are included. Basic characteristics are given along with sale prices. See README.

meteospain v0.0.3: Provides access to different Spanish meteorological stations data services and APIs including AEMET, SMC, MG, and Meteoclimatic. There are seven short vignettes including API limits and loops and Compatibility between services.

owidR v1.1.0: Provides function to import, search, download, and visualize data from the Our World in Data website. See the vignette.

QBMS v0.6: Enables users to query the Breeding Management System database. There is a short Tutorial.

Genomics

digitalDLSorteR v0.1.1: Implements tools for the deconvolution of bulk RNA-Seq data using context-specific deconvolution models based on Deep Neural Networks using scRNA-Seq data as input. See Torroja & Sanchez-Cabo (2019) for details. There are five vignettes including Keras/TensorFlow installation and configuration and Building new deconvolution models.

Platypus v3.2.3: Implements an open-source software platform for investigating B-cell receptor and T-cell receptor repertoires from scSeq experiments which also incorporates transcriptional information involving unsupervised clustering, gene expression and gene ontology. See Yermanos et al. (2021) and the vignette to get started.

quincunx v0.1.4: Provides programmatic access to the PGS Catalog through the REST API. See README for the cheatsheet.

SNPfilter v0.1.0: Provides functions to interactively and reproducibly visualize and filter SNP (single-nucleotide polymorphism) datasets, including functions for visualizing various quality and missing data metrics for a SNP dataset, and then filtering the dataset based on user specified cutoffs. See the vignette.

Machine Learning

conText v1.0.0: Implements a framework to estimate context-specific word and short document embeddings using the a la carte embeddings approach developed by Khodak et al. (2018) and evaluate hypotheses about covariate effects on embeddings using the regression framework developed by Rodriguez et al. (2021). There is a Quick Start Guide.

pixelclassifier v1.0.0: Implements a Support Vector Machine to classify the pixels of an image file by its color. See the vignette.

predtools v0.0.2: Provides functions for evaluating predictive models, including plotting calibration curves and model-based Receiver Operating Characteristic (mROC) curves based on Sadatsafavi et al. (2021). There are three vignettes: Calibration Plot, Intercept Adjustment, and Model-based ROC.

text2sdg v0.1.1: Provides functions to identify Sustainable Development Goals in text using scientifically developed query systems, opening up the opportunity to monitor any type of text-based data, such as scientific output or corporate publications. There is a vignette.

Twitmo v0.1.1: Provides functions to collect, pre-process and analyze the contents of tweets using LDA and STM models including functions to generate tweet and hashtag maps and built-in support for LDAvis. See README for examples.

Medicine

ctrialsgov v0.2.5: Provides tools to access and query the U.S. National Library of Medicine’s Clinical Trials database including functions for searching the data using range queries, categorical filtering, and by searching for full-text keywords. There is an Introduction and a vignette on Text Analysis.

epidemia v1.0.0: Implements functions to specify and fit Bayesian statistical models for epidemics. Infections are propagated over time using self-exciting point processes. Multiple regions can be modeled simultaneously with multilevel models. See Bhatt et al. (2021) for background on the models and the package website for examples and extensive documentation.

Networks

dissCqN v0.1.0: Provides functions to calculate multiple or pairwise dissimilarity for orders q = 0-N for a set of species assemblages or interaction networks. See Chao et al. 2008 for background and the vignette for an example.

modnets v0.9.0: Implements methods for modeling moderator variables in cross-sectional, temporal, and multi-level networks including model selection techniques and a variety of plotting functions. See Swanson (2020) for details and the README for an example.

Science

cmpsR v0.1.0: Implements the Congruent Matching Profile Segments (CMPS) method Chen et al. (2019) to provide an objective comparison of striated tool marks for fired bullet correlation. See the vignette.

pvcurveanalysis v1.0.0: Provides functions to analyze and display pressure volume curves which enable the derivation of the turgor loss point, osmotic potential and apoplastic fraction. See Bartlett et al. (2012) for background and the vignette for examples.

rcbayes v0.1.0: Provides functions to estimate Rogers-Castro migration age schedules using Stan. See Rogers and Castro (1978) for background and the vignettes on Model Convergence and Rogers Castro Migration Models.

QCAcluster v0.1.0: Provides functions to allow Qualitative Comparative Analysis researchers to supplement the analysis of pooled data with a disaggregated perspective focusing on selected partitions of the data. The pooled data can be partitioned along the dimensions of the clustered data (individual cross sections or time series) to perform partition-specific truth table minimizations. There are vignettes that discuss the Aggregation, Diversity, Miimization, and Weight of partitions.

Statistics

bayesPO v0.3.1: Provides functions to create Bayesian point process models of presence only data. See Moreira (2020) for background, and the vignette for examples.

hierbase v0.1.2: Implements the algorithms for hierarchical testing of variable importance described in Meinshausen (2008) which control for family-wise error rate. See the vignette for an example.

pedmod v0.1.0: Provides functions to estimate mixed probit models, commonly called liability threshold models, for pedigree data such as that studied in Pawitan et al. (2004). See Genz & Bretz (2012) for background. There is an introduction to pedigree models and a vignette on positioning families.

slasso v1.0.0: Implements the smooth LASSO estimator for the function-on-function linear regression model described in Centofanti et al. (2020). See README for examples.

StanMoMo v1.0.0: Implements Bayesian mortality models including those developed in Lee & Carter (1992) and Cairns, Blake & Dowd (2006). There is a vignette introducing the Lee & Carter model and another on Bayesian Model Averaging.

Time Series

esemifar v1.0.1: Implements algorithms that provide non-parametric estimates of trend and its derivatives in equidistant time series with long-memory errors. See Letmathe et al. (2021) for a description of the smoothing methods employed, and README for an example.

GDPuc v0.5.1: Provides a function to convert GDP time series from one unit to another. All common GDP units are included, i.e. current and constant local currency units, US$ via market exchange rates and international dollars via purchasing power parities. See README for examples.

Utilities

framecleaner v0.2.0: Provides a friendly interface for modifying data frames with a sequence of piped commands built upon the tidyverse. See the vignette for examples.

presentr v0.1.1: Implements wrapper functions using packages openxlsx, flextable, and officer to create highly formatted MS office friendly output of data frames. Vignettes include: exportToExcel, flextableAndPowerpoint, and formattedFlextable.

shinyGovstyle v0.0.7: Implements a collection of shiny application styling tools that are the based on the GOV.UK Design System. See README for an introduction and examples.

validata v0.1.0: Provides functions for validating the structure and properties of data frames helping users to answer essential questions about a data set such as: What are the unique or missing values? and What columns form a primary key?. See the vignette for an introduction.

Visualization

easylabel v0.2.4: Implements interactive labeling of scatter plots, volcano plots and Manhattan plots using a shiny or plotly interface. Users can hover over points to see where specific points are located and click points on/off to easily label them. See the vignette for examples.

ggokabeito v0.1.0: Provides discrete scales for the colorblind-friendly Okabe-Ito palette, including color, fill, and edge_colour. See README for examples.

metaconfoundr v0.1.0: Implements an approach for evaluating bias in meta-analysis studies based on the causal inference framework. See the vignette for examples.

scatterPlotMatrix v0.1.0: Provides functions which make use of the packages htmlwidgets package and d3.js to create scatter plot matrices. See the vignette.

How to Scrape and Store Strava Data Using R

Mon, 22 Nov 2021 00:00:00 +0000

This post by Julian During is the third place winner in the Call for Documentation contest. Julian is a data scientist from Germany working in the manufacturing industry. Julian loves working with R (especially the tidyverse ecosystem), sports, black coffee and cycling.

I am an avid runner and cyclist. For the past couple of years, I have recorded almost all my activities on some kind of GPS device.

I record my runs with a Garmin device and my bike rides with a Wahoo device, and I synchronize both accounts on Strava. I figured that it would be nice to directly access my data from my Strava account.

In the following text, I will describe the progress to get Strava data into R, process the data, and then create a visualization of activity routes. You can find the original analysis in this Github repository.

You will need the following packages:

library(tarchetypes)
library(conflicted)
library(tidyverse)
library(lubridate)
library(jsonlite)
library(targets)
library(httpuv)
library(httr)
library(pins)
library(httr)
library(fs)
library(readr)

conflict_prefer("filter", "dplyr")

Set Up Targets

The whole data pipeline is implemented with the help of the targets package. You can learn more about the package and its functionalities here.

In order to reproduce the analysis, perform the following steps:

Clone the repository: https://github.com/duju211/pin_strava
Install the packages listed in the ‘libraries.R’ file
Run the target pipeline by executing targets::tar_make() command
Follow the instructions printed in the console

Target Plan

The manifest of the target plan looks like this:

name	command	pattern	cue_mode
my_app	define_strava_app()	NA	thorough
my_endpoint	define_strava_endpoint()	NA	thorough
act_col_types	list(moving = col_logical(), velocity_smooth = col_number(), grade_smooth = col_number(), distance = col_number(), altitude = col_number(), time = col_integer(), lat = col_number(), lng = col_number(), cadence = col_integer(), watts = col_integer())	NA	thorough
my_sig	define_strava_sig(my_endpoint, my_app)	NA	always
df_act_raw	read_all_activities(my_sig)	NA	thorough
df_act	pre_process_act(df_act_raw, athlete_id)	NA	thorough
act_ids	pull(distinct(df_act, id))	NA	thorough
df_meas	read_activity_stream(act_ids, my_sig)	map(act_ids)	never
df_meas_all	bind_rows(df_meas)	NA	thorough
df_meas_wide	meas_wide(df_meas_all)	NA	thorough
df_meas_pro	meas_pro(df_meas_wide)	NA	thorough
gg_meas	vis_meas(df_meas_pro)	NA	thorough
df_meas_norm	meas_norm(df_meas_pro)	NA	thorough
gg_meas_save	save_gg_meas(gg_meas)	NA	thorough

We will go through the most important targets in detail.

OAuth Dance from R

The Strava API requires an ‘OAuth dance’, described below.

1. Create an OAuth Strava app
To get access to your Strava data from R, you must first create a Strava API. The steps are documented on the Strava Developer site. While creating the app, you’ll have to give it a name. In my case, I named it r_api.

After you have created your personal API, you can find your Client ID and Client Secret variables in the Strava API settings. Save the Client ID as STRAVA_KEY and the Client Secret as STRAVA_SECRET in your R environment.¹

STRAVA_KEY=<Client ID>
STRAVA_SECRET=<Client Secret>

Then, you can run the function define_strava_app shown below.

name	command	pattern	cue_mode
my_app	define_strava_app()	NA	thorough

define_strava_app <- function() {
  oauth_app(
    appname = "r_api",
    key = Sys.getenv("STRAVA_KEY"),
    secret = Sys.getenv("STRAVA_SECRET")
  )
}

2. Define an endpoint

Define an endpoint called my_endpoint using the function define_strava_endpoint.

The authorize parameter describes the authorization url and the access argument exchanges the authenticated token.

name	command	pattern	cue_mode
my_endpoint	define_strava_endpoint()	NA	thorough

define_strava_endpoint <- function() {
  oauth_endpoint(request = NULL,
                 authorize = "https://www.strava.com/oauth/authorize",
                 access = "https://www.strava.com/oauth/token")
}

3. The final authentication step
Before you can execute the following steps, you have to authenticate the API in the web browser.

name	command	pattern	cue_mode
my_sig	define_strava_sig(my_endpoint, my_app)	NA	always

define_strava_sig <- function(endpoint, app) {
  oauth2.0_token(
    endpoint,
    app,
    scope = "activity:read_all,activity:read,profile:read_all",
    type = NULL,
    use_oob = FALSE,
    as_header = FALSE,
    use_basic_auth = FALSE,
    cache = FALSE
  )
}

The information in my_sig can now be used to access Strava data. Set the cue_mode of the target to ‘always’ so that the following API calls are always executed with an up-to-date authorization token.

Access Activities

You are now authenticated and can directly access your Strava data.

1. Load all activities
Load a table that gives an overview of all the activities from the data. Because the total number of activities is unknown, use a while loop. It will break the execution of the loop if there are no more activities to read.

name	command	pattern	cue_mode
df_act_raw	read_all_activities(my_sig)	NA	thorough

read_all_activities <- function(sig) {
  activities_url <- parse_url("https://www.strava.com/api/v3/athlete/activities")

  act_vec <- vector(mode = "list")
  df_act <- tibble::tibble(init = "init")
  i <- 1L

  while (nrow(df_act) != 0) {
    r <- activities_url %>%
      modify_url(query = list(
        access_token = sig$credentials$access_token[[1]],
        page = i
      )) %>%
      GET()

    df_act <- content(r, as = "text") %>%
      fromJSON(flatten = TRUE) %>%
      as_tibble()
    if (nrow(df_act) != 0)
      act_vec[[i]] <- df_act
    i <- i + 1L
  }

  df_activities <- act_vec %>%
    bind_rows() %>%
    mutate(start_date = ymd_hms(start_date))
}

The resulting data frame consists of one row per activity:

## # A tibble: 605 x 60
##    resource_state name  distance moving_time elapsed_time total_elevation~ type 
##             <int> <chr>    <dbl>       <int>        <int>            <dbl> <chr>
##  1              2 "Hes~   31153.        4699         5267            450   Ride 
##  2              2 "Bam~    5888.        2421         2869            102.  Run  
##  3              2 "Lin~   33208.        4909         6071            430   Ride 
##  4              2 "Mon~   74154.       10721        12500            641   Ride 
##  5              2 "Cha~   34380         5001         5388            464.  Ride 
##  6              2 "Mor~    5518.        2345         2563             49.1 Run  
##  7              2 "Bin~   10022.        3681         6447            131   Run  
##  8              2 "Tru~   47179.        8416        10102            898   Ride 
##  9              2 "Sho~   32580.        5646         6027            329.  Ride 
## 10              2 "Mit~   33862.        5293         6958            372   Ride 
## # ... with 595 more rows, and 53 more variables: workout_type <int>, id <dbl>,
## #   external_id <chr>, upload_id <dbl>, start_date <dttm>,
## #   start_date_local <chr>, timezone <chr>, utc_offset <dbl>,
## #   start_latlng <list>, end_latlng <list>, location_city <lgl>,
## #   location_state <lgl>, location_country <chr>, start_latitude <dbl>,
## #   start_longitude <dbl>, achievement_count <int>, kudos_count <int>,
## #   comment_count <int>, athlete_count <int>, photo_count <int>, ...

2. Preprocess activities
Make sure that all ID columns have a character format and improve the column names.

name	command	pattern	cue_mode
df_act	pre_process_act(df_act_raw, athlete_id)	NA	thorough

pre_process_act <- function(df_act_raw, athlete_id) {
  df_act <- df_act_raw %>%
    mutate(across(contains("id"), as.character),
           `athlete.id` = athlete_id)
}

3. Extract activity IDs
Use dplyr::pull() to extract all activity IDs.

name	command	pattern	cue_mode
act_ids	pull(distinct(df_act, id))	NA	thorough

Read Measurements

1. Read the ‘stream’ data from Strava
A ‘stream’ is a nested list (JSON format) with all available measurements of the corresponding activity.

To get the
available variables and turn the result into a data frame, define a helper function read_activity_stream. This function takes an ID of an activity and an authentication token, which you created earlier.

The target is defined with dynamic branching which maps over all activity IDs. Define the cue mode as never to make sure that every target runs exactly once.

name	command	pattern	cue_mode
df_meas	read_activity_stream(act_ids, my_sig)	map(act_ids)	never

read_activity_stream <- function(id, sig) {
  act_url <-
    parse_url(stringr::str_glue("https://www.strava.com/api/v3/activities/{id}/streams"))
  access_token <- sig$credentials$access_token[[1]]

  r <- modify_url(act_url,
                  query = list(
                    access_token = access_token,
                    keys = str_glue(
                      "distance,time,latlng,altitude,velocity_smooth,cadence,watts,
                      temp,moving,grade_smooth"
                    )
                  )) %>%
    GET()

  stop_for_status(r)

  fromJSON(content(r, as = "text"), flatten = TRUE) %>%
    as_tibble() %>%
    mutate(id = id)
}

2. Bind the single targets into one data frame
You can do this using dplyr::bind_rows().

name	command	pattern	cue_mode
df_meas_all	bind_rows(df_meas)	NA	thorough

The data now is represented by one row per measurement series:

## # A tibble: 4,821 x 6
##    type            data              series_type original_size resolution id    
##    <chr>           <list>            <chr>               <int> <chr>      <chr> 
##  1 moving          <lgl [4,706]>     distance             4706 high       62186~
##  2 latlng          <dbl [4,706 x 2]> distance             4706 high       62186~
##  3 velocity_smooth <dbl [4,706]>     distance             4706 high       62186~
##  4 grade_smooth    <dbl [4,706]>     distance             4706 high       62186~
##  5 distance        <dbl [4,706]>     distance             4706 high       62186~
##  6 altitude        <dbl [4,706]>     distance             4706 high       62186~
##  7 heartrate       <int [4,706]>     distance             4706 high       62186~
##  8 time            <int [4,706]>     distance             4706 high       62186~
##  9 moving          <lgl [301]>       distance              301 high       62138~
## 10 latlng          <dbl [301 x 2]>   distance              301 high       62138~
## # ... with 4,811 more rows

3. Turn the data into a wide format

name	command	pattern	cue_mode
df_meas_wide	meas_wide(df_meas_all)	NA	thorough

meas_wide <- function(df_meas) {
  pivot_wider(df_meas, names_from = type, values_from = data)
}

In this format, every activity is one row again:

## # A tibble: 605 x 14
##    series_type original_size resolution id         moving latlng velocity_smooth
##    <chr>               <int> <chr>      <chr>      <list> <list> <list>         
##  1 distance             4706 high       6218628649 <lgl ~ <dbl ~ <dbl [4,706]>  
##  2 distance              301 high       6213800583 <lgl ~ <dbl ~ <dbl [301]>    
##  3 distance             4905 high       6179655557 <lgl ~ <dbl ~ <dbl [4,905]>  
##  4 distance            10640 high       6160486739 <lgl ~ <dbl ~ <dbl [10,640]> 
##  5 distance             4969 high       6153936896 <lgl ~ <dbl ~ <dbl [4,969]>  
##  6 distance             2073 high       6115020306 <lgl ~ <dbl ~ <dbl [2,073]>  
##  7 distance             1158 high       6097842884 <lgl ~ <dbl ~ <dbl [1,158]>  
##  8 distance             8387 high       6091990268 <lgl ~ <dbl ~ <dbl [8,387]>  
##  9 distance             5587 high       6073551706 <lgl ~ <dbl ~ <dbl [5,587]>  
## 10 distance             5281 high       6057232328 <lgl ~ <dbl ~ <dbl [5,281]>  
## # ... with 595 more rows, and 7 more variables: grade_smooth <list>,
## #   distance <list>, altitude <list>, heartrate <list>, time <list>,
## #   cadence <list>, watts <list>

4. Preprocess and unnest the data

The column latlng needs special attention, because it contains latitude and longitude information. Separate the two measurements before unnesting all list columns.

name	command	pattern	cue_mode
df_meas_pro	meas_pro(df_meas_wide)	NA	thorough

meas_pro <- function(df_meas_wide) {
  df_meas_wide %>%
    mutate(
      lat = map_if(
        .x = latlng,
        .p = ~ !is.null(.x),
        .f = ~ .x[, 1]
      ),
      lng = map_if(
        .x = latlng,
        .p = ~ !is.null(.x),
        .f = ~ .x[, 2]
      )
    ) %>%
    select(-c(latlng, original_size, resolution, series_type)) %>%
    unnest(where(is_list))
}

After this step, every row is one point in time and every column is a measurement at this point in time (if there was any activity at that moment).

## # A tibble: 2,176,926 x 12
##    id      moving velocity_smooth grade_smooth distance altitude heartrate  time
##    <chr>   <lgl>            <dbl>        <dbl>    <dbl>    <dbl>     <dbl> <dbl>
##  1 621862~ FALSE             0             1.8      0       527        149     0
##  2 621862~ TRUE              0             1.2      5       527.       150     1
##  3 621862~ TRUE              0             0.9     10.9     527.       150     2
##  4 621862~ TRUE              5.68          0.8     17       527.       150     3
##  5 621862~ TRUE              5.81          0.8     23.3     527.       150     4
##  6 621862~ TRUE              5.88          0.8     29.4     527.       150     5
##  7 621862~ TRUE              6.13          0.8     35.6     527.       151     6
##  8 621862~ TRUE              6.15          0       41.6     527.       150     7
##  9 621862~ TRUE              6.14          0       47.8     527.       150     8
## 10 621862~ TRUE              6.13          0.8     53.9     527.       150     9
## # ... with 2,176,916 more rows, and 4 more variables: cadence <dbl>,
## #   watts <dbl>, lat <dbl>, lng <dbl>

Create Visualisation

Visualize the final data by displaying the geospatial information in the data. Every facet is one activity. Keep the rest of the plot as minimal as possible.

name	command	pattern	cue_mode
gg_meas	vis_meas(df_meas_pro)	NA	thorough

vis_meas <- function(df_meas_pro) {
  df_meas_pro %>%
    filter(!is.na(lat)) %>%
    ggplot(aes(x = lng, y = lat)) +
    geom_path() +
    facet_wrap( ~ id, scales = "free") +
    theme(
      axis.line = element_blank(),
      axis.text.x = element_blank(),
      axis.text.y = element_blank(),
      axis.ticks = element_blank(),
      axis.title.x = element_blank(),
      axis.title.y = element_blank(),
      legend.position = "bottom",
      panel.background = element_blank(),
      panel.border = element_blank(),
      panel.grid.major = element_blank(),
      panel.grid.minor = element_blank(),
      plot.background = element_blank(),
      strip.text = element_blank()
    )
}

And there it is: All your Strava data in a few tidy data frames and a nice-looking plot. Future updates to the data shouldn’t take too long, because only measurements from new activities will be downloaded. With all your Strava data up to date, there are a lot of appealing possibilities for further data analyses of your fitness data.

Note from the Editor: Julian’s post neatly breaks down complex tasks, walking readers through the steps as well as rationale of his decisions. His use of the targets package demonstrates how an organized workflow enables replicability and ease. In addition, Julian showcases how the R programming language can fulfill a vision sparked by one’s passions. It is an inspiring example of how we can use R to create something that is informative, beautiful, and personal.

You can edit your R environment by running usethis::edit_r_environ(), saving the keys, and then restarting R.↩︎

Deploying xaringan Slides with GitHub Pages

Thu, 18 Nov 2021 00:00:00 +0000

Silvia Canelón is a researcher, community organizer, and R educator. Her research leverages electronic health record data to study pregnancy-related outcomes, and her organizing values data literacy and communication as ways to build power and effect change.

This post will guide you step-by-step through the process of creating an HTML xaringan slide deck and deploying it to the web for easy sharing with others. We will be using the xaringan package to build the slide deck, GitHub to help us host our slides for free with GitHub Pages, and the usethis package to help us out along the way. You will get the most out of this workflow if you are already familiar with R Markdown and GitHub, and if you have already connected RStudio (or your preferred IDE) to Git and GitHub.¹ The post will not cover the nuts and bolts of xaringan or talk about slide design & customization, but you can find lots of learning resources listed at the end.

Choose your own adventure

Option 1: Start at the beginning of the workflow to make a slide deck using the R Markdown template built into the xaringan package. The built-in template doubles as documentation for the xaringan package, so it is a great way to familiarize yourself with the package features, but it also includes a lot of content that will probably want to remove and modify when creating your presentation.
Option 2: Start with an R-Ladies themed xaringan template (embedded below). This is an example slide deck originally created as a teaching tool to highlight some of the main features of the xaringan package, and to demo some customization that incorporates the R-Ladies CSS theme built into xaringan. Please feel welcome to use/modify it to suit your needs! When you are ready, you can follow the steps immediately below 👇 to download the files to your machine, and then skip down to Initialize version control with git.

usethis::use_course(
  repo_spec = "spcanelon/RLadies-xaringan-template",
  destdir = "filepath/for/your/presentation"
  )

Note: After copying the files to your machine you’ll probably want to rename the file folder to whatever makes sense for your presentation.

Try navigating through the slides ☝️ with your left/right arrow keys and press the letter “P” on your keyboard to see some notes in Presenter View.

The Ten-Step Workflow

Packages

This workflow was developed using:

Software / package	Version
R	4.0.3
RStudio	1.4.1103
`xaringan`	0.19
`usethis`	2.0.0

install.packages("xaringan")
install.packages("usethis")

Creating your xaringan slide deck

1. Create a new RStudio project for your presentation:

usethis::create_project("filepath/for/your/presentation/repo-name")

📍 If you’re not sure where you are on your computer, check your working directory with getwd() and use it as a filepath reference point

2. Create a xaringan deck using a xaringan template:
File > New File > R Markdown > From Template > Ninja Presentation > OK

3. Delete what you don’t need and save your R Markdown file with whatever name you like. If you pick index.Rmd the live link you share at the end will be relatively short.

4. Render HTML slides from the open Rmd file using xaringan’s infinite moon reader:

  xaringan::infinite_moon_reader()

Initialize version control with git

5. Initialize version control of your slides with git:

usethis::use_git()

You’ll be asked if you want to commit the files in your project (with the message “Initial commit”) and then if you want to restart to activate the Git pane. Say yes to both ✅

Note: At the moment usethis names the primary branch “master” by default. Issue #1341 suggests the option to instead name it “main” is in the works.

6. Connect your local project with a GitHub repo:

usethis::use_github()

You could use the function argument private = TRUE to create a private GitHub repository. But you may have to remember to change the visibility before deploying to GitHub Pages.

7. Your new GitHub repo with all of your xaringan project files will automatically open up in your browser

Repo for the R-Ladies xaringan template:
https://github.com/spcanelon/RLadies-xaringan-template

Making and committing changes to your slides

8. Edit your slides as you wish. Commit often! And then push to GitHub. Use the tools provided by the Git pane in RStudio, or use the following commands in the Terminal:

# Step 1: Stage all modified files
git add .

# Step 2: Describe the changes you made to your files
git commit -m "<A brief but descriptive commit message>"

Consider writing a commit message that finishes the following sentence:² “If applied, this commit will…” (e.g. “Change the slide theme”, “Add hello slide”)

# Step 3: Push the changes to your GitHub repository
git push

Deploying your slides

9. When you’re ready to deploy your slides, you can use the usethis::use_github_pages() function which makes the process of deploying via GitHub Pages super easy. I recommend pointing branch to the name of your primary branch.

usethis::use_github_pages(branch = "master")

Note: Your repository must be public for your deployed slides to be available publicly, unless you have a paid GitHub account.

Also, you only need to follow this step once to deploy your slides to the web. As long as you remember to push to your repo any changes that you make to your slides (Rmd and HTML), GitHub Pages will know how to render them.

10. Visit the link provided to see your newly deployed slides! 🚀
Don’t panic if you don’t see them right away, sometimes it takes a little time. This is the link you will share with the world when you present. Notice it looks very similar to your GitHub repo link.

Link to the R-Ladies xaringan template rendered slides:
https://spcanelon.github.io/RLadies-xaringan-template

Bonus steps

11. Go to the repository home page and find the About section on the right hand side. Add a description of your presentation and the link to your slides, that way your presentation is easily available to anyone visiting your repo.

12. Check out Garrick Aden-Buie’s blog post Sharing Your xaringan Slides to learn how to create a social media card for your slides and use your new link to share your slides in more places (e.g. embedded on a website, etc.)

13. This GitHub Pages workflow is not exclusive to xaringan slides! Try it out with any other HTML file.

Learn more

Foundation

Sharing Your Work with xaringan • Silvia Canelón – workshop site
Introducción al Paquete xaringan • Silvia Canelón – R-Ladies Meetup
Making Slides with R Markdown • Alison Hill – workshop slides
Presentation Ninja • xaringan Official Document – package documentation
Chapter 7 xaringan Presentations • R Markdown: The Definitive Guide – book chapter

Making your slides extra special

Professional, Polished, Presentable • Garrick Aden‑Buie & Silvia Canelón • useR!2021 – workshop site
Home · yihui/xaringan Wiki • GitHub – wiki of customizations for xaringan
Making Extra Great Slides • Garrick Aden‑Buie – talk & slides with xaringan overview and featuring CSS styling and xaringanthemer
Applying design guidelines to slides with {xaringanthemer} • katie jolly – blog post
A playground of extensions for xaringan • xaringanExtra – package site
Custom xaringan CSS Themes • xaringanthemer – package site

Note from the Editor: Silvia’s post is a mini masterpiece of clear, concise writing that elucidates complex technology within the narrow context of explaining a single well-defined task. Silvia does not attempt to say everything she knows about the subject, and she resists digressions that might obscure the path she is laying out. It is an example of achieving clarity through saying less.

An R Community Public Library

Thu, 04 Nov 2021 00:00:00 +0000

If you haven’t recently visited bookdown.org, RStudio’s free site for publishing books written with the bookdown R package, you many be amazed at what is available. Currently, there are over one hundred fifty titles listed under the Books tab. These are written in a panoply of languages including Bulgarian, Chinese, English, French, German, Hindi, Italian, Japanese, Korean, Lithuanian (maybe), Norwegian, Portuguese, Russian, Slovenian (I think) and Vietnamese. The breadth of topics is extraordinary! Most books are concerned with R programming or statistical analyses, but you can find the odd volume of Russian poetry or treatise on Capital. Even with the aid of the bubble chart of tags there are hours of browsing here.

The sophistication of the content ranges from student notes to texts that have been published in hardcover. The latter include:

Happy Reading!! And, please let us know if you would like review one of these books.

September 2021: "Top 40" New CRAN Packages

Thu, 28 Oct 2021 00:00:00 +0000

Two hundred twenty new packages stuck to CRAN in September. Here are my “Top 40” picks in fourteen categories: Art, Computational Methods, Data, Econometrics, Finance and Insurance, Genomics, Machine Learning, Medicine, Networks and Graphs, Science, Statistics, Time Series, Utilities, and Visualization.

Art

rfishdraw v0.1.0: Automatically generates fish drawings using the fishdraw JavaScript library. See the vignette.

Computational Mehods

abmR v1.0.4: Supplies tools for running agent-based models (ABM) as discussed in Gochanour et al. (2021) including two movement functions based on the Ornstein-Uhlenbeck model (1930). See the vignette to get started.

SemiEstimate v1.1.3: Implements an improved method of two-step Newton-Raphson iteration based on implicit profiling. See the vignette for examples.

SimEngine v1.0.0: Implements functions for structuring, maintaining, running, and debugging statistical simulations on both local and cluster-based computing environments. Emphasis is placed on documentation and scalability. There is an Introduction, an example Power Calculation, and a vignette Comparing SE Estimators.

Data

asylum v1.0.1: Provides data on asylum and resettlement from the UK Home Office. See README to get started.

hockeyR v0.1.1: Provides functions to scrape hockey play-by-play data from NHL.com and Hockey-Reference.com including standings, player stats, and jersey number history. There is a Getting Started Guide and a vignette on Scraping.

yowie v0.1.0: Provides longitudinal wages data sets and several demographic variables from the National Longitudinal Survey of Youth from 1979 to 2018 including: the wages data from the cohort whose highest grade completed is up to high school; the wages data of the high school dropouts and; the demographic data of the cohort in the survey year 1979. See the vignette for details.

Econometrics

COINr v0.5.5: Implements a development environment for composite indicators and scoreboards including utilities for construction (indicator selection, denomination, imputation, data treatment, normalization, weighting and aggregation) and analysis (multivariate analysis, correlation plotting. Look here for an online book, and see the vignette for an extended overview.

spflow v0.1.0: Provides functions to estimate spatial econometric models of origin-destination flows which may exhibit spatial autocorrelation in both the dependent variable and the explanatory variables. See LeSage and Pace (2008) for information on the model, Dargel (2021) for information on the estimation procedures, and the vignette for examples.

Finance and Insurance

DOSPortfolio v0.1.0: Implements dynamic optimal shrinkage estimators for the weights of the global minimum variance portfolio reconstructed at given reallocation points as derived in Bodnar et al. (2021). See the Introduction.

SPLICE v1.0.0: Extends the individual claim simulator SynthETIC to simulate the evolution of case estimates of incurred losses through the lifetime of an insurance claim. See Taylor & Wang (2021) for background and the vignette to get started.

Genomics

MAnorm2 v1.2.0: Implements a method for normalizing ChIP-seq signals across individual samples or groups of samples and a system of statistical models for calling differential ChIP-seq signals between two or more biological conditions. Refer to Tu et al. (2021) and Chen et al. (2021) for the statistical details. The vignette provides several examples.

RevGadgets v1.0.0: Provides functions to process and visualize the output of complex phylogenetic analyses from the RevBayes phylogenetic graphical modeling software. Look here for a tutorial.

Machine Learning

morphemepiece v1.0.1: Provides functions to tokenize text into morphemes ether by table lookup or a modified wordpiece tokenization algorithm. There is a vignette on testing the fall-through algorithm and another on Generating a Vocabulary.

NIMAA v0.1.0: Implements a pipeline for nominal data mining, which can effectively find special relationships between data. See Jafari et al. (2021) for a description of the method and the vignette for an introduction.

Medicine

gapclosing v1.0.2: Provides functions to estimate the disparities across categories (e.g. Black and white) that persists if a treatment variable (e.g. college) is equalized. See Lundberg (2021) for the methodology and the vignette for an overview with examples.

iDOVE v1.3: Implements a nonparametric maximum likelihood method for assessing potentially time-varying vaccine efficacy against SARS-CoV-2 infection under staggered enrollment and time-varying community transmission, allowing crossover of placebo volunteers to the vaccine arm. See Lin et al. (2021) for background and the vignette for the details.

powerly v1.5.2: Implements the sample size computation method for network models proposed by Constantin et al. (2021) which takes the form of a three-step recursive algorithm to find an optimal sample size given a model specification and a performance measure of interest. See README for an example.

ppseq v0.1.1: Provides functions to design clinical trials using sequential predictive probability monitoring. See the vignette for details.

Networks and Graphs

multigrapher v0.1.0: Implements methods and models for analyzing multigraphs as introduced by Shafie (2015) including methods to study local and global properties and goodness of fit tests. See Shafle (2016). The vignette provides an introduction.

nevada v0.1.0: Implements a statistical framework for network-valued data analysis which leverages the complexity of the space of distributions on graphs. See Lovato et al. (2020) and Lovato et al. (2021) for the statistical background, and the vignette for an example.

robber v0.2.2: Implementation of a variety of methods to compute the robustness of ecological interaction networks with binary interactions as described in Chabert-Liddell et al. (2021). There is a vignette on Topological Analysis.

Science

argoFloats v1.0.3: Supports the analysis of oceanographic data recorded by Argo autonomous drifting profiling floats. See Kelley et al. (2021) for more on the scientific context and applications. There is an Introduction and a vignette on Quality Control and Adjusted Data.

biosurvey v0.1.1: Provides tools that allow users to plan systems of sampling sites with the goals of increasing the efficiency of biodiversity monitoring. See Arita et al. (2011) and Soberón & Cavner (2015) for background. There are vignettes on Preparing Data, Selecting Sample Sites, Using Preselected Points, and Testing Efficacy of Selected Sites.

HostSwitch v0.1.0: Provides functions to aims to investigate the dynamics of the host switch in the population of an organism that interacts with current and potential hosts over generations. The underlying model is based on Araujo et al. (2015). See the vignette for an example.

Statistics

cvam v0.9.2: Extends R’s implementation of categorical variables (factors) to handle coarsened observations and implements log-linear models for coarsened categorical data, including latent-class models. There is a vignette describing Coarsened Factors and another on Fitting Log-linear Models.

JustifyAlpha v0.1.1: Provides functions to justify alpha levels for statistical hypothesis tests by avoiding Lindley’s paradox, or by minimizing or balancing error rates. See Maier & Lakens (2021) for the theory and the vignette for an introduction.

sphunif v1.0.1: Implements functions to test for uniformity on circles and hyperspheres. See García-Portugués et al. (2020) for background and the vignette for examples.

spatialRF v1.1.3: Implements methods to automatically generate and select spatial predictors for spatial regression with Random Forest. See Dray et al. (2006) and RFsp Hengl et al. (2017). Look here for documentation and examples.

vdra v1.0.0: Implements three protocols for performing secure linear, logistic, and Cox regression on vertically partitioned data across several data partners in such a way that data is not shared among data partners. See Slavkovic et. al. (2007) for background. There is an Introduction and vignettes on Using PopMedNet, Communications and Files, and Workflow.

Time Series

GlarmaVarSel v1.0: Implements functions for variable selection in high-dimensional sparse GLARMA models. See Gomtsyan et al. (2020) for the theory and the vignette for an example.

mrf v0.5.1: Implements a method to forecast univariate times series using a feature extraction algorithm based on the Haar wavelet transform. See the vignette.

Utilities

colorblindcheck v1.0.0: Provides functions to compare color palettes with simulations of color vision deficiencies - deuteranopia, protanopia, and tritanopia. It includes a calculation of distances between colors and creates summaries of differences between color palettes and simulations of color vision deficiencies. See the post by G. Aisch for background and the vignette for details.

filearray v0.1.1: Implements file-backed arrays for out-of-memory computation using gigabyte-level multi-threaded read/write via OpenMP. The vignette compares performance with native R.

httr2 v0.1.1: Implements tools for creating and modifying HTTP requests, executing them, and then processing the results. httr2 is a modern re-imagining of httr that uses a pipe-based interface and solves more of the problems that API wrapping packages face. See the vignettes httr2 and Wrapping APIs.

quadtree v0.1.2: Provides region quadtrees for working with spatial data, which allow for variable-sized cells. There are vignettes on code, creation, lcp, and usage.

tabxplor v1.0.2: Implements tools to create, manipulate, and color highlight cross-tables, and export them to Excel or HTML with formats and colors. See the vignette.

Visualization

bittermelon v0.1.3: Provides functions for creating and modifying bitmaps with special emphasis on bitmap fonts and their glyphs including native read/write support for the hex and yaff bitmap font formats. Look here for documentation.

d3po v0.3.2: Implements an Apache licensed alternative to Highcharter which provides functions for interactive visualization for Markdown and Shiny. See the vignette for examples.

ggHoriPlot v1.0.0: implements geom_horizon() and other functions for building horizon plots with ggplot2. There is an Introduction and a vignette with examples.

plotbb v0.0.6: Provides a proof of concept for implementing a grammar for base R plots.See the vignette.

semptools v0.2.9.3: Implements functions for customizing structural equation plots that can be chained using a pipe operator. There is a Quick Start Guide, and vignettes on Nodes, Matrix Layout, CFA, and SEM.

A beginner's guide to Shiny modules

Wed, 20 Oct 2021 00:00:00 +0000

This post by Emily Riederer is the winning entry in our recent Call for Documentation contest. Emily is a Senior Analytics Manager at Capital One where she leads a team building internal analytical tools including R packages, datamarts, and Shiny apps. Outside of work, Emily can be found sharing more code and ideas about analytics on her website, Twitter (@emilyriederer) and GitHub (@emilyriederer).

Shiny modules are often taught as an advanced topic, but they can also be a great way for novice Shiny developers to start building more complex applications. If you already are an R user who likes to think and write functions and understand Shiny basics (i.e. reactivity), then modules for certain types of tasks (discussed at the end of this post) are an excellent way to up your game.

Shiny’s tendency toward monolithic scripts and lack of function-based thinking in introductory materials felt so unlike normal R programming. So, not only is it possible to learn modules early, it may actually be decidedly easier than the alternative depending on your frame of mind.

In this post, I walk through a toy example of building a reporting app from the flights data in the nycflights13 package to demonstrate how modules help scale basic Shiny skills.

Why Modules?

Modules act like functions by wrapping up sets of Shiny UI and server elements. You may wonder why you cannot just accomplish this with the normal R functions. The reason for this is a bit technical and is explained well in Mastering Shiny. However, the technical knowledge needed to understand “why functions don’t work” should not make modules an advanced topic. If you see the value of writing functions, you are more than ready to take advantage of modules in app development.

Motivating Example

For the sake of argument, let’s pretend that we are building an airline dashboard to track travel delays against established thresholds. We have the following requirements:

Let users pick a month of interest to visualize
For each metric of interest, users should be able to:
- See a time-series plot of the average daily value of the metric
- Download a PNG of the plot
- Read a text summary of how often the value breached the threshold
The metrics of interest are:
- Average departure delay
- Average arrival delay
- Proportion of daily flights with an arrival delay exceeding 5 minutes

Below is a preview of the final application. The full app is currently hosted in ShinyApps.io and the code available on GitHub. It would not win any beauty contests, but its simple style allows us to focus on modules in the code.

Two line graphs for departure and arrival times where we see variation compared to the average

Set-Up

library(shiny)
library(nycflights13)
library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(ggplot2)

To understand the following explanation, it helps to familiarize yourself with the data. We filter the flights data down to a single airline and aggregate the results by day.

ua_data <-
  nycflights13::flights %>%
  filter(carrier == "UA") %>%
  mutate(ind_arr_delay = (arr_delay > 5)) %>%
  group_by(year, month, day) %>%
  summarize(
    n = n(),
    across(ends_with("delay"), mean, na.rm = TRUE)
    ) %>%
  ungroup()

## `summarise()` has grouped output by 'year', 'month'. You can override using the `.groups` argument.

head(ua_data)

## # A tibble: 6 x 7
##    year month   day     n dep_delay arr_delay ind_arr_delay
##   <int> <int> <int> <int>     <dbl>     <dbl>         <dbl>
## 1  2013     1     1   165      7.65     6.27          0.476
## 2  2013     1     2   170     12.8      7.04          0.458
## 3  2013     1     3   159      8.66    -2.76          0.357
## 4  2013     1     4   161      6.84    -9.86          0.180
## 5  2013     1     5   117      9.66     0.786         0.274
## 6  2013     1     6   137      9.79     3.53          0.409

Next, we define the plotting function that we will use to visualize a month-long timeseries of data for each metric.

viz_monthly <- function(df, y_var, threshhold = NULL) {
  
  ggplot(df) +
    aes(
      x = .data[["day"]],
      y = .data[[y_var]]
    ) +
    geom_line() +
    geom_hline(yintercept = threshhold, color = "red", linetype = 2) +
    scale_x_continuous(breaks = seq(1, 29, by = 7)) +
    theme_minimal()
}

For example, to visualize the average arrival delay by day for all of March and compare it to a threshold of 10 minutes, we can write:

ua_data %>%
  filter(month == 3) %>%
  viz_monthly("arr_delay", threshhold = 10)

One Module at a Time

Modules don’t just help organize your code; they help you organize your thinking. Given our app requirements, it might feel overwhelming where to start. Filtering the data? Making the plots? Wiring up buttons? Inevitably, when juggling many components, you’re likely to introduce a bug by copy-pasting a line with the wrong id or getting nested parentheses out of whack.

Instead, modules essentially allow your to write many simple Shiny apps and compose them together.

For example, we might decide that first we just want to focus on a very simple app: given a monthly subset of the data, a metric, and a threshold of interest. Let’s write a simple text summary of the flights performance. We now know we just need to define a UI (text_ui) with a single call to textOutput(), a server (text_server) that does a single calculation and calls renderText(). Best of all, we can immediately see whether or not our “app” works by writing a minimalist testing function (text_demo) which renders the text for a small, fake dataset.

Those steps are implemented in a file called mod-text.R:

You can find all of the code for this tutorial in this demo Shiny application repository. We can follow the same pattern to create a module for the plot itself (in the file mod-plot.R) consisting of a UI (plot_ui), a server (plot_server), and a testing function (plot_demo). This module is responsible for plotting one metric and allowing users to download the results.

Composing Modules

We now have a text module and a plot module. However, for each metric of interest, we want to produce both. We could call these two modules one-at-a-time, but we can also compose multiple modules together so that we can produce in single commands everything that we need for a given metric.

With all of the underlying plot and text module logic abstracted, our metric module definition (in the mod-metr.R file) is very clean and simple:

# metric module ----
metric_ui <- function(id) {
  
  fluidRow(
    text_ui(NS(id, "metric")),
    plot_ui(NS(id, "metric"))
  )
  
}
metric_server <- function(id, df, vbl, threshhold) {
  
  moduleServer(id, function(input, output, session) {
    
    text_server("metric", df, vbl, threshhold)
    plot_server("metric", df, vbl, threshhold)
    
  })
  
}
metric_demo <- function() {
  
  df <- data.frame(day = 1:30, arr_delay = 1:30)
  ui <- fluidPage(metric_ui("x"))
  server <- function(input, output, session) {
    metric_server("x", reactive({df}), "arr_delay", 15)
  }
  shinyApp(ui, server)
  
}

Again, we can test that these components went together as we intended by running the metric_demo() function. We see the text from our text module on top of the plot and button from our plot module:

Line graph for flight arrival times where we see how many days exceeded the average

This may be overkill for a simple app, but composing modules is very useful as your application grows in complexity. Everything you bundle into a module gives you a license to forget about how the next layer lower is implemented and frees up your mind to take on the next challenge.

Putting it all together

Finally, we are ready to write our complete application in a file called flights-app.R:

viz_monthly <- function(df, y_var, threshhold = NULL) {
  
  ggplot(df) +
    aes(
      x = .data[["day"]],
      y = .data[[y_var]]
    ) +
    geom_line() +
    geom_hline(yintercept = threshhold, color = "red", linetype = 2) +
    scale_x_continuous(breaks = seq(1, 29, by = 7)) +
    theme_minimal()
}

# text module ----
text_ui <- function(id) {
  
  fluidRow(
    textOutput(NS(id, "text"))
  )
  
}

text_server <- function(id, df, vbl, threshhold) {
  
  moduleServer(id, function(input, output, session) {
    
    n <- reactive({sum(df()[[vbl]] > threshhold)})
    output$text <- renderText({
      paste("In this month", 
            vbl, 
            "exceeded the average daily threshhold of",
            threshhold,
            "a total of", 
            n(), 
            "days")
    })
    
  })
  
}

text_demo <- function() {
  
  df <- data.frame(day = 1:30, arr_delay = 1:30)
  ui <- fluidPage(text_ui("x"))
  server <- function(input, output, session) {
    text_server("x", reactive({df}), "arr_delay", 15)
  }
  shinyApp(ui, server)
}
  
# plot module ----
plot_ui <- function(id) {
  
  fluidRow(
    column(11, plotOutput(NS(id, "plot"))),
    column( 1, downloadButton(NS(id, "dnld"), label = ""))
  )
  
}

plot_server <- function(id, df, vbl, threshhold = NULL) {
  
  moduleServer(id, function(input, output, session) {
    
    plot <- reactive({viz_monthly(df(), vbl, threshhold)})
    output$plot <- renderPlot({plot()})
    output$dnld <- downloadHandler(
      filename = function() {paste0(vbl, '.png')},
      content = function(file) {ggsave(file, plot())}
    )
    
  })
}

plot_demo <- function() {
  
  df <- data.frame(day = 1:30, arr_delay = 1:30)
  ui <- fluidPage(plot_ui("x"))
  server <- function(input, output, session) {
    plot_server("x", reactive({df}), "arr_delay")
  }
  shinyApp(ui, server)
}

# metric module ----
metric_ui <- function(id) {
  
  fluidRow(
    text_ui(NS(id, "metric")),
    plot_ui(NS(id, "metric"))
  )
  
}

metric_server <- function(id, df, vbl, threshhold) {
  
  moduleServer(id, function(input, output, session) {
    
    text_server("metric", df, vbl, threshhold)
    plot_server("metric", df, vbl, threshhold)
    
  })
  
}

metric_demo <- function() {
  
  df <- data.frame(day = 1:30, arr_delay = 1:30)
  ui <- fluidPage(metric_ui("x"))
  server <- function(input, output, session) {
    metric_server("x", reactive({df}), "arr_delay", 15)
  }
  shinyApp(ui, server)
  
}

# load libraries ----
library(nycflights13)
library(shiny)
library(ggplot2)
library(dplyr)
# data prep ----
ua_data <-
  nycflights13::flights %>%
  filter(carrier == "UA") %>%
  mutate(ind_arr_delay = (arr_delay > 5)) %>%
  group_by(year, month, day) %>%
  summarize(
    n = n(),
    across(ends_with("delay"), mean, na.rm = TRUE)
    ) %>%
  ungroup()
# full application ----
ui <- fluidPage(
  
  titlePanel("Flight Delay Report"),
  
  sidebarLayout(
  sidebarPanel = sidebarPanel(
    selectInput("month", "Month", 
                choices = setNames(1:12, month.abb),
                selected = 1
    )
  ),
  mainPanel = mainPanel(
    h2(textOutput("title")),
    h3("Average Departure Delay"),
    metric_ui("dep_delay"),
    h3("Average Arrival Delay"),
    metric_ui("arr_delay"),
    h3("Proportion Flights with >5 Min Arrival Delay"),
    metric_ui("ind_arr_delay")
  )
)
)
server <- function(input, output, session) {
  
  output$title <- renderText({paste(month.abb[as.integer(input$month)], "Report")})
  df_month <- reactive({filter(ua_data, month == input$month)})
  metric_server("dep_delay", df_month, vbl = "dep_delay", threshhold = 10)
  metric_server("arr_delay", df_month, vbl = "arr_delay", threshhold = 10)
  metric_server("ind_arr_delay", df_month, vbl = "ind_arr_delay", threshhold = 0.5)
  
}
shinyApp(ui, server)

You can find all of the code for this tutorial in this demo Shiny application repository.

Notice how few lines of code this file requires to create all of our components and interactions! We’ve eliminated much of the dense code, nesting, and potential duplication we might have encountered if trying to write our application without modules. The top-level code is accessible, semantic, and declarative. It is easy to understand the intent of each line and which pieces of UI and server logic are responsible for which components.

Not all modules are made alike, and in this walk-through I chose simple tasks to demonstrate. Note that our modules consume a reactive variable (data) from the environment, but they do not attempt to alter the global environment or exchange information between them. Modules that simply consume reactives are the easiest way to start out.

Note from the Editor: Emily’s winning entry displays several characteristics of good technical writing. She is laser focused on her audience: R users who have some Shiny experience but who are not familiar with modules. Emily’s prose is direct, simple, informal and engaging. Reading it, I feel as if Emily is beside me showing me how things work, and that she cares about me understanding it. Emily’s code is spacious, and laid out to be read by others. And maybe not so obvious on the first reading, Emily’s writing is disciplined. It is clear that she knows a lot more about things than she has has put into the post, but is working under self imposed constraints in pursuit of clarity.

FDA and the Dynamics of Curves

Thu, 14 Oct 2021 00:00:00 +0000

An elegant application of Functional Data Analysis is to model longitudinal data as a curve and then study the curve’s dynamics. For example, in pharmacokinetics and other medical studies analyzing multiple measurements of drug or protein concentrations in blood samples, it may be interest to determine if the concentrations in subjects undergoing one type of treatment rise quicker than those undergoing an alternative treatment. In this post, I will generate some plausible fake data for measurements taken over time for two groups of subjects, use the techniques of Functional Data Analysis to represent these data as a continuous curve for each subject, and look at some of the dynamic properties of the curves, in particular their velocities and accelerations.

Synthetic Data

The fdaoutlier package contains functions to generate a number of stochastic models with a mechanism to generate reasonable outliers for each type of model. THe curves produced by model 4 look like they can serve plausible synthetic concentration curves. Instead of thinking of normal curves and outliers, I imagine the two related sets of curves to be the results of two different treatments influencing some measured concentration curve. In the example below, the curves associated with treatment 2 are of the form: \[X_i(t) = \mu t(1 - t)^m + e_i(t),\] while those associated with treatment 1 are of the form: \[X_i(t) = \mu(1 - t)t^m + e_i(t)\] where:

$t\in [0,1]$,
$e_i(t)$ is a Gaussian process with zero mean and covariance function of the form: \[\gamma(s,t) = \alpha\exp\{-\beta|t-s|^\nu\},\]
$m$ is a constant
$\mu$ is the mean value
$\alpha$, $\beta$, and $\nu$ are coefficients in the covariance function

library(fdaoutlier)
library(tidyverse)
library(fda)
library(gganimate)
library(gridExtra)

The following code generates two different sets of longitudinal data from Model 4 which is described in the vignette to the fdaoutlier package.

set.seed(95139)
n_curves <- 100
n_obs <- 50
mod4 <- simulation_model4(n = n_curves, p = n_obs, outlier_rate = .5, seed = 50, plot = FALSE)
index <- 1:n_curves
index1 <- mod4$true_outliers
# curves_mat is an n_curves x n_obs matrix
curves_mat <- mod4$data
treat <- rep(2,n_obs)
curves <- data.frame(index, treat, curves_mat)
curves <- curves %>% mutate(treat = if_else((index %in% index1),1,2))

There are 50 curves for each treatment and 50 points for each curve. This is probably the simplest case possible, and it is good enough to show how to explore the dynamics of curves. However, if you work with real longitudinal data you know that things are rarely this simple. But please be assured that FDA can deal with considerably more complexity, including a variable number of measurements for each subject, different measurement times for each subject, and situations where you have far fewer than 50 points for each subject. I have explored some of these situations in previous posts. For example, look here to see how to work with different time points, and here for some ideas for working with sparse data. The first link above also points to basic references that should help you to get started with your data.

Now, I reformat the data into a long form data frame and plot both sets of curves.

time <- 1:n_obs

curves_l <- pivot_longer(curves, cols = !c("index", "treat"), names_to = "Xval") %>%
            mutate(time = rep(time,100), .before = "treat", treat = as.factor(treat)) %>%
            dplyr::select(-Xval)

p <- curves_l %>% ggplot(aes(time,value, color = treat)) +
     geom_line(aes(group = index)) + 
     scale_color_manual(values=c("navy blue", "dark grey")) +
     ggtitle("Model 4 Curves")
p

So far we just have plots based on a relatively small number of data points. Nevertheless, our eyes extrapolate, and we imagine seeing two sets of continuous curves with one set clearly rising to its maximum value faster than the other set. But we really don’t have continuous curves yet, we just have points.

The next step is to invoke the mathematics of FDA to embed the points in an infinite dimensional vector space where the data for each subject is modeled by a continuous function (curve). Exactly how this happens is a bit involved, but the general idea is that we create a basis and then use the data and some techniques from the linear algebra of Hilbert spaces to estimate the time dependent coefficients that enable modeling each subject’s data points as a linear combination of the basis functions. So, from here on out we are not going to be working with the raw data anymore. We will be working with vector space models of the data. The upside is that we can now now have real curves (continuous function), and can calculate the values of the functions and other properties such as the first and second derivatives at any time point. If you are interested in the math, please have a look at the references listed at the bottom of my post on Functional PCA.

Modeling the data as functions with a B-Spline Basis

Here we use the function fda::create.bspline.basis() to create a B-Spline basis covering the interval of our observations. The plot of the curves above indicates that specifying a knot at every multiple of 5 ought to be adequate for representing our data. Note that using cubic splines n_order = 4 ensures that the splines will have continuous first and second derivatives at the knots.

knots    = c(seq(0,n_obs,5)) #Location of knots
n_knots   = length(knots) #Number of knots
n_order   = 4 # order of basis functions: for cubic b-splines: order = 3 + 1
n_basis   = length(knots) + n_order - 2;
spline_basis = create.bspline.basis(rangeval = c(0,n_obs), nbasis = n_basis, norder = n_order)
#plot(spline_basis)

The following code uses the function fda::Data2fda() and the spline-basis to convert the Model 4 data points into functions and produce one fda object for each of the two groups of treatments.

# df1 is an (n_curves/2) x (n_obs) matrix 
df1 <- curves_mat[index1,] # data for treatment 1
index2 <- index[!(index %in% index1)]
df2 <- curves_mat[index2,]  # data for treatment 2
# Use the b-spline basis to create represent the curves as vectors in the function space
df1_obj <- Data2fd(argvals = 1:n_obs, y = t(df1), basisobj = spline_basis, lambda = 0.5)
df2_obj <- Data2fd(argvals = 1:n_obs, y = t(df2), basisobj = spline_basis, lambda = 0.5)

Computing Derivatives

Here we evaluate each function along with its first and second derivatives at a finer time scale than we used to originally display our data.

tfine <- seq(0,50,by=.5)
# Each matrix is 101 x 50 rows are different times, columns are curves
pos1 <- as.vector(eval.fd(tfine, df1_obj));     pos2 <- as.vector(eval.fd(tfine, df2_obj))
vel1 <- as.vector(eval.fd(tfine, df1_obj,1));   vel2 <- as.vector(eval.fd(tfine, df2_obj,1)) 
acc1 <- as.vector(eval.fd(tfine, df1_obj,2));   acc2 <- as.vector(eval.fd(tfine, df2_obj,2))

Note that the velocities and accelerations computed above were returned as matrices. We convert them to vectors and put them into a data frame for plotting.

time <- rep(tfine,50)
id1 <- rep(1:50,each=101)
id2 <- rep(51:100,each=101)

derv1 <- data.frame(time, id1, pos1, vel1, acc1)
derv2 <- data.frame(time, id2, pos2, vel2, acc2)

pv1 <- derv1 %>% ggplot(aes(time,vel1,col=id1)) + geom_line(aes(group = id1)) + ggtitle("Velocity Treatment 1")
pv2 <- derv2 %>% ggplot(aes(time,vel2,col=id2)) + geom_line(aes(group = id2)) + ggtitle("Velocity Treatment 2")
pa1 <- derv1 %>% ggplot(aes(time,acc1,col=id1)) + geom_line(aes(group = id1)) +ggtitle("Acceleration Treatment 1")
pa2 <- derv2 %>% ggplot(aes(time,acc2,col=id2)) + geom_line(aes(group = id2)) + ggtitle("Acceleration Treatment 2")

grid.arrange(pv1, pa1, pv2, pa2, nrow = 2,ncol = 2, padding = unit(1, "line"))

The velocities for treatment 1 head upwards before they decrease while the velocities for treatment 2 go straight down, and the curve up again. The accelerations for treatment 1 slope downward slightly while those of treatment two slope upward. So the plots indicate that the two sets of curves look like they behave differently.

Testing for Differences

We can test for differences using the function fda::tperm.fd() which implements a resampling method to do pointwise t-tests. The functional data representing the curves for the two samples are combined in a single array and the labels for the curves are randomly shuffled. Recalculating the maximum value of the t-statistic for each point enables computing a null distribution. Then, at each time point, the observed data are compared with the 1 - $\alpha$ quantile of the null distribution.

The plot below shows the result of performing the t-test to compare the first derivatives of the two treatments. I use the function fda.deriv.fd() to calculate the first derivative for each treatment on the fly, just to show another way of doing things in the fda package. You can easily modify the code to compare accelerations.

dfdt1 <- deriv.fd(df1_obj,1)
dfdt2 <- deriv.fd(df2_obj,1)
tres <- tperm.fd(dfdt1,dfdt2,plotres=FALSE)

max_q <- tres$qval
tres_dat <-tibble(time = tres$argvals, t_values = tres$Tvals,
                  q_vals = tres$qvals.pts)

p <- tres_dat %>% ggplot(aes(time,t_values, colour = "t_value")) + geom_line() +
    geom_line(aes(time, q_vals, colour = "q_vals")) +
    geom_line(aes(time, max_q,colour = "max_q"), linetype= "dashed") + 
    labs(x = "time", y = "") +
    ggtitle("Statistics for Pointwise t-test")
p

The blue curve shows the t-statistic for the observed values. The green curve represents the 95% quantiles, and the dashed red line is the 95% quantile of the maximum of null distribution t-statistics. The t-test confirms that the derivatives are indeed different except in the regions of overlap around time = 10 and time = 40.

Examine Phase Plots

Phase plots are useful for evaluating how the curves develop in the abstract phase space created by looking at position versus velocity, or velocity versus acceleration. Here we pick three curves from each treatment and plot velocity versus acceleration.

phase1 <- derv1 %>% filter(id1 %in% 15:17)
phase2 <- derv2 %>% filter(id2 %in% 51:53)
pph1 <- phase1 %>% ggplot(aes(vel1,acc1,col=id1)) + geom_point() + ggtitle("Treat 1 v vs. acc")
pph2 <- phase2 %>% ggplot(aes(vel2,acc2,col=id2)) + geom_point() + ggtitle("Treat 2 v vs. acc")

grid.arrange(pph1, pph2, ncol = 2, padding = unit(1, "line"))

Based on this small sample, it certainly appears that the curves associated with the two different treatments inhabit different regions of phase space.

Finally, it is easy and helpful to animate a phase diagram to show how a function develops in phase space over time. The animation below shows the first curve for treatment 1.

pos <- eval.fd(tfine, df2_obj[1])
vel <- eval.fd(tfine, df2_obj[1],1)
acc <- eval.fd(tfine, df2_obj[1],2)
phase_dat <- tibble(tfine, vel, acc)

p <- ggplot(phase_dat, aes(vel,acc)) +  geom_point() + ggtitle("Trajectory in Phase Space")

anim <- p + transition_time(tfine) + shadow_mark()
anim_save("anim.gif", anim)
#anim

August 2021: "Top 40" New CRAN Packages

Mon, 27 Sep 2021 00:00:00 +0000

One hundred sixty new packages covering a wide array of topics made it to CRAN in August. I thought I would emphasize the breadth of topics by expanding the number of categories organizing my “Top 40” selections beyond core categories that appear month after month. Here are my picks in fourteen categories: Archaeology, Computational Methods, Data, Education, Finance, Forestry, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization. Based on informal impressions formed over the last several months, I believe a new category combining applications in forestry, animal populations, climate change could become a regular core category.

Archaeology

DIGSS v1.0.2: Provides a simulation tool to estimate the rate of success that surveys including user-specific characteristics have in identifying archaeological sites given specific parameters of survey area, survey methods, and site properties. See Kintigh (1988) for background and the vignette for examples.

Computational Methods

simlandr v0.1.1: Provides a set of tools for constructing potential landscapes for dynamical systems using Monte-Carlo simulation which is especially suitable for formal psychological models. There are vignettes on Dynamic Models and Simulations, Constructing Potential Landscapes, and Calculating the Lowest Elivation Path.

Data

metaboData v0.6.2: Provides access to remotely stored data sets from a variety of biological sample matrices analyzed using mass spectrometry metabolomic analytical techniques. See the vignette.

metadat v1.0-0: Contains a collection of data sets useful for teaching meta analysis. See README for more information.

nflreadr v1.1.0: Provides functions for downloading data from the GitHub repository for the nflverse project. There is a brief Introduction and several short vignettes that serve as the data dictionary for the various files Draft Picks, Rankings, etc.

OCSdata v1.0.2: Provides functions to access and download data from the Open Case Studies repositories on GitHub. See the vignette to get started.

rATTAINS v0.1.2: Implements an interface to United States Environmental Protection Agency (EPA) ATTAINS database used to track information provided by states about water quality assessments conducted under federal Clean Water Act requirements. There is a vignette.

taylor v0.2.1: Provides access to a curated data set of Taylor Swift songs, including lyrics and audio characteristics. Data comes Genius and the Spotify API. See README for examples,

Education

karel v0.1.0: Provides an R implementation of Karel the robot, a programming language for teaching introductory concepts about general programming in an interactive and fun way, by writing programs to make Karel achieve tasks in the world she lives in. There are several vignettes including one on Control Structures and another on Algorithmic Decomposition.

roger v0.99-0: Implements tools for grading the coding style and documentation of R scripts. This is the R component of Roger the Omni Grader, an automated grading system for computer programming projects based on Unix shell scripts. Look here for more information.

Finance

dispositionEffect v1.0.0: Implements four different methodologies to evaluate the presence of the disposition effect and other irrational investor behaviors based on investor transactions and financial market data. There is a Getting Started Guide, and vignettes on Analysis, Disposition Effects in Parallel, and Time Series Disposition Effects.

HDShOP v0.1.1: Provides functions to construct shrinkage estimators of high-dimensional mean-variance portfolios and performs high-dimensional tests on optimality of a given portfolio. See Bodnar et al. (2018), Bodnar et al. (2019), and Bodnar et al. (2020) for background.

tcsinvest v0.1.1: Implements an interface to the Tinkoff Investments API which enables analysts and traders can interact with account and market data from within R. Clients for both REST and Streaming protocols have been implemented. There is a vignette.

Forestry

APAtree v1.0.1: Provides functions to map the area potentially available (APA) using the approach from Gspaltl et al. (2012) and also aggregation functions to calculate stand characteristics based on APA-maps and the neighborhood diversity index as described in Glatthorn (2021). See the vignette for examples.

efdm v0.1.0: Implements the European Forestry Dynamics Model (EFDM), a large-scale forest model that simulates the development of a forest and estimates volume of wood harvested for any given forested area. See Packalen et al. (2015) for background and the vignette for examples.

Genomics

molnet v0.1.0: Implements a network analysis pipeline that enables integrative analysis of multi-omics data including metabolomics. It allows for comparative conclusions between two different conditions, such as tumor subgroups, healthy vs. disease, or generally control vs. perturbed. The case study presented in the vignette uses data published by Krug (2020).

simtrait v1.0.21P Provides functions to simulate complex traits given a SNP genotype matrix and model parameters with an emphasis on avoiding common biases due to the use of estimated allele frequencies. Traits can follow three models: random coefficients, fixed effect sizes, and multivariate normal. GWAS method benchmarking functions as described in Yao and Ochoa (2019) are also provided. See the vignette.

statgenIBD v1.0.1: Provides functions to calculate biparental, three and four-way crosses Identity by Descent (IBD) probabilities using Hidden Markov Models and inheritance vectors following Lander & Green (1987) and Huang (2011). See the vignette for examples.

Machine Learning

text2map v0.1.0: Provides functions for computational text analysis for the social sciences including functions for working with word embeddings, text networks, and document-term matrices. For background on the methods used see Stoltz and Taylor (2019), Taylor and Stoltz (2020), Taylor and Stoltz (2020), and Stoltz and Taylor (2021). There is a Quick Start Guide and a vignette on Concept Class Analysis.

NPRED v1.0.5: Uses partial informational correlation (PIC) to identify the meaningful predictors from a large set of potential predictors. Details can be found in Sharma & Mehrotra, (2014), Sharma et al.(2016), and Mehrotra & Sharma (2006). See the vignette for examples.

stabiliser v0.1.0: Implements an approach to variable selection through stability selection and the use of an objective threshold based on permuted data. See Lima et al (2021) and Meinshausen & Buhlmann (2010) for details and the vignette for an example.

Medicine

dreamer v3.0.0: Fits longitudinal dose-response models utilizing a Bayesian model averaging approach as outlined in Gould (2019) for both continuous and binary responses. See the vignette.

smartDesign v0.72: Implements the SMART trial design, as described by He et al. (2021) which includes multiple stages of randomization where participants are randomized to an initial treatment in the first stage and then subsequently re-randomized between treatments in the following stage. There is a Dynamic Treatment Tutorial and a Sequential Design Tutorial.

Science

bootf2 v0.4.1: Provides functions to compare dissolution profiles with confidence intervals of the similarity factor f2 and also functions to simulate dissolution profiles. There are multiple vignettes including and Introduction a Simulation Example.

track2KBA v1.0.1: Provides functions to prepare and analyze animal tracking data in order to identify areas of potential interest for population level conservation. See Lascelles et al. (2016) for background on the methodology employed and the vignette for examples and workflow.

Statistics

chyper v0.3.1: Provides functions to work with the conditional hypergeometric distribution. See the vignette.

sprtt v0.1.0: Provides functions to perform sequential t-tests including those of Wald (1947), Rushton (1950), Rushton (1952), and Hajnal (1961). There is an Introduction to the package, a Use Case, and a vignette on the Sequential t-test.

SurvMetrics v0.3.5: Implements popular evaluation metrics commonly used in survival prediction including Concordance Index, Brier Score, Integrated Brier Score, Integrated Square Error, Integrated Absolute Error and Mean Absolute Error. For detailed information, see Ishwaran et al. (2008) and Moradian et al. (2017). The vignette offers examples.

Time Series

DCSmooth v1.0.2: Implements nonparametric smoothing techniques for data on a lattice or functional time series which allow for modeling a dependency structure of the error terms of the nonparametric regression model. See Beran & Feng (2002), Mueller & Wang (1994), Feng & Schaefer (2021), and Schaefer & Feng (2021) for the background and the vignette for examples.

STFTS v0.1.0: Implements statistical hypothesis tests of functional time series including a functional stationarity test, a functional trend stationarity test and a functional unit root test.

WASP v1.4.1: Implements wavelet-based variance transformation methods for system modeling and prediction. For details see Jiang et al. (2020), Jiang et al. (2020), and Jiag et al. (2021) There is a vignette with examples.

Utilities

ExpImage v0.2.0: Provides an image editing tool for researchers which includes functions for segmentation and for obtaining biometric measurements. There are several vignettes including: Contagem de bovinos, Contagem de objetos, and Como editar imagens.

meltr v1.0.0: Provides functions to read non-rectangular data, such as ragged forms of csv (comma-separated values), tsv (tab-separated values), and fwf (fixed-width format) files. See README to get started.

plumbertableau v0.1.0: Implements tools for building plumber APIs that can be used in Tableau workbooks. There is a package Introduction and vignettes on Writing Extensions, Using Extensions in Tableau, and Publishing Extensions to RStudio Connect.

string2path v0.0.2: Provides functions to extract glyph information from a font file, translate the outline curves to flattened paths or tessellated polygons, and return the results as a data.frame. See README for an example.

trackdown v1.0.0: Uses Googel Drive to implement tools for collaborative writing and editing of R Markdown and Sweave documents. There are some Tech Notes and vignettes on Features and Workflow.

Visualization

aRtsy v0.1.1: Provides algorithms for creating artwork in the ggplot2 language that incorporate some form of randomness. See README for examples and package use.

ggcleveland v0.1.0: Provides functions to produce ggplot2 versions of the visualization tools described in William Cleveland’s book Visualizing Data. The vignette contains several examples.

ggtikz v0.0.1: Provides tools to annotate ggplot2 plots with TikZ code using absolute data or relative coordinates. See the vignette.

tidycharts v0.1.2: Provides functions to generate charts compliant with the International Business Communication Standards (IBCS) including unified bar widths, colors, chart sizes, etc. There is a Getting Started guide and vignettes on EDA, Customization, and Joining Charts.

A Guide to Binge Watching R / Medicine 2021

Thu, 09 Sep 2021 00:00:00 +0000

R / Medicine is a big deal. This year, the conference grew by 13% with 665 people from over 60 countries signing up for the virtual event which was held last month. 34% percent of the registrants were from outside of the United States and 17% identified as physicians.

The conference is now an established international event where experts report on the advanced use of the R language, Machine Learning, and statistical analysis, and discuss the successes and challenges associated with bringing these technologies to day-to-day medical practice.

Almost all of the talks, including keynotes, regular talks, lightning talks, pre-conference workshops and poster sessions are available online. Find the links on the R / Medicine site or look through the playlist on the R Consortium Youtube Channel. Note that the posters can be viewed by going to the conference spatial.chat site. (If you and a friend visit at the same time you should be able to “walk around” the posters and chat about what you see.)

To kick off an evening of binge watching the conference I would begin with the keynotes.

The Keynotes

Dr. Karandeep Singh sets the hook for his talk, Bringing Machine Learning Models to the Bedside at Scale, two minutes into the video when he asks:

Who are the twenty sickest patients in the hospital right now who are not in the ICU?

This straightforward question immediately gets to the promise and the problems of introducing large scale machine learning algorithms into the hospital, and indicates how medical practice interacts with big money questions about allocating resources. Both physicians and administrators would like to identify high risk patients and treat them proactively while being able to confidently spend less on unnecessary test for low risk patients. About (5:10) into the talk, Karandeep begins discussing the challenges associated with introducing machine learning models.

In the remainder of the talk he describes the technical infrastructure and then the governance or “social infrastructure” needed for success.

If you enjoy a good detective story, and take pride in your ability to interpret a well-done statistical plot you are certainly going to want to watch Ziad Obermeyer’s keynote Dissecting Algorithmic Bias. About two minutes into the video Professor Obermeyer sets the stage with the warning:

The single greatest threat to all of the gains that we can make in using algorithms in medicine is letting them go wrong in increasingly well known ways.

and the observation that due to the focus of the US health care management on “high risk care management” an estimated 150 to 200 million Americans are sorted by algorithms every year. He goes on to work through a case study that illustrates how an algorithm built with good intentions had the effect of scaling up racial bias.

A second case study features an algorithm that “fights against” racial bias. Along the way, Ziad weaves two common themes into his presentation:

So many of the ways that algorithms can go wrong come from training algorithms with the wrong target variables, often “convenient and tempting proxies”.
The necessity of follow-up work to fix underlying problems.

In the remainder of this post, I have organized the talks into six categories that you may find helpful for setting your viewing program: Clinical Practice, Clinical Trials, Medical Data, R in Production, R Tools, and Short Courses. The majority of the talks have a machine learning angle. There is quite a bit of Shiny and several R packages, not all of them on CRAN, are featured. I have provided links when I could find them. I don’t want to spoil anyone’s fun in searching through the videos for “Easter Eggs”, but the Reproducible Research with R short course contains the first preview on the Quarto Publishing system in a talk from anyone at RStudio. (Note that the video needs some editing. Start watching at 9 minutes.)

Clinical Practice

Building an Interpretable ML Model API for Interpretation of CNVs in Patients with Rare Diseases - Francisco Requena
Subgroup Identification and Precision Medicine with the personalized R Package - Jared Huling
R and Shiny Dashboards to Facilitate Quality Improvement in Anesthesiology and Periopeartive Care - Robert Lobato
tidytof: Predicting Patient Outcomes from Single-cell Data using Tidy Data Principles - Timothy Keyes
Assessing ML Model Performance in DIverse Populations and Across Time - Victor Castro, Roy Perlis

Clinical Trials

Designing Early Phase Clinical Trials with ppseq - Emily Zabor
Collaborative, Reproducible Exploration of Clinical Trial Data - Michael Kane
Graphical Displays in R for Clinical Trials - Steven Schwager
ctrialsgov: Access, Visualization, and Discovery of the ClinicalTrials.gov Database - Taylor Arnold

Medical Data

Scaling Up and Deploying Shiny and Text Mining for National Health Decisions - Andreas Soteriade, Chris Beeley
Mapping African Health Data with afrimapr Package, Training & Community - Andy South
You R What You Measure: Digital Biomarkers for Insights in Personalized Health - Irene van den Broek
Shiny and REDCap for a Global Research Consortium - Judith Lewis, Stephany Duda
Diving into Registry Data: Using R for Large Norwegian Health Registries - Julia Romanowska
ReviewR: A Shiny App for Reviewing Clinical Records - Laura Wiley, David Mayer
DOPE: An R package for Processing and Classifying Drug Names - Layla Bouzoubaa
medicaldata for Teaching #Rstats - Peter Higgins
Stem Cell Transplant Outcomes Reporting using R/Shiny - Richard Hanna, Stephan Kadauke

R in Production

Second Server to the Right and Straight On ‘til Production: Deploying a GxP Shiny Application - Marcus Adams
Target Markdown and stantargets for Bayesian model validation pipelines - Will Landau
GENETEX: A Genomics Report Text Mining R Package to Capture Real-world Clinico-genomic Data - David Miller, Sophia Shalhout

R Tools

Generalized Additive Models for Longitudinal Biomedical Data - Ariel Mundo
Multistate Data Using the survival Package - Beth Atkinson
Bayesian Random-Effects Meta-analysis using bayesmeta - Christian Rover
An arsenal of R Functions for Statistical Summaries - Ethan Heinzen, Beth Atkinson, Jason Sinnwell
R Markdown and officedown to Automate Clinical Trial Reporting - Damian Rodziewicz
Creating and Styling PPTX Slides with rmarkdown - Emil Hvitfeldt
runway: an R Package to Visualize Prediction Model Performance - Jie Cao, Karandeep Singh
clinspacy: An R package for Clinical Natural Language Processing - Jie Cao, Karandeep Singh
Data Visualization for Machine Learning Practitioners - Julie Silge
Animated Data Visualizations with gganimate for Science Communication during the Pandemic - Kristen Panthagani
Incorporating Risk-of-Bias Assessments into Evidence Syntheses with robvis - Luke McGuinness, Randall Boyes, Alex Fowler
‘gpmodels’: A Grammar of Prediction Models - Sean Meyer, Karandeep Singh
CONSORT Diagrams in R with ggconsort - Travis Gerke

Short Courses

Secure Medical Data Collection: Best Practices with Excel, and Leveling Up to REDCap and CollaboratoR - Peter Higgins, Will Beasley, Kenneth MacLean, Amanda Miller
Introduction to R for Medical Data - Ted Laderas, Daniel Chen, Mara Alexeev
An Introductory R Guide for Targeted Maximum Likelihood Estimation in Medical Research - Ehsan Karim, Hanna Frank
Mapping Spatial Health Data - Marynia Kolak, Susan Paykin
From SAS to R - Joe Krsszun
Reproducible Research with R - Alison Hill, Stephan Kaduke, Paul Villanueva

July 2021: "Top 40" New CRAN Packages

Thu, 26 Aug 2021 00:00:00 +0000

One hundred eighty-three new packages stuck to CRAN in July. Here are my “Top 40” picks in eleven categories: Data, Ecology, Finance, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization. Although I don’t have any formal specification for these categories, I do my best to main my subjective sense of consistency from month to month. Nevertheless, watching the monthly ebb and flow of the number of packages that fit into the various categories is interesting. This month, developers seemed to be focused on utilities. I classified forty-five packages as utilities this month.

Data

beans v0.1.0: Contains data on 13,611 beans from Koklu and Ozkan (2020). The beans have been quantified using 16 morphologic image features and labeled with one of 6 values.

geckor v0.1.1: Provides functions to collect current and historical cryptocurrency market data using the public CoinGecko API. There is a brief overview and vignettes on Current and Historical market data.

ISLR2 v1.0: Contains the data sets used in the book An Introduction to Statistical Learning with Applications in R, Second Edition.

sageR v0.3.0: Provides the data sets used in the book Statistiques pour l’économie et la gestion, Théorie et applications en entreprise. Look here for descriptions of the data with code and plots.

Ecology

DA v1.2.0: Provides functions for Discriminant Analysis (DA) for evolutionary inference, especially population genetic structure and community structure inference as described in Qin et al. (2020). See the Introduction.

Ostats v0.1.0: Provides functions to calculate O-statistics, or overlap statistics, which measure the degree of community-level trait overlap. See Read et al. (2018) for background. There is an Introduction and a vignette on Multivariate Ostats.

Finance

ffp v0.1.0: Implements numerical entropy-pooling for scenario analysis as described in Meucci (2010) See the vignette.

Genomics

utr.annotation v1.0.4: Implements a fast tool for annotating potential deleterious variants in the untranslated regions for both human and mouse species. See Liu & Dougherty (2021) for background and the vignette for and introduction and examples.

OmicNavigator v1.4.3: Provides a for interactive exploration of the results from ‘omics’ experiments to facilitate novel discoveries from high-throughput biology. There is a User’s Guide and a vignette on the package’s API.

Machine Learning

BOSO v1.0.3: Implements the BiLevel Optimization Selector Operator feature selection algorithm for linear regression as described in Valcarcel et al. (2020). See the vignette for examples.

FeatureTerminatoR v1.0.0: Implements a feature selection engine that removes features with minimal predictive power. See Boughaci (2018) for background and the vignette for and example.

hhcartr v1.0.0: Implements the HHCART-G algorithm described in Wickramarachchi et al. (2019). See the vignette.

recometrics v0.1.3: Implements evaluation metrics for implicit-feedback recommender systems based on low-rank matrix factorization models, given the fitted model matrices and data. See the vignette for examples.

tidylda v0.0.1: Implements an algorithm for Latent Dirichlet Allocation (LDA) as described in Blei et at. (2003) using tidyverse principles. See README for an overview.

Medicine

dcurves v0.2.0: Implements a decision curve analysis method for evaluating and comparing prediction models that incorporates clinical consequences and requires only the data set on which the models are tested. See Vickers (2006), Vickers (2008), and Pfeiffer (2020) for background, and the vignette for examples.

KHQ v0.2.0: Provides methods to calculate scores for each dimension of the The King’s Health Questionnaire (KHQ) ; converts KHQ item scores to KHQ5D scores; and also calculates the utility index of the KHQ5D. See the vignette.

Science

AvInertia v0.0.1: Provides functions to compute the center of gravity and moment of inertia tensor of any flying bird. See the vignette for some insight into bird design.

bayesnec v1.0.1: Provides functions to fit dose concentration response curves to toxicity data, and derive No-Effect-Concentration (NEC), No-Significant-Effect-Concentration (NSEC), and Effect-Concentration (of specified percentage ‘x’, ECx) thresholds from non-linear models fitted using Bayesian MCMC fitting methods via brms and Stan. There are vignettes on Single model usage, Multi model usage, Model details, Priors, and Comparing posterior predictions.

Statistics

itdr v1.0: Provides functions to estimate the sufficient dimension reduction subspaces, i.e., central mean subspace or central subspace in regression, using Fourier transformation proposed by Zhu & Zeng (2006), the convolution transformation proposed by Zeng & Zhu (2010), and an iterative Hessian transformation methods proposed by Cook & Li (2002). See the vignette for an example.

mlfit v0.5.2: Extends the [Iterative Proportional Fitting] (IPF) algorithm which operates on count data to nested structures with constraints. See Müller & Axhausen (2011) for background and look here for and example.

optimall v0.1.0: Provides functions for the survey sampling design process with specific tools for multi-wave and multi-phase designs. Users can perform optimum allocation according to Neyman (1934) or Wright (2012). There is a User’s Guide, there are vignettes Multiwave Object and Splitting Strata with Optimall Shiny.

posterior v1.0.1: Provides tools for both users and package developers for fitting and working with Bayesian models; including tools to: efficiently convert between different formats of draws from distributions, provide consistent methods for common operations, provide summaries in convenient formats, and implement state of the art posterior inference diagnostics. See Vehtari et al. (2021) for background. There is an Introduction and a vignette on the Random Variable Datatype.

PropensitySub v0.2.0: Provides functions to estimate treatment effects in strata via inverse probability weighting or propensity score matching when subjects have missing strata labels. See the vignette for examples.

ReplicationSuccess v1.0.0: Provides utilities for the design and analysis of replication studies featuring both traditional methods based on statistical significance and more recent methods such as the skeptical p-value Held L. (2020), the harmonic mean chi-squared test Held, L. (2020), and intrinsic credibility Held, L. (2019). See the vignette for examples.

Time Series

kinematics V1.0.0: Provides functions to analyze time series representing two-dimensional movements. It accepts a data frame with a time (t), horizontal (x) and vertical (y) coordinate as columns, and returns several dynamical properties such as speed, acceleration or curvature. See the vignette for examples.

profoc v0.8.3: Provides methods to combine probabilistic forecasts using CRPS learning algorithms proposed in Berrisch & Ziel (2021) including multiple online learning algorithms such as Bernstein online aggregation as described in Wintenberger (2014).

tscopula v0.2.1: Provides functions for the analysis of time series using copula models. See McNeil (2021) and Bladt & McNeil (2020) for background. There are vignettes on Bitcoin Analysis, Models with Margins, Basic Time Series Copula Processes, and Copula Processes with V-Transforms.

Utility

codemeta v0.1.0: Provides core utilities to generate metadata with a minimum of dependencies as specified by the Codemata Project. See README for examples.

ctf v0.1.0: Provides functions to read and write data in Column Text Format (CTF), a new tabular data format that is a simple column store. See the vignette.

fedmatch v2.0.2: Provides tools for matching two un-linked data sets using exact matches, fuzzy matches, or multi-variable matches. There is an Introduction and vignettes on Fuzzy Matching, Multivar Matching, Tier Matching, and Using clean_strings.

fusen v0.2.4: Implements a method to use R Markdown to build an R package. Users start by including documentation, functions, examples and tests in the same file. Then inflating the R Markdown template copies the relevant chunks and sections in the appropriate files required for package development. There is an Introduction and a vignette on maintaining packages.

katex v1.10: Provides functions to convert LaTeX math expressions to HTML and MATHML for use in markdown documents or package manual pages in a way which eliminates the need for embedding the MathJax library into your web pages. See the vignette.

pacs v0.3.3: Provides utilities for CRAN package maintainers and R packages developers, including tools for validating packages and exploring the complexity of a specific package. See README for a list of features and examples.

RcppFarmHash v0.0.2 Implements an interface to the The Google FarmHash family of hash functions is used by the Google BigQuery data warehouse. Look here to get started.

shinyauthr v10.0: Provides in-app user authentication for Shiny applications, allowing developers to secure publicly hosted apps and build dynamic user interfaces from user information. See README to get started.

tantram.pipe v1.0.0: Allows users to build tables with customizable rows by specifying the type of data to use for each row, as well as how to handle missing data and the types of comparison tests to run on the table columns. See the vignette.

Visualization

chameleon v0.2-0: Provides functions to assign distinct colors to arbitrary multi-dimensional data, considering its structure. See the vignette for examples.

deeptime v0.1.0: Extends plotting packages such as ggplot2 and lattice to facilitate the plotting of data over long time intervals, including, but not limited to, geological, evolutionary, and ecological data. Look here for examples.

figpatch v0.1.0.1: Provides functions to arrange external figures with patchwork alongside ggplot2 plots. See README for examples.

spiralize v1.0.2: Provides functions to visualize data along an Archimedean spiral which has the advantages of being able to visualize data with very a long axis with high resolution and reveal periodic patterns in time series. Look here for links to the five vignettes.

R / Medicine 2021

Thu, 12 Aug 2021 00:00:00 +0000

R/Medicine 2021, the premier conference for the use of R in clinical applications is less than two weeks away! This conference reflects the increasing importance of data science, computational statistics and machine learning to clinical applications, and emphasizes the effectiveness of the R language as a vehicle for making data driven medicine accessible to clinicians with diverse backgrounds. The two conference keynote talks Bringing Machine Learning Models to the Bedside by Karandeep Singh and Dissecting Algorithmic Bias by Ziad Obermeyer directly address important technical and ethical issues confronting modern data driven medicine.

R/Medicine will offer six short courses spread out over the two days of August 24th and August 25th. These courses are included in the registration price. The Tuesday short courses will be:

Secure Medical Data Collection: Best Practices with Excel, and Leveling Up to REDCap and collaboratoR taught by Peter Higgins, Will Beasley, Kenneth McLean
Intro to R for Medical Data taught by Ted Laderas, Daniel Chen, Mara Alexeev
An Introductory R Guide for Targeted Maximum Likelihood Estimation in Medical Research taught by Ehsan Karim

The Wednesday short courses will be:

Mapping Spatial Health Data taught by Marynia Kolak
Reproducible Research with R taught by Alison Hill and Stephan Kadauke
From SAS to R taught by Joe Korszun

The conference will proper will run from 11:00 to 19:30 EDT on Thursday, August 26th, and from 10:50 to 18:00 EDT on Friday, August 27th. The full schedule is here. Priced at only $50 full fare, $25 for academics and $10 for students and trainees, this is an affordable, important conference that you will not want to miss.

You can register here.

To get an idea of the international scope of the conference, and a feel for what the virtual conference experience might be like, have a look at the R Journal article written by the organizing team about last year’s conference: R Medicine 2020: The Power of Going Virtual.

And finally, to get an idea of the various ways that R is contributing to getting everyday work done in clinical practice: the following are the packages from the Medicine category that made it to my lists of “Top 40” new CRAN packages in posts over the past twelve months.

“Top 40” Picks for new CRAN Packages for Medicine

afdx v1.1.1: Provides functions to estimate diagnosis performance (Sensitivity, Specificity, Positive predictive value, Negative predicted value) of a diagnostic test when there is no golden standard by estimating the attributable fraction using either a logitexponential model or a latent class model.

aldvmm v0.8.4: Fits health state utility adjusted limited dependent variable mixture models, i.e. finite mixtures of normal distributions with an accumulation of density mass at the limits, and a gap between 100% quality of life and the next smaller utility value. See Alava and Wailoo (2015) for background and the vignette for an example.

babsim.hospital v11.5.14: Implements a discrete-event simulation model for a hospital resource planning. Motivated by the challenges faced by health care institutions in the current COVID-19 pandemic, it can be used by health departments to forecast demand for intensive care beds, ventilators, and staff resources. See Ucar, Smeets & Azcorra (2019), Lawton & McCooe (2019) and the website for background, and the vignette to get started.

beats v0.1.1: Provides functions to import data from UFI devices and process electrocardiogram (ECG) data. It also includes a Shiny app for finding and exporting heart beats. See README to get started.

bhmbasket v0.9.1: Provides functions to evaluate basket trial designs with binary endpoints using Bayesian hierarchical models and Bayesian decision rules. See Berry et al. (2013), Neuenschwander et al. (2016) and Fisch et al. (2015) for background and the vignette for an example.

bp v1.0.1: Provides functions to aid in the analysis of blood pressure data of all forms by providing both descriptive and visualization tools for researchers. There is a vignette.

card v0.1.0: Provides tools to help assess the autonomic regulation of cardiovascular physiology with respect to electrocardiography, circadian rhythms, and the clinical risk of autonomic dysfunction on cardiovascular health through the perspective of epidemiology and causality. For background on the analysis of circadian rhythms through cosinor analysis see Cornelissen (2014) and Refinetti et al. (2014). There are two vignettes: circadian and cosinor.

causalCmprisk v1.0.0: Provides functions to estimate average treatment effects of two static treatment regimes on time-to-event outcomes with competing events. The method uses propensity scores weighting for emulation of baseline randomization. See the vignette.

ccoptimalmatch v0.1.0: Uses sub-sampling to create pseudo-observations of controls to optimally match cases with controls. See Mamoiris (2021) for the theory and the vignette for examples.

CHOIRBM v0.0.2: Provides functions for visualizing body map data collected with the Collaborative Health Outcomes Information Registry CHOIR). See the vignette.

clinDataReview v1.1.0: Provides functions to create interactive tables, listings and figures and associated reports for exploratory analysis in a clinical trial setting. There are vignettes on Prerocessing, Visualization, and Creating Reports.

clinUtils v0.0.4: Provides utility functions to facilitate importing, exploring, and reporting clinical data along with datasets in SDTM and ADaM format. See the Introduction.

cmprskcoxmsm v0.2.0: Provides functions to estimate treatment effect a under marginal structure model for the cause-specific hazard of competing risk events. Functions also estimate the risk of the potential outcomes, risk difference and risk ratio. See Hernan et al. (2001) for the theory and the vignette for examples.

coder v0.13.5: Provides functions to classify individuals or items based on external code data identified by regular expressions. A typical use case considers patients with medically coded data, such as codes from the International Classification of Diseases. There is an overview and vignettes on class codes, interpreting regular expressions, and example data.

covid19us v0.1.2: Implements wrapper around the COVID Tracking Project API providing data on cases of COVID-19 in the US.

covidcast v0.4.2: Provides an interface to Delphi’s COVIDcast Epidata including tools for data access, maps and time series plotting, and basic signal processing, and a collection of numerous indicators relevant to the COVID-19 pandemic in the United States. There is a Getting Started Guide, and vignettes on Computing Signal Correlations, Combining Data Sources, Manipulating Multiple Signals, and Plotting and Mapping Signals.

dataQuieR v1.0.4: Provides functions to assess data quality issues in studies. See the TMF Guideline and the DFG Project for background, and the vignette for examples.

epigraphdb v0.2.1: Provides access to the EpiGraphDB platform. There is an overview, vignettes on the API, Platform Functionality, Meta Functions and three case studies on SNP protein associations, Drug Targets and Causal Evidence

EpiNow2 v1.2.1: Provides functions to estimate the time-varying reproduction number, rate of spread, and doubling time using a range of open-source tools Abbott et al. (2020) for background, Gostic et al. (2020) for current best practices, and README for examples.

escalation v0.1.2: Implements methods for working with dose-finding clinical trials and includes a common interface to various dose-finding methodologies such as the continual reassessment method (CRM) by O’Quigley et al. (1990), the Bayesian optimal interval design (BOIN) by Liu & Yuan (2015), and the 3+3 described by Korn et al. (1994). There are vignettes on Working with dose-paths, Working with dose selectors, and Simulating dose-escalation trials.

eventglm v1.0.2 Implements methods for doing event history regression for marginal estimands, including cumulative incidence the restricted mean survival, as described in the methodology reviewed in Andersen & Perme (2010). See the vignette for examples.

eventTrack v1.0.0: Implements the hybrid framework for event prediction in clinical trials as described in Fang & Zheng (2011). See the vignette for an example.

goldilocks v0.3.0: Implements the Goldilocks adaptive trial design for a time to event outcome using a piecewise exponential model and conjugate Gamma prior distributions as described in Broglio et al. (2014). See the vignette for an example.

healthyR v0.1.1: Implements hospital data analysis workflow tools including modeling tools, and tools to review common administrative hospital data such as average length of stay, readmission rates, average net pay amounts by service lines, and more. See the vignette.

hlaR v0.1.0: Implements a tool for the eplet analysis of donor and recipient HLA (human leukocyte antigen) mismatches. There are vignettes on Imputation and Eplet Mismatch and a Shiny App as well.

inTextSummaryTable v3.0.1: Provides functions to create tables of summary statistics or counts for clinical data for TLFs. These tables can be exported as in-text table for a Clinical Study Report in MS Word format or a presentation MS PowerPoint format, or as interactive table. There is an Introduction and six more vignettes including Aesthetics and Visualization.

IPDfromKM v0.1.10: Implements a method to reconstruct individual patient data from Kaplan-Meier (KM) survival curves, visualize and assess the accuracy of the reconstruction, and perform secondary analysis on the reconstructed data. The package also implements iterative KM estimation algorithm proposed in Guyot (2012).

metaSurvival v0.1.0: Provides a function to assess information from a summary survival curve and test the between-strata heterogeneity. See the GitHub repo for an example.

nCov2019 v0.4.4: Implements an interface to disease.sh - Open Disease Data API to access real time and historical data of COVID-19 cases, vaccine and therapeutics data. There is a vignette.

NHSDataDictionaRy v1.2.1: Provides a common set of simplified web scraping tools for working with the NHS Data Dictionary.This package was commissioned by the NHS-R community to provide this consistency of lookups. See the vignette to get started.

NMADiagT v0.1.2: Implements the hierarchical summary receiver operating characteristic model developed by Ma et al. (2018) and the hierarchical model developed by Lian et al. (2019) for performing meta-analysis. It is able to simultaneously compare one to five diagnostic tests within a missing data framework.

packDAMipd v0.1.2: Provides functions to construct both time-homogenous and time-dependent Markov models for cost-effectiveness analyses, perform decision analyses, and conduct deterministic and probabilistic sensitivity analyses. There are vignettes on deterministic and probabilistic sensitivity analyses, simple “sick-sicker” models, age-dependent “sick-sicker” models, and cycle dependent models.

patientProfilesVis v2.0.1: Provides functions to create patient specific profile visualizations for exploration, diagnostic or monitoring purposes during a clinical trial which display the evolution of parameters such as laboratory measurements, ECG data, vital signs, adverse events and more. There is template for creating patient profiles from CDISC SDTM datasets, and an Introduction to the package.

psrwe v1.2: Provides tools to incorporate real-world evidence (RWE) into regulatory and health care decision making and includes functions which implement the PS-integrated RWE analysis methods proposed in Wang et al. (2019), Wang et al. (2020), and Chen et al. (2020). There is a vignette on propensity score integration.

QDiabetes v1.0-2: Calculates the risk of developing type 2 diabetes using risk prediction algorithms derived by ClinRisk. Look here for information and examples.

raveio v0.0.3: implements an interface to the RAVE (R analysis and visualization of human intracranial electroencephalography data) project which aims at analyzing brain recordings from patients with electrodes placed on the cortical surface or inserted into the brain. See Mafnotti et al. (2020) for background.

reconstructKM v0.3.0: Provides functions for reconstructing individual-level data (time, status, arm) from Kaplan-MEIER curves published in academic journals. See Sun et al. (2018) for background and the vignette for the reconstruction procedure.

RevieweR v2.3.6: Implements a portable Shiny tool to explore patient-level electronic health record data and perform chart review in a single integrated framework. This tool supports the OMOP common data model as well as the MIMIC-III data model, and chart review through a REDCap API. See the RevieweR Website for more information. There are several vignettes including Local, Docker, BigQuery and Shiny Server deployment and performing a Chart Review.

RHRT v1.0.1: Provides methods to scan RR interval data for Premature Ventricular Complexes and parameterise and plot the resulting Heart Rate Turbulence. See Schmidt et al. (1999) and Blesius et al. (2020) and the vignette for examples.

SAMBA v0.9.0: Implements several methods, as proposed in Beesley & Mukherjee (2020) for obtaining bias-corrected point estimates along with valid standard errors using electronic health records data with misclassifird EHR-derived disease status. See the vignette for details.

SteppedPower v0.1.0: Provides tools for power and sample size calculations and design diagnostics for longitudinal mixed models with a focus on stepped wedge designs using methods introduced in Hussey and Hughes (2007) and extensions discussed in Li et al. (2020). See the vignette to get started.

tboot v0.2.0: Provides functions to simulate clinical trial data with realistic correlation structures and assumed efficacy levels by using a tilted bootstrap resampling approach. There is a tutorial on The Tilted Bootstrap and another on Bayesian Marginal Reconstruction.

Tplyr v0.1.3: Implement a tool to simplify table creation and the data manipulation necessary to create clinical reports. There is a Getting Started Guide, and vignettes on Layers, Options, and Tables.

visR v0.2.0: Provides functions to generate clinical graphs and tables with sensible defaults based on graphical principles as described in: Vandemeulebroecke et al. (2018), Vandemeulebroecke et al. (2019), and Morris et al. (2019). Vignettes include Survival Analysis using CDISC ADaM standard, Creating Consort Flow Diagram, Styling Survival Plots and Survival Analysis.

R Views Call for Documentation

Wed, 04 Aug 2021 00:00:00 +0000

We at R Views believe that the R ecosystem, centered around CRAN and Bioconductor, is the world’s largest open repository of statistical knowledge. R packages provide implementations and examples for a tremendous number of statistical methods, procedures, and algorithms. Yet, the virtual library of the R ecosystem is far from being complete. There is plenty of room for examples that explore some little travelled path of computational statistics or illuminate a familiar field with clarity. We invite you to contribute to expanding the knowledge available in the R ecosystem through blogging.

Documentation for R packages begins with the package pdf file which provides a detailed description for each individual function, indicates relationships among functions and often includes references to the statistical and scientific literature. Going down an organizational level you can view the source code for each function. Moving up a level, README files and package vignettes provide a coherent overview of the capabilities of the package as a whole and often include example and elaborate use cases.

While the burden to explain what R packages do falls mainly on package authors, all of us data scientists, statisticians, researchers, students and “ordinary” R users can add to the knowledge encoded in R by elaborating on the capabilities of R functions and packages, developing additional examples and contributing statistical analyses in areas where we may have developed some expertise.

^{xkcd 1805 Unpublished Discoveries}

We are not talking about Nobel Laureate work here. But nearly everyone who regularly uses R has some gem stashed away somewhere. We are looking to mine those gems and organize them in a way that can be shared with the R Community.

We are asking you to please dig up your gem and organize it into a blog post. Tell us what your gem does, how it works, and why it is of some value. Some of the most popular R Views posts introduce technical topics by providing R based tutorials. Others show how to efficiently accomplish everyday tasks or present an elegant calculation or visualization. Think in terms of a well explained example, or if you are really ambitious, you could write that package vignette that you wish existed. Look below for some guidance on more elaborate ideas.

Call for Posts

Please submit you posts using the following form: rstd.io/rviews-2021. All submissions will appear on the Call for Docs page on the RStudio Community Site, and you will need an RStudio Community profile.

The deadline for submission is Friday, September 23, 2021. We planning awards of a sort to the best entries and would like to announce these by November 1, 2021.

Awards

The award categories and the associated prizes are as follows:

Honorable Mentions:
- One year free RStudio Cloud Premium
- A bunch of hex stickers of RStudio packages
Runners Up:
- All prizes listed above, plus
- Some number of RStudio t-shirts, books, and mugs (worth up to $200)
Grand Prizes:
- All prizes listed above, plus
- Special & persistent recognition by RStudio in the form of a winners page, and a badge that will be publicly visible on your RStudio Community profile
- A selected group of submissions will be invited to appear on R Views

^{* Please note that we may not be able to send t-shirts, books, or other larger items to non-US addresses.}
^{* Please note that all articles on R Views will go through an editorial review before being published there.}
^{* With the author’s permission, some posts may appear on R Views before the awards announcements.}.

Announcement

The names and work of all winners will be highlighted in an announcement on R Views, and we will announce them on RStudio’s social platforms, including RStudio Community (unless the winner prefers not to be mentioned).

Need inspiration?

Edgar Ruiz Gentle Introduction to tidymodels is a good example of a homemade package vignette.
Jonathan Regenstein’s series on Reproducible Finance with R presents several masterful examples of Financial use cases.

Guidance

Posts should be well-written, contain enough code to support the main argument and have a few plots and/or tables.
Novelty - We are looking for original content. That is, your article should not have appeared on blogging platforms, other than your personal website or blog.
Contribution - Your article should provide novel insight and utility. For example, an exploration of a package or use case that is currently poorly documented will be better received than one which already has excellent documentation and vignettes either on CRAN or in the R community.
Reproducible - Your submission will need to link to a repo which includes an .Rmd file that reproduces all code, images, and other output in the submitted article. If there is code that requires special resources, such as access to a private database, it is fine to simply refer to these in your repo.
Please confine your posts to less than 1,000 words, excluding code and image captions. Your article should be self-contained, but may refer to additional content on your repo or personal website.

June 2021: "Top 40" New CRAN Packages

Mon, 26 Jul 2021 00:00:00 +0000

One hundred ninety-seven new packages made it to CRAN in June. Here are my selections for the “Top 40” in ten categories: Computational Methods, Data, Finance, Genomics, Machine Learning, Medicine, Statistics, Time Series, Utilities, and Visualization. The Medicine category includes multiple packages for medical reporting and table building. Note that eight new packages were removed from CRAN by the time I began my research for this post on July 16th, so they were not considered for the “Top 40”.

Computational Methods

disordR v0.0-2: Provides tools for manipulating values of associative maps which are stored in arbitrary order. When associating keys with values one needs both parts to be in 1-1 correspondence. See the vignette for the theory and examples.

ICvectorfields v0.0.2: Provides functions for converting time series of spatial abundance or density data in raster format to vector fields of population movement using the digital image correlation technique. See the vignette.

rim v0.4.1: Provides an interface to the computer algebra system Maxima which includes running Maxima commands from within R, generating output in LaTeX and MathML and R Markdown. Look here for examples.

Data

PlanetNICFI v1.0.3: Provides functions to download and process Planet NICFI satellite imagery from Norway’s International Climate and Forest Initiative utilizing the Planet Mosaics API. See the vignette.

rgovcan v1.0.3: Provides access to data and other resources available through the Canadian Open Government portal. See README for examples.

wcde v0.0.1: Implements an interface to the Wittgenstein Human Capital Data Explorer. See the vignette for an overview.

Finance

etrm v1.0.1: Provides functions to perform core tasks within Energy Trading and Risk Management including calculating maximum smoothness forward price curves for electricity and natural gas contracts with flow delivery, as presented in Benth et al. (2007) and portfolio insurance trading strategies for price risk management in the forward market as described in Black (1976). There are vignettes describing the Maximum Smoothness Forward Curve and Portfolio Insurance Trading Strategies.

mshap v0.1.0: Provides functions to compute mSHAP values on two-part models as proposed by Matthews & Hartman (2021) using the TreeSHAP algorithm described in Lundberg et al. (2020). See the vignettes mSHAP and mshap plots.

Genomics

harmony v0.1.0: Implements the harmony algorithm for single cell integration described in Korsunsky et al. (2019). See the vignette.

Machine Learning

AutoScore v0.2.0: Implements an interpretable machine learning framework to automate the development of a clinical scoring model for predefined outcomes. See Xie et al. 2020 for the details, and the Guide Book to get started.

daiR v0.9.0: Implements an interface for the Google Cloud Services Document AI API with additional tools for output file parsing and text reconstruction. See the package website for more information and examples. There are six vignettes including a User Guide, Basic processing, and Extracting Tables.

luz v0.1.0: Implements a high level interface to torch providing utilities to reduce the the amount of code needed for common tasks, abstract away torch details and make the same code work on both CPUs and GPUs. See Howard et al. (2020) and Falcon et al. (2019) for background and the vignettes Get started with Luz, Custom Loops and Accelerator API.

mcboost v0.3.0: Implements Multi-Calibration Boosting and Multi-Accuracy Boosting to for the multi-calibrate the predictions of machine learning models. See the vignettes Basics and Extensions and Health Survey Example.

RFpredInterval v1.0.2: Implements various prediction interval methods with random forests and boosted forests. See Alakus et al. (2021) and Roy and Larocque (2020) for the mathematical background.

Medicine

clinUtils v0.0.4: Provides utility functions to facilitate importing, exploring, and reporting clinical data along with datasets in SDTM and ADaM format. See the Introduction.

Statistics

admix v0.3.2: Implements several methods to estimate the unknown quantities related to two-component admixture models, where the two components can belong to any distribution. See Bordes & Vandekerkhove (2010), Patra & Sen (2016), and Milhaud et al. (2021) for background. There are vignettes on Clusterine, Estimation, and Hypothesis Testing.

ahMLE v1.18: Implements methods for fitting additive hazards model which include the maximum likelihood method as well as Aalen’s method for estimating the additive hazards model. See Chengyuan Lu(2021) for details and the vignette for an example.

bayesrules v0.0.1: Provides datasets and functions used for analysis and visualizations in the online Bayes Rules book. There is a vignette on Conjugate Families and another on Model Evaluation.

dbglm v1.0.0: Provides a function to fit generalized linear models on moderately large datasets, by taking an initial sample, fitting in memory, then evaluating the score function for the full data in the database. See Lumley (2019) for the details.

fasano.franceschini.test v1.0.1: Implements the the 2-D Kolmogorov-Smirnov (KS) two-sample test as defined in Fasano & Franceschini (1987). See the vignette for examples.

flatness v0.1.4: Provides S3 classes, plotting functions, indices and tests to analyze the flatness of histograms including functions for flatness tests introduced in Jolliffe & Primo (2008), flatness indices described in Wilks (2019), and the procedure for multiple hypothesis described in Benjamini & Hochberg (1995). See the vignette for examples.

outlierensembles v0.1.0: Provides ensemble functions for detecting outliers and anomalies including a new method based on Item Response Theory described in Kandanaarachchi (2021) and methods described in Schubert et al. (2012), Chiang et al. (2017), and Aggarwal and Sathe (2015). See the vignette for examples.

susieR v0.11.42: Implements methods for variable selection in linear regression based on the Sum of Single Effects model, as described in Wang et al (2020). The Iterative Bayesian Stepwise Selection algorithm allows fitting models to large data sets with thousands of samples and hundreds of thousands of variables. There are ten short vignettes including Trend Fitting and a minimal example.

Time Series

psdr v1.0.1: Provides functions to generate and compare power spectral density plots given time series data and to compare the dominant frequencies of multiple groups of time series. Look here and here for the mathematical background. For examples look here or see this vignette. There is also an Introduction.

proteus v1.0.0 Implements a Sequence-to-Sequence Variational Model for time-feature analysis based on a wide range of different distributions. Look here for an overview with examples.

Utilities

archive v1.0.2: Implements bindings to libarchive, the multi-format archive and compression library which offer connections and direct extraction for many archive formats including tar, ZIP, 7-zip, RAR, CAB and compression formats including gzip, bzip2, compress, lzma and xz. See README for examples.

pasadr v1.0: Implements the anomaly detection method described in Aoudi et al. (2018).

recogito v0.1.1: Provides htmlwidgets bindings to the recogito and annotorious libraries to annotate text and areas of interest in images. See README to get started.

rextendr v0.2.0: Provides functions to compile and load Rust code from R along with helper functions to create R packages that use Rust code. There is a vignette on Using Rust code in R packages and another on Setting up Rust.

shinymeta v0.2.0.1: Provides tools for capturing logic in a Shiny app and exposing it as code that can be run outside of Shiny (e.g., from an R console). It also provides tools for bundling both the code and results to the end user. See README for examples

Visualization

dynplot v1.1.1: Provides functions to visualize a single-cell trajectory as a graph or dendrogram as a dimensionality reduction or heatmap of the expression data, or a comparison between two trajectories as a pairwise scatterplot or dimensionality reduction projection. See Saelens et al. (2019) for background and the vignette for examples.

gridpattern 0.2.1: Provides grid grabs to fill in a user-defined plot area with various patterns, including geometric and image-based patterns, and support for custom user-defined patterns. There is a vignette.

netplot v0.1-1: Implements a graph visualization engine that puts an emphasis on aesthetics while providing default parameters that yield out-of-the-box-nice visualizations. There is a vignette with base plot examples, and another showing graph drawing with netplot.

NGLVieweR v1.3.1: Implements an htmlwidgets interface to NGL.js enabling users to visualize and interact with protein databank PDB and structural files in R and Shiny applications. See the vignette for examples.

precisePlacement v0.1.0: Provides a selection of tools that make it easier to place elements onto a base R plot exactly where you want them. See the vignette for an overview.

Exploratory Functional PCA with Sparse Data

Thu, 08 Jul 2021 00:00:00 +0000

I have written about the basics of Functional Data Analysis in three prior posts. In Post 1, I used the fda package to introduce the fundamental concept of using basis vectors to represent longitudinal or time series data as a curve in an abstract vector space. In Post 2, I continued to rely on the fda package to show basic FDA descriptive statistics. In Post 3, I introduced Functional PCA with the help of the fdapace package. In this post, I pick up where that last post left off and look at how one might explore a sparse, longitudinal data set with the FPCA tools provided in the fdapace package. I will begin by highlighting some of the really nice tools available in the brolgar package for doing exploratory longitudinal data analysis. While the first three posts made do with artificial data constructed with an algorithm for generating data from a Wiener Process, in this post I’ll use the wages data set available in brolgar.

brolgar is a beautiful, tidyverse inspired package, based on the tsibble data structure that offers a number of super helpful functions for visualizing and manipulating longitudinal data. See the arXiv paper Tierney, Cook and Prvan (2021) for an overview, and the seven package vignettes for examples. Collectively, these vignettes offer a pretty thorough exploration of the wages data set. Using wages for this post should provide a feeling for what additional insight FDA may have to offer.

Longitudinal Data

Let’s look at the data. wages contains measurements on hourly wages associated with years of experience in the workforce along with several covariates for male high school dropouts who were between 14 and 17 years old when first measured. In what follows, I will use onlyln_wages, the natural log of wages in 1990 dollars, and xp the number of years of work experience.

library(brolgar)
library(fdapace)
library(tidyverse)
library(plotly)
dim(wages)

## [1] 6402    9

head(wages)

## # A tsibble: 6 x 9 [!]
## # Key:       id [1]
##      id ln_wages    xp   ged xp_since_ged black hispanic high_grade
##   <int>    <dbl> <dbl> <int>        <dbl> <int>    <int>      <int>
## 1    31     1.49 0.015     1        0.015     0        1          8
## 2    31     1.43 0.715     1        0.715     0        1          8
## 3    31     1.47 1.73      1        1.73      0        1          8
## 4    31     1.75 2.77      1        2.77      0        1          8
## 5    31     1.93 3.93      1        3.93      0        1          8
## 6    31     1.71 4.95      1        4.95      0        1          8
## # … with 1 more variable: unemploy_rate <dbl>

A summary of the variables shows that wages vary from 2 dollars per hour to a high of about 73 dollars per hour. Time varies between 0 and almost 13 years.

wages %>% select(ln_wages, xp) %>% summary()

##     ln_wages           xp               id       
##  Min.   :0.708   Min.   : 0.001   Min.   :   31  
##  1st Qu.:1.591   1st Qu.: 1.609   1st Qu.: 3194  
##  Median :1.842   Median : 3.451   Median : 6582  
##  Mean   :1.897   Mean   : 3.957   Mean   : 6301  
##  3rd Qu.:2.140   3rd Qu.: 5.949   3rd Qu.: 9300  
##  Max.   :4.304   Max.   :12.700   Max.   :12543

Next, we construct the tsibble data structure to make use of some of the very convenient brolgar sampling functions, and count the number of observations for each subject.

wages_t <- as_tsibble(wages,
                    key = id,
                    index = xp,
                    regular = FALSE)

num_obs <- wages_t %>% features(ln_wages,n_obs)
summary(num_obs$n_obs)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    5.00    8.00    7.21    9.00   13.00

Just like the brolgar package vignettes, we filter out subjects with less than 3 observations. Then we use the sample_n_keys() function to generate a random sample of 10 wages versus year’s experience curves and plot them.

df <- wages_t %>%  add_n_obs() %>% filter(n_obs > 3)
set.seed(123)
df %>%
  sample_n_keys(size = 10) %>%
  ggplot(aes(x = xp,  y = ln_wages, group = id, color = id)) + 
  geom_line()

brolgar makes it easy to generates lots of panels with small numbers of curves in order to get a feel for the data.

df %>% ggplot(aes(x = xp, y = ln_wages, group = id, colour = id)) +
        geom_line() +
        facet_sample(n_per_facet = 3, n_facets = 20)

Finally, for comparison with the plots produced by fdapace we plot the curves for the first two subjects in the tsibble.

df  %>% filter(id == 31 | id == 36)  %>% 
        ggplot(aes(x = xp, y = ln_wages, group = id, color = id)) + 
         geom_line()

We finish our initial look at the data by noting that we are really dealing with sparse data here. Some curves have only 4 measurements, no curve has more than 13 measurements, and all subjects were measured at different times. This is a classic longitudinal data set.

Functional PCA

As I mentioned in my previous post, Principal Components by Conditional Expectation (PACE), described in Yao, Müller & Wang (2005), was designed for sparse data. The method works by pooling the data. Curves are not individually smoothed. Instead, estimates of the FPC scores are obtained from the entire ensemble of data. (See equation (5) in the reference above.)

The first step towards using FPCA functions in the fdapace package is to reshape the data so that the time and wages data for each subject are stored as lists in separate columns of a tibble where each row contains all of the data for a single id. (Standard dplyr operations might not work as expected on a tsibble.) The following code pulls just the required data into a tibble before the dplyr code in the somewhat untidy ‘for loop’ builds the data structure we will use for the analysis.

df2 <- df %>% select(id, xp, ln_wages) %>% as_tibble()

uid <- unique(df2$id)
N <- length(uid)
Wages <- rep(0,N)
Exp <- rep(0,N)

for (k in 1:N){
  Wages[k] <-  df2 %>% filter(id == uid[k]) %>% select(ln_wages) %>% pull()  %>% list()
  Exp[k]  <-   df2 %>% filter(id == uid[k]) %>% select(xp)  %>% pull() %>% list()
}
df3 <- tibble( uid, Wages, Exp )
glimpse(df3)

## Rows: 764
## Columns: 3
## $ uid   <int> 31, 36, 53, 122, 134, 145, 155, 173, 207, 222, 223, 226, 234, 24…
## $ Wages <list> <1.491, 1.433, 1.469, 1.749, 1.931, 1.709, 2.086, 2.129>, <1.98…
## $ Exp   <list> <0.015, 0.715, 1.734, 2.773, 3.927, 4.946, 5.965, 6.984>, <0.31…

he FPCA() function computes the functional principal components. Note that in the function call the input data are the two columns of lists we created above. The dataType parameter specifies the data as being sparse. error = FALSE means we are using a simple model that does not account for unobserved error. kernel =epan` means that the we are using the Epanechnikov for smoothing the pooled data to compute the mean and covariance. (For this data set, Epanechnikov seems to yield better results than the default Gaussian kernel.)

res_wages <- FPCA(df3$Wages,df3$Exp, list(dataType='Sparse', error=FALSE, kernel='epan', verbose=TRUE))

The plot method for the resulting FPCA object provides a visual summary of the results. Going clockwise from the upper left, the Design Plot shows that collectively the data are fairly dense over the support except at the upper range near twelve years. The computed mean function for the data shows a little dip in wages near the beginning and then a clear upward trend until it abruptly drops towards the end. The first three eigenfunctions are plotted at the bottom right, and the scree plot at the bottom left shows that the first eigenfunction accounts for about 60% of the total variation and that it takes about eight eigenfunctions to account for 99% of the variance. Note that the default label for the time access in all of the fdapace plots is s for support.

plot(res_wages)

We can obtain the exact estimates for the cumulative Fraction of Variance Explained by picking cumFVE out of the FPCA object.

round(res_wages$cumFVE,3)

##  [1] 0.591 0.739 0.806 0.860 0.908 0.936 0.957 0.975 0.983 0.989 0.993

The following plot shows the smoothed curves estimated for the first two subjects. These are the same subject plotted in the third plot above. The circles indicate the input data. Everything looks pretty good here.

CreatePathPlot(res_wages, subset=1:2, main = 'Estimated Paths for IDs 31 and 36'); grid()

But looking at just one more subject shows how quickly things can apparently go off the rails. The green curve for subject id 53 after 1.77 years is pure algorithmic imagination. Although there are several data points in the first two years, there is nothing thereafter.

CreatePathPlot(res_wages, subset= 1:3, showMean=TRUE)

The value of the FPCA analysis lies in estimating the mean function and the covariance operator which are constructed from the pooled data, and not in predicting an individual paths.

The covariance surface is easily plotted with the help of the extractor function GetCovSurface() which fetches the time grid and associated covariance surface stored in the FPCA object. These are in the right format for a three dimensional, interactive plotly visualization. Rotating the plot and changing the viewing angle reveals quite a bit about the details of the estimated covariance surface.

covS <- GetCovSurface(df3$Wages,df3$Exp)
x <- covS$workGrid
Surf <- covS$cov

fig <- plot_ly(x = x, y = x, z = ~Surf) %>% 
  add_surface(contours = list(
    z = list(show=TRUE,usecolormap=TRUE, 
             highlightcolor="#ff0000", project=list(z=TRUE))))

fig <- fig %>% 
  layout(scene = list(camera=list(eye = list(x=1.87, y=0.88, z=-0.64))))

fig

The next plot, the Modes of Variation Plot shows the modes of variation about the mean for the first two eigenfunctions. The mean function is indicated by the red line. The dark gray shows the variation of the first eigenfunction, and the light gray shows the variation of the second. The plot indicates that all of the subjects begin their careers with similar entry level wages, start to diverge by their second year, reach peak variation around year eight and then begin to converge again by year eleven.

CreateModeOfVarPlot(res_wages, main = "Modes of Variation of Eigenvectors")

### Some Conclusions

This short post neither presents a comprehensive view of functional principal components analysis, nor does it provide the last word on the wages data set. Nevertheless, juxtaposing the highly visual but traditional exploratory analysis conducted by the authors of the brolgar package with a basic FPCA look does provide some insight into the promise and pitfalls of using FPCA to explore sparse, longitudinal data sets.

On the promise side:

FDA offers a global perspective that facilitates thinking about an individual subject’s time path as a whole. For the wages data set, we see individual trajectories developing in a wages/time space that can be parsimoniously represented, analyzed and compared.
FPCA works with very sparse data, does not require the same number of observations for each subject, and does not demand that the observations be taken at the same time points.
There is really no concept of missing data per se, and no need for data imputation. The amount of information required to represent a subject can vary over a wide range.

As to the pitfalls:

As with plain old multivariate PCA, eigenvectors may not have any obvious meaning, and trajectories in an abstract space may be difficult to interpret.
There is really no way to avoid the missing data daemon. The wages data set shows the sensitivity of FPCA trajectories to both the number and the locations of the observed data points. It is not possible to reconstruct individual trajectories for which there are too few observations.

That’s all for now. Thank you for following along.

May 2021: "Top 40" New CRAN Packages

Thu, 24 Jun 2021 00:00:00 +0000

Two hundred five packages made it to CRAN in May, but seven were removed before this post went to print. Here are my “Top40” picks in ten categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

madgrad v0.1.0: Implements MADGRAD, a Momentumized, Adaptive Dual Averaged Gradient method for stochastic optimization. See Defazio & Jelassi (2021) for details and README to get started.

TriDimRegression v1.0.0.0: Provides functions to fit 2D and 3D transformations using Stan which return posterior distributed for fitted parameters. There are vignettes on Transformation Matrices, Eye Gaze Mapping, and Comparing Faces. See README to get started.

Data

AtmChile v0.1.0: Provides access to air quality and meteorological information from the Chile’s National Air Quality System (S.I.N.C.A.). See READMW to get started.

basemaps v0.0.1: Provides a lightweight interface to access spatial basemaps from open sources such as OpenStreetMap, Mapbox and others. See README to get started.

causaldata v0.1.1: Contains the data sets to run the example problems in the online causal inference textbooks The Effect and Causal Inference: What If and more.

exoplanets v0.2.1: Provides access to NASA’s Exoplanet Archive. See the vignette to get started.

frenchdata v0.1.1: Provides access to Kenneth’s French finance data library. See the vignette for basic usage.

tradepolicy v0.5.0: Provides access to the data sets from Yotov et al. (2016) along with an online book containing commentary and the code to recreate the original analysis.

Genomics

artemis v1.0.7: Provides a modeling framework for the design and analysis of experiments collecting environmental DNA. There is an Introduction and also vignettes on Modeling eDNA and qPCR Data and Simulating eDNA Data.

MAGEE v1.0.0: Provides functions to perform variant set-based main effect tests, gene-environment interaction tests, and joint tests for association, as proposed in Wang et al. (2020). See the vignette for details.

MultIS v0.5.1: Implements a bioinformatical approach to detect the multiple integration of viral vectors within the same clone. See the vignette for how to use the package.

TopDom v0.10.0: Provides functions to identify topological domains in genomes from Hi-C sequence data as described in Shin et al. (2016). See README to get started.

Machine Learning

cjbart v0.1.0: Implements a tool for analyzing conjoint experiments using Bayesian Additive Regression Trees (BART), a machine learning method developed by Chipman & McCulloch (2010). See the vignette for examples.

fastText v1.0.1: Implements an interface to Facebook’s fastText Library. See Bojanowski et al. (2017) for a description of the algorithm. There is a Benchmark vignette and an Introduction.

Medicine

eventTrack v1.0.0: Implements the hybrid framework for event prediction in clinical trials as described in Fang & Zheng (2011). See the vignette for an example.

Science

CopernicusDEM v1.0.1: Provides an interface to the Copernicus DEM Digital Elevation Model of the European Space Agency with 90 and 30 meters resolution using the AWS CLI command line tool. See the vignette for an example.

nimbleCarbon v0.1.2: Provides functions and a custom probability distribution for Bayesian analyses of radiocarbon dates within the nimble modeling framework, including a suite of functions for prior and posterior predictive checks for demographic inference as described in Crema & Shoda (2021). See the Introduction.

Statistics

bayesmodels v0.1.0: Implements a framework to bring a number of Bayesian models into the tidymodels ecosystem. See the vignette for an overview.

div v0.3.1: Provides functions to facilitate the analysis of teams in a corporate setting, assess the diversity per grade and job, search for bias and also provides methods to simulate the effects of bias. See De Brouwer (2021) and De Brouwer (2020) for background. Look here to get started.

HotellingEllipse v0.1.1: Provides functions to compute the semi-axes lengths and coordinate points of Hotelling ellipse. See Bro & Smilde (2014) and Brenton (2016) for background. Look here and at the vignette for examples.

makemyprior v1.0.0: Provides tools to construct and visualize joint priors for variance parameters. Vignettes provide examples for Latin Square, i.i.d. models, neonatal mortality, and wheat breeding.

Rage v1.0.0: Provides functions for calculating life history metrics using matrix population models (MPMs) as described in Jones et al. (2021). There is a Getting Started Guide and vignettes on Vital Rates, Life History Traits, Deriving Age, and Ternary Plots.

unusualprofile v0.1.0: Provides functions to calculate Mahalanobis distance for every row of a set of outcome variables. There is an Introduction and a vignette on the calculations.

Time Series

gsignal v0.3-2: Implements the Ovtave signal package which provides a variety of signal processing tools, such as signal generation and measurement, correlation and convolution, filtering, filter design, filter analysis and conversion, power spectrum analysis, system identification, decimation and sample rate change, and windowing. See the vignette for an introduction.

legion v0.1.0 Provides functions for implementing multivariate state space models such as Vector Exponential Smoothing and Vector Error-Trend-Seasonal models, for time series analysis and forecasting as described in de Silva et al. (2010) There is a Function Overview and vignettes on Vector ES and Vector ETS.

Utilities

parsermd v0.1.2: Implements formal grammar and parser for R Markdown documents using the Boost Spirit X3 library. It also includes a collection of high level functions for working with the resulting abstract syntax tree. There is a Getting Started Guide and a vignette on Rmd Templates.

riskmetric v0.1.0: Provides facilities for assessing R packages against a number of metrics to help quantify their robustness. Look here for background on the package and here for background on the R Consortium, R Validation Hub project. There is a Quick Start Guide and a vignette on Extending riskmetric.

shinyvalidate v0.1.0: Provides functions to improve the user experience of Shiny apps by providing feedback when required inputs are missing, or input values are not valid. See README to get started.

ttt v1.0: Provides tools to create structured, formatted HTML tables. See the vignette.

Visualization

fitbitViz v1.0.1: Implements a connection to the Fitbit Web API to provide ggplot2, Leaflet and Rayshader visualizations. See the vignette for examples.

ggbreak v0.0.3: Implements scale functions for setting axis breaks for ggplot2. See the vignette.

ggpp v0.4.0: Provides extensions to ggplot2 to add inserts to plots using both native and npc data coordinates. See the vignette for examples.

ggseg v1.6.3: Implements a ggplot geom for plotting brain atlases using simple features. The largest component of the package is the data for two built-in atlases. See Mowinckel & Vidal-Piñero (2020) for background. There is an Introduction along with vignettes on external data, Freesurfer files, using atlases.

ichimoku v0.2.0: Implements Ichimoku Kinko Hyo, also commonly known as cloud charts, including static and interactive visualizations with tools for creating, backtesting and developing quantitative ichimoku strategies. There is a Reference and a vignette on Strategies.

liminal v0.1.2: Provides functions for composing interactive visualizations and creating linked interactive graphics for exploratory high-dimensional data analysis. See Lee et al. (2020) for background. There is a vignette on Exploring Non-linear Embeddings and another on the the geometry of Parameter Space.

mipplot v0.3.1: Provides generic functions to produce area, bar, box, and line plots following Integrated Assessment Modeling Consortium (IAMC) submission format in order to visualize climate migration scenarios. See the vignette for first steps.

qqboxplot v0.1.0: Implements Q-Q boxplots as an extension to ggplot2. There is a vignette on Basic Usage and another that provides Examples.

Summer Conferences!

Thu, 17 Jun 2021 00:00:00 +0000

Summer is here, but it is not too late sign up for some summer conferences. The following short list promises interesting speakers, a wide range of topics and plenty of R content.

June (21 - 23) - The PSI 2021 conference is online and the Registration Portal is still open. Keynote speakers Alan Smith and Ian Bott from the Financial Times, and Janet Wittes, President of the WCG Statistics Collaborative, head the program.

June (21 - 24) - The Nonclinical Biostatistics Conference 2021 is virtual. Wendy Martinez of the Bureau of Labor statistics will present A Conversation About Data Ethics, Nassim Taleb of Black Swan fame will deliver the keynote on Statistical Consequences of Fat Tails, and Di Cook and RStudio’s Carson Sievert will both talk in the Statistical Computational & Visualization Session. Registration is open.

July (1 to 2) - R for HTA Annual Workshop This online workshop from the Health Technology Assessment Consortium will be focused on R for trial and model-based cost-effectiveness analysis. Registration is open until June 30.

July (5 - 9) - useR!2021 looks like it is going to be a blockbuster of a conference. The keynote talks alone would be worth the price of admission. This exceptional lineup comprises a remarkably diverse, international group of long-time contributors, new faces, R developers, statisticians, journalists, and educators representing the global R community and speaking on a wide range of topics. I am very pleased to be presenting A little bit about RStudio on July 9 at UTC 9PM. Registration closes on June 25.

July (28 - 30) - Juliacon 2021 will be online and everywhere and Free! Long time R contributor Jan Vitek, Xiaoye Li of the Lawrence Livermore National Laboratory, and Soumith Chintala of Facebook AI Research will be the keynote speakers. It is free but you need to Register.

Aug (8 - 12) - The JSM will be online. A keyword search using R and Shiny will turn up quite a few interesting talks. I am particularly looking forward to the talk Simulating Clinical Trials Data with Synthetic.Cdisc.Data and Respectables] by Gabe Becker and Adrian Waddell and the session on Tools to Enable the Use of R by the Biopharmaceutical Industry in a Regulatory Setting which contains five talks from members of the R Consortium’s R Validation Hub working group.

August (24 - 27) - R/Medicine 2021 is online and on track to repeat the last year’s international success. Registration is open. The deadline for submitting Abstracts is June 25. Workshops being planned include:

R/Med 101: Intro to R for Clinicians and Healthcare Professionals
R Markdown for Reproducible Research (R³)
SAS 2 R: Getting off the Island!
From Excel to R+REDcap
Spatial Analysis of Healthcare Data

Sept (6 to 9) - RSS 2021 International Conference The Royal Statistical Society conference hopes to be in person in Manchester, UK. The keynote speakers will be Tom Chivers and David Chivers, Melinda Mills, Jonty Rougier, Eric Tchetgen Tchetgen, Bin Yu, and Hadley Wickham. Submissions for poster presentations are currently open with a deadline of July 1. Registration is open with an early booking discount available until June 4.

Functional PCA with R

Thu, 10 Jun 2021 00:00:00 +0000

In two previous posts, Introduction to Functional Data Analysis with R and Basic FDA Descriptive Statistics with R, I began looking into FDA from a beginners perspective. In this post, I would like to continue where I left off and investigate Functional Principal Components Analysis (FPCA), the analog of ordinary Principal Components Analysis in multivariate statistics. I’ll begin with the math, and then show how to compute FPCs with R.

As I have discussed previously, although the theoretical foundations of FDA depend on some pretty advanced mathematics, it is not necessary to master this math to do basic analyses. The R functions in the various packages insulate the user from most of the underlying theory. Nevertheless, attaining a deep understanding of what the R functions are doing, or looking into any of the background references requires some level of comfort with the notation and fundamental mathematical ideas.

I will define some of the basic concepts and then provide a high level roadmap of the mathematical argument required to develop FPCA from first principals. It is my hope that if you are a total newcomer to Functional Data Analysis you will find this roadmap useful in apprehending the big picture. This synopsis closely follows the presentation by Kokoszka and Reimherr (Reference 1. below).

We are working in $\mathscr{H}$, a separable Hilbert space of square integrable random functions where each random function, $X(\omega,t)$, where $\omega \in \Omega$ the underlying space of probabilistic outcomes, and $t \in [0,1]$. (After the definitions below, I will suppress the independent variables and in most equations assume $EX = 0$.)

Definitions

A Hilbert Space $\mathscr{H}$ is an infinite dimensional vector space with an inner product denoted by $<.,.>$. In our case, the vectors are functions.
$\mathscr{H}$ is separable if there exists an orthonormal basis. That is, there is an orthogonal collection of functions $(e_i)$ in $\mathscr{H}$ such that $<e_i,e_j>\; = 0$ if $i = j$ and 0 otherwise, and every function in $\mathscr{H}$ can be represented as a linear combination of these functions.
The inner product of two functions $X$ and $Y$ in $\mathscr{H}$ is defined as $<X,Y>\; = \int X(\omega,t) Y(\omega,t)dt$.
The norm of $X$ is defined in terms of the inner product: $\parallel X(\omega) \parallel ^2\; = \int X(\omega, t)^2 dt < \infty$.
$X$ is said to be square integrable if $E\parallel X(\omega) \parallel ^2 < \infty$.
The covariance operator $C(y): \mathscr{H} \Rightarrow \mathscr{H}$ for any square integrable function $X$ is given by: $C(y) = E[<X - EX,y>(X - EX)]$

The Road to Functional Principal Components

As we have seen, the fundamental idea of Functional Data Analysis is to represent a function $X$ by a linear combination of basis elements. In the previous posts we showed how to accomplish this using a basis constructed from more or less arbitrarily selected B-spline vectors. But, is there a an empirical, some would say natural basis that can be estimated from the data? The answer is yes, and that is what FPCA is all about.

A good way to start is to look at the distance between a vector $X$ and its projection down into the space spanned by some finite, p-dimensional basis $(u_k)$ which is expressed in the following equation,

$D = E\parallel X - \sum_{k=1}^{p}<X, u_k>u_k\parallel^2$ $(*)$

This expands out to:

$= E [< (X - \sum_{k=1}^{p}<X, u_k>u_k, X - \sum_{k=1}^{p}<X, u_k>u_k)>]$

and with a little algebra this:

$= E\parallel X \parallel^2 - \sum_{k=1}^{p}E<X, u_k>^2$

It should be clear that we would want to find a basis that makes $D$ as small as possible, and that minimizing $D$ is equivalent to maximizing the term to be subtracted in the line above.

A little algebra shows that, $E<X, u_k>^2 \;=\; <C(u_k),u_k>$ where $C(u_k)$ is the covariance operator defined above.

Now, we are almost at our destination. There is a theorem (e.g. Theorem 11.4.1 in reference 1.) that says for any fixed number of basis elements p, the distance D above is minimized if $u_j = v_j$ where the $v_j$ are the eigenfunctions of $C(y)$ with respect to the unit norm. From this it follows that $E<X, v_j>^2 \;=\; <C(v_,),v_j>\; =\; <\lambda_j, v_j>\; =\; \lambda_j$.

Going back to equation $(*)$, we can expand $X$ in terms of the basis $(v_j)$ so $D = 0$ and we have what is called the Karhunen–Loève expansion: $X = \mu + \sum_{j=1}^{\infty}\xi_jv_j$

where:

$\mu = EX$
The deterministic basis functions $(v_j)$ are called the functional principal components
The $(v_j)$ have unit norm and are unique up to their signs. (You can work with $v_j$ or $-v_j$.)
The eigenvalues are such that: $\lambda_1 > \lambda_2 > . . . \lambda_p$
The random variables $\xi_j =\; <X - \mu,v_j>$ are called the scores.
$E\xi_j = 0$, $E\xi_j^2 = \lambda_j$ and $E|\xi_i\xi_j| = 0,\; if\; i\;\neq\;j$.

And finally, with one more line:
$\sum_{j=1}^{\infty}\lambda_j \: = \: \sum_{j=1}^{\infty}E[<X,v_j>^2] = E\sum_{j=1}^{\infty}<X,v_j>^2 \; = \; E\parallel X \parallel^2 \; < \infty$

we arrive at our destination, the variance decomposition: $E\parallel X - \mu \parallel^2 \;= \;\sum_{j=1}^{\infty}\lambda_j$

Let’s Calculate

Now that we have enough math to set the context, let’s calculate. We will use the same simulated Brownian motion data that we used in the previous posts, and also construct the same B-spline basis that we used before and save it in the fda object W.obj. I won’t repeat the code here.

The following plot shows 120 simulated curves, each having 1000 points scattered over the interval [0, 100]. Each curve has unique observation times over that interval.

For first attempt at calculating functional principal components we’ll use the pca.fd() function from the fda package. So set up wise, we are picking up our calculations exactly where we left off in the previous post. As before, the basis representations of these curves are packed into the fda object W.obj. The function pca.fd() takes W.obj as input. It needs the non-orthogonal B-spline basis to seed its computations and the estimate the covariance matrix and the orthogonal eigenvector basis $v_j$. The nharm = 5 parameter requests computing 5 eigenvalues.

The method of calculation roughly follows the theory outlined above. It starts with a basis representation of the functions, computes the covariance matrix, and calculates the eigenfunctions.

fun_pca <- pca.fd(W.obj, nharm = 5)

The object produced by pca.fd() is fairly complicated. For example, the list fun_pca$harmonics does not contain the eignevectors themselves, but rather coefficients that enable the eigenvectors to be computed from the original basis. However, because there is a special plot method for plot.pca.fd() it is easy to plot the eigenvectors.

plot(fun_pca$harmonics, lwd = 3)

## [1] "done"

It is also to obtain the eivenvalues $\lambda_j$,

fun_pca$values

##  [1] 37.232207  3.724524  1.703604  0.763120  0.547976  0.389431  0.196101
##  [8]  0.163289  0.144052  0.116587  0.089307  0.057999  0.054246  0.050683
## [15]  0.042738  0.035107  0.031283  0.024905  0.019079  0.016428  0.011657
## [22]  0.007392  0.002664

and, the proportion of the variance explained by each eigenvalue.

fun_pca$varprop

## [1] 0.81965 0.08199 0.03750 0.01680 0.01206

A Different Approach

So far in this short series of FDA posts, I have been mostly using the fda package to calculate. In 2003 when it was released, it was ground breaking work. It is still the package that you are most likely to find when doing internet searches, and is the foundation for many subsequent R packages. However, as the CRAN Task View on Functional Data Analysis indicates, new work in FDA has resulted in several new R packages. The more recent fdapace takes a different approach to calculating principal components. The package takes its name from the (PACE) Principal Components by Conditional Expectation algorithm described in the paper by Yao, Müller and Wang (Reference 4. below). The package vignette is exemplary. It describes the methods of calculation, develops clear examples and provides a list of references to guide your reading about PACE and FDA in general.

A very notable feature of the PACE algorithm is that it is designed specifically to work with sparse data. The vignette describes the two different methods of calculation that package functions employ for sparse and non-sparse data. In this post, In this post we are not working with sparse data, but hope to do so in the future. See the vignette for examples of FPCA with sparse data.

library(fdapace)

The fdapace package requires data for the functions (curves) and associated times be organized in lists. We begin by using the fdapace::CheckData() function to check the data set up in the tibble df. (See previous post on descriptive statistics for the details on the data construction.)

CheckData(df$Curve,df$Time)

No error message is generated, so we move on th having the FPCA() function calculate the FPCA outputs including:

W_fpca <- FPCA(df$Curve,df$Time)

the eigenvalues:

W_fpca$lambda

## [1] 37.5386  3.2098  1.0210  0.3533

the cumulative percentage of variance explained by the eigenvalue

W_fpca$cumFVE

## [1] 0.8875 0.9634 0.9875 0.9959

and the scores:

head(W_fpca$xiEst)

##         [,1]    [,2]     [,3]     [,4]
## [1,]  0.8187  1.1971 -1.77755  0.20087
## [2,] -8.7396  2.4165 -0.33768 -0.10565
## [3,] -2.7517 -0.4879 -0.06747 -0.13953
## [4,]  3.9218  0.9419  0.39098 -0.25449
## [5,] -1.4400  0.7691  2.11549 -0.59947
## [6,]  7.3952  0.5114 -2.16391  0.05199

All of these are in fairly good agreement with what we computed above. I am, however, a little surprised by the discrepancy in the value of the second eigenvalue. The default plot method for FPCA() produces a plot indicating the density of the data, a plot of the mean of the functions reconstructed from the eigenfunction expansion, a scree plot of the eigenvalues and a plot of the first three eigenfunctions.

plot(W_fpca)

Finally, it has probably already occurred to you that if you know the eigenvalues and scores, the Karhunen–Loève expansion can be used to simulate random functions. It can be shown that for the Wiener process:

$v_j(t) = \sqrt2 sin(( j - \frac{1}{2})\pi t)$ and $\lambda_j = \frac{1}{(j - \frac{1}{2})^2\pi^2}$

This gives us:

$W(t) = \sum_{j=1}^{\infty} \frac {\sqrt2}{(j - \frac{1}{2})\pi)} N_j sin(( j - \frac{1}{2})\pi t$

where the $N_j \:are\; iid \;N(0,1)$.

The fdapace function fdapace::Wiener() uses this information to simulate an alternative, smoothed version of the Brownian motion, Wiener process.

set.seed(123)
w <- Wiener(n = 1, pts = seq(0,1, length = 100))
t <- 1:100
df_w <- tibble(t, as.vector(w))
ggplot(df_w, aes(x = t, y = w)) + geom_line()

In future posts, I hope to continue exploring the fdapace package, including its ability to work with sparse data.

References

I found the following references particularly helpful.

Kokoszka, P. and Reimherr, M. (2017). Introduction to Functional Data Analysis. CRC.
Hsing, T and Eubank, R. (2015). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators Wiley
Wang, J., Chiou, J. and Müller, H. (2015). Review of Functional Data Analysis
Yao, F., Müller, H, Wang, J. (2012). Functional Data Analysis for Sparse Longitudinal Data JASA J100, I 470

R for Public Health

Wed, 02 Jun 2021 00:00:00 +0000

The COVID19 pandemic has raised the profile of public health workers at all levels from the nurses and doctors working on the front lines at our hospitals, to high level state and federal public health officials. I think its a good bet that eighteen months ago few of us had any clear idea about how the public health care system works, or thought much about the people charged with the awesome responsibility to keep us safe. We are all a little bit wiser now. It strikes me as obvious that we will have a continuing need to improve our public health systems and that this need will create opportunities for data scientists to make significant contributions, in research, logistics, data management, reporting, public communication and many more areas. The recent post by my colleague Jesse Mostipak describes how R and Shiny made a big difference in the nitty gritty work required to roll out vaccine distribution in West Virginia. This four minute video about how the West Virginia Army National Guard built a COVID vaccine inventory management system is inspiring.

The following are some R resources that you may find helpful if you are seeking to increase your R skills with a eye toward public health applications. Some may be useful to public health professionals seeking to learn R, others may be interesting to R users who want to investigate data science in a public health context.

Courses

The Johns Hopkins Graduate Institute of Epidemiology and Biostatistics has a slew of of summer courses (many of them online), including Introduction to R For Public Health Researchers.
If you are feeling optimistic about traveling and can make your to Estonia this summer you might consider Statistical Practice in Epidemiology using R.
If you hurry, you may be able to get into the Coursera course offered by Imperial College London Statistical Analysis with R for Public Health Specialization.
There are several Coursera Courses for R in a public health context.
Frank Harrell’s free online course Biostatistics for Biomedical Research, available on YouTube: BBRcourse, is an excellent introduction to the basic statistical concepts underlying all medical applications.

Books

If you can learn on your own with the help of a good book, here are some ideas.

The Epidemiologist R Handbook: R for applied epidemiology and public health is a delightful, brief, modern introduction to R that covers the basics of the Tidyverse, R Markdown, Shiny and data.table.
Modern Statistics for Modern Biology: This book is focused more on genomics than public health applications, but it is probably the best introductory statistical text available. The modern statistics part in the title means statistics as a data driven, computational based science. Every topic is illustrated with well-crafted R code and visualizations. The book is great read.
Population Health Data Science with R: An introduction to R by authors who see population health as a systems framework for studying and improving the health of populations through collective action and learning.
Epidemiology with R: This is a new, reasonably priced book that covers the basics. It emphasizes reproducibility with R Markdown. Code examples are written in a base R style that matches the extensively used Epi package.

R Packages

If working through a book on you own seems too much of a stretch, then digest an R package or two. Here are a few examples of packages on public health themes with sufficient documentation to make interesting self-learning projects.

EpiModel: An R Package for Mathematical Modeling of Infectious Disease over Networks
PHEindicators: Common Public Health Statistics and their Confidence Intervals
SimInf: An R Package for Data-Driven Stochastic Disease Simulations
SPARSEMODr: Construct spatial, stochastic disease models that show how parameter values fluctuate in response to public health interventions

Shiny

Finally, if you find yourself in a situation similar to the West Virginia Army National Guard team featured in the video, you may just want to teach yourself Shiny. The RStudio Shiny Tutorial, along with Hadley Wickham’s book Mastering Shiny, is a very good place to start. If you need more structure, you might check out the Udemy Courses, or work through the online workshops from Duke University or the University of Manchester. And by all means, immerse yourself in the examples, posts and podcasts of the Shiny Developer Series.

April 2021: "Top 40" New CRAN Packages

Tue, 25 May 2021 00:00:00 +0000

One hundred seventy-nine new packages made it to CRAN in April. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Networks, Operations Research, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

abess v0.1.0: Provides a toolkit for solving the best subset selection problem in linear regression, logistic regression, Poisson regression, Cox proportional hazard model, multiple-response Gaussian, and multinomial regression. It implements and generalizes algorithms described in Zhu et al. (2020) that exploit a novel sequencing-and-splicing technique to guarantee exact support recovery and globally optimal solution in polynomial times. There is an Introduction.

eat v0.1.0: Provides functions to determine production frontiers and technical efficiency measures through non-parametric techniques based upon regression trees. See Esteve et al. (2020) for details. There is an Introduction.

Data

childdevdata v1.1.0: Bundles publicly available data sets with individual milestone data for children aged 0-5 years, with the aim of supporting the construction, evaluation, validation and interpretation of methodologies that aggregate milestone data into informative measures of child development. See README.

datagovindia v0.0.3: Allows users to search the open data platform of the government of India to communicate with the more than 80,000 available APIs. See the vignette.

lehdr v0.2.4: Provides functions to query the LODES FTP server to obtain longitudinal Employer-Household Dynamics data and optionally aggregate Census block-level data. See the vignette.

rbioapi v0.7.0: Provides a consistent R interface to the Biologic Web Services API and fully supports miEAA, PANTHER, Reactome, String, and UniProt. See this vignette to get started.

tidywikidatar v0.2.0: Provides functions to query Wilidata, get tidy data frames in response, and cache data in a local SQLite database. See README.

Genomics

protti v0.1.1: Provides functions and workflows for proteomics quality control and data analysis of both limited proteolysis-coupled mass spectrometry and regular bottom-up proteomics experiments. See Feng et. al. (2014) for background. There are vignettes for various workflows: Dose Response, Single Treatment Dose Response, Input Preparation, and Quality Control.

Rediscover v0.1.0: Implements an optimized method for identifying mutually exclusive genomic events based on the Poisson-Binomial distribution that takes into account that some samples are more mutated than others. See Canisius et al. (2016). The vignette provides an introduction.

Machine Learning

geocmeans v0.1.1: Provides functions to apply spatial fuzzy unsupervised classification, visualize and interpret results, as well as indices for estimating the spatial consistency and classification quality. See Cai et al. (2007), Zaho et al. (2013), and Gelb & Appaericio (2021) for background. There is an Introduction and an additional vignette.

Rforestry v0.9.0.4: Provides fast implementations of Honest Random Forests, Gradient Boosting, and Linear Random Forests, with an emphasis on inference and interpretability. See Kunzel et al. (2019). See README to get started.

Mathematics

elasdics v0.1.2: Provides functions to align curves and to compute mean curves based on the elastic distance defined in the square-root-velocity framework. For information on the framework see Srivastava and Klassen (2016), For more theoretical details see Steyer et al. (2021)

jordan v1.0-1: Provides functions to manipulate Jordan Algebras, commutative but non-associative algebraic structures that satisfy the Jordan Identify: (xy)x² = x(yx²). See McCrimmon (204).

Medicine

ccoptimalmatch v0.1.0: Uses sub-sampling to create pseudo-observations of controls to optimally match cases with controls. See Mamoiris (2021) for the theory and the vignette for examples.

nCov2019 v0.4.4: Implements an interface to disease.sh - Open Disease Data API to access real time and historical data of COVID-19 cases, vaccine and therapeutics data. There is a vignette.

hlaR v0.1.0: Implements a tool for the eplet analysis of donor and recipient HLA (human leukocyte antigen) mismatches. There are vignettes on Imputation and Eplet Mismatch and a Shiny App as well.

Networks

greed v0.5.1: Provides an ensemble of algorithms to enable clustering of networks and data matrices with different type of generative models. Model selection and clustering is performed in combination by optimizing the Integrated Classification Likelihood. The optimization is performed with a combination of greedy local search and a genetic algorithm. See Côme et al. (2021) for background and the vignettes on Gaussian Mixture Models and Clustering.

Operations Research

critpath v0.1.2: Provides functions to compute critical paths, schedules, PERT charts and Gantt charts. There is a vignette on CPM and PERT and another on the LESS Method.

himach v0.1.2: Provides functions to compute the best routes between airports for supersonic aircraft flying subsonic over land. There is an Introduction to Supersonic Routing and a vignette on Advanced Supersonic Routing.

Statistics

convdistr v1.5.3: Provides functions to compute convolutions of probability distributions via a method that creates a new random number function for individual random samples from the random generator function of each distribution. There is an Introduction and a vignette on Sample Size.

gamlss.lasso v1.0-0: Provides an interface for extra high-dimensional smooth functions for Generalized Additive Models for Location Scale and Shape (GAMLSS) including lasso, ridge, elastic net and least angle regression. The gamlss website provides considerable information.

GGMnonreg v1.0.0: Provides functions to estimate non-regularized Gaussian graphical models, Ising models, and mixed graphical models. See Williams et al. (2019), Williams & Rast (2019), and Williams (2020) for details. README contains examples.

relevance v1.1: Implements the concepts of relevance and significance measures introduced in Stahel (2021) to augment inference with p-values. See the vignette for examples.

sasfunclust v1.0.0: Implements the sparse and smooth functional clustering method described in Centofanti et al. (2021) that aims to classify a sample of curves into homogeneous groups while jointly detecting the most informative portions of domain. See README to get started.

survMS v0.0.1: Provides functions to simulate data from the Accelerated Hazard, Accelerated Failure Time, and Cox survival models. See Bender et al. (2004) for the methods used to implement the Cox model, and the vignette and GitHub for an introduction and examples.

TestGardener v0.1.4: Provides functions to develop, evaluate, and score multiple choice examinations, psychological scales, questionnaires, and similar types of data involving sequences of choices among one or more sets of answers. See Ramsay et al. (2020) and Ramsay et al. (2019) for the methodology and the vignettes Symptom Distress Analysis and SweSAT Quantitative Analysis.

wpa v1.5.0: Provides opinionated functions to enable easier and faster analysis of Workplace Analytics data. See the vignette for an introduction.

Time Series

garchmodels v0.1.1: Implements a framework for using GARCH models with the tidymodels ecosystem. It includes both univariate and multivariate methods from the rugarch and rmgarch packages. There is a Getting Started Guide and a vignette on tuning univariate GARCH models.

tensorTS v0.1.1: Provides functions for estimating, simulating and predicting factor and autoregressive models for matrix and tensor valued time series. See Chen et al. (2020), Chen et al. (2020), and Han et al. (2020) for the math.

Utilities

diffmatchpatch v0.1.0: Implements a wrapper for Google’s diff-match-patch library. It provides basic tools for computing diffs, finding fuzzy matches, and constructing / applying patches to strings. See README for examples.

erify v0.2.0: Provides several validator functions to check if arguments passed by users have valid types, lengths, etc., and if not, to generate informative and good-formatted error messages in a consistent style. See the vignette to get started.

juicr v0.1: Provides a GUI interface for automating data extraction from multiple images containing scatter and bar plots, semi-automated tools to tinker with extraction attempts, and a fully-loaded point-and-click manual extractor with image zoom, calibrator, and classifier. See the vignette for examples, and the Youtube channel for a course on meta analysis.

mailmerge v 0.2.1: Allows users to mail merge using markdown documents and gmail, parse markdown documents as the body of email, use the yaml header to specify the subject line of the email, preview the email in the RStudio viewer pane, and send (draft) email using gmailr. See the vignette for examples.

m61r v0.0.2: Provides dplyr and tidyr like data manipulation functions using only base R and no dependencies. See the vignette for examples.

Visualization

flametree v0.1.2: Implements a generative art system for producing tree-like images using an L-system to create the structures. See README to get started.

leafdown v1.0.0: Provides drill down functionality for leaflet choropleths in shiny apps. There is an Introduction and a Showcase example.

mapping v1.2: Provides coordinates, linking and mapping functions for mapping workflows of different geographical statistical units. Geographical coordinates automatically link with the input data to generate maps. See the vignette to get started.

materialmodifier v1.0.0: Provides functions to apply image processing effects to modify the perceived material properties such as gloss, smoothness, and blemishes. Look here for documentation and practical tips of the package is available at

svplots v0.1.0: Implements two versions of sample variance plots illustrating the squared deviations from sample variance as described in Wijesuriya (2020). See the vignette.

vivid v0.1.0: Provides a suite of plots for displaying variable importance and two-way variable interaction. Plots include partial dependence plots laid out in “pairs plot”” or zenplots style. There is an Introduction and a Quick Start Guide.

Basic FDA Descriptive Statistics with R

Fri, 14 May 2021 00:00:00 +0000

In a previous post, I introduced the topic of Functional Data Analysis (FDA). In that post, I provided some background on Functional Analysis, the mathematical theory that makes FDA possible, identified FDA resources that might be of interest R users, and showed how to turn a series of data points into an FDA object. In this post, I will pick up where I left off and move on to doing some very basic FDA descriptive statistics.

Let’s continue with the same motivating example from last time. We will use synthetic data generated by a Brownian motion process and pretend that it is observed longitudinal data. However, before getting to the statistics, I would like to take a tiny, tidy diversion. The functions in fda and other fundamental FDA R packages require data structured in matrices. Consequently, the examples in the basic FDA reference works (listed below) construct matrices using code that seems to be convenient for the occasion. I think this makes adapting sample code to user data a little harder than it needs to be. There ought to be standard data structures for working with FDA data. I propose tibbles or data frames with function values packed into lists.

The following function generates n_points data points for each of n_curve Brownian motion curves that represent the longitudinal data collected from n_curve subjects.

library(fda)
library(tidyverse)
library(plotly)

# Function to simulate data
fake_curves <- function(n_curves = 100, n_points = 80, max_time = 100){
  ID <- 1:n_curves
  x <- vector(mode = "list", length = n_curves)
  t <- vector(mode = "list", length = n_curves)
  
  for (i in 1:n_curves){
    t[i] <- list(sort(runif(n_points,0,max_time)))
    x[i] <- list(cumsum(rnorm(n_points)) / sqrt(n_points))
  }
  df <- tibble(ID,t,x)
  names(df) <- c("ID", "Time", "Curve")
  return(df)
}

Notice that each curve is associated with a unique set of random time points that lie in the interval [0, max_time]. Not being restricted to situations where data from all subjects must be observed at the same times is a big deal. However, in practice you may encounter problems that will require curve alignment procedures. We will ignore this for now. Note that the variables Time and Curve contain lists of data points in each cell.

set.seed(123)
n_curves <- 40
n_points <- 80
max_time <- 100
df <- fake_curves(n_curves = n_curves,n_points = n_points, max_time = max_time)
head(df)

## # A tibble: 6 x 3
##      ID Time       Curve     
##   <int> <list>     <list>    
## 1     1 <dbl [80]> <dbl [80]>
## 2     2 <dbl [80]> <dbl [80]>
## 3     3 <dbl [80]> <dbl [80]>
## 4     4 <dbl [80]> <dbl [80]>
## 5     5 <dbl [80]> <dbl [80]>
## 6     6 <dbl [80]> <dbl [80]>

Later on, this kind of structure will be convenient for data sets that contain both FDA curves and other scalar covariates. Note that if you are using the RStudio IDE running the function View(df) will show you an expanded view of the tibble under a tab labeled df that should look something like this:

Next, we unpack the data into a long form tibble and plot.

df_1 <- df %>% select(!c(ID,Curve)) %>% unnest_longer(Time) 
df_2 <- df %>% select(!c(ID,Time)) %>% unnest_longer(Curve)
ID <- sort(rep(1:n_curves,n_points))
df_l <- cbind(ID,df_1,df_2)
p <- ggplot(df_l, aes(x = Time, y = Curve, group = ID, col = ID)) +
      geom_line()
p

Now that we have the data, remember that FDA treats each curve as a basic data element living in an infinite dimensional vector space. The vectors, $X$, are random functions: $X: \Omega \Rightarrow \mathscr{H}$ where $\Omega$ is an underlying probability space and $\mathscr{H}$ is typically a complete Hilbert Space of square integrable functions. That is, for each $\omega \in \Omega$, $\parallel X(\omega) \parallel ^2 = \int X((\omega)(t))^2 dt < \infty$. In multivariate statistics we work with random variables that live in a Euclidean space, here we are dealing with random functions that live in a Hilbert space. In this context, square integrable means $E \parallel X \parallel ^2 < \infty$. You will find lucid elaborations of all of this in the references below which I have reproduced below from the previous post.

The bridge from the theory to practice is the ability to represent the random functions as a linear combination of basis vectors. This was the topic of the previous post. Here is some compact code to construct the basis.

knots    = c(seq(0,max_time,5)) #Location of knots
n_knots   = length(knots) #Number of knots
n_order   = 4 # order of basis functions: for cubic b-splines: order = 3 + 1
n_basis   = length(knots) + n_order - 2;
basis = create.bspline.basis(rangeval = c(0,max_time), n_basis)
plot(basis)

This next bit of code formats the data in the long form tibble into the matrix input expected by the fda functions and creates an fda object that contains the coefficients and basis functions used to smooth data. Note the smoothing constant of lambda = .5.

argvals <- matrix(df_l$Time, nrow = n_points, ncol = n_curves)
y_mat <- matrix(df_l$Curve, nrow = n_points, ncol = n_curves)

W.obj <- Data2fd(argvals = argvals, y = y_mat, basisobj = basis, lambda = 0.5)

Next somewhat anticlimactically after all of the preparation and theory, we use the fda functions mean.fd() and std.fd() to calculate the pointwise mean and standard deviation from information contained in fda object. In order to use these objects to calculate the pointwise confidence interval for the mean it is necessary to construct a couple of new fda objects for the upper and lower curves. Then, we plot the smoothed curves for our data along with the pointwise mean and pointwise 95% confidence bands for the mean.

W_mean <- mean.fd(W.obj)
W_sd <- std.fd(W.obj)
# Create objects for the standard upper and lower standard deviation
SE_u <- fd(basisobj = basis)
SE_l <- fd(basisobj = basis)
# Fill in the sd values
SE_u$coefs <- W_mean$coefs +  1.96 * W_sd$coefs/sqrt(n_curves) 
SE_l$coefs <- W_mean$coefs -  1.96 * W_sd$coefs/sqrt(n_curves)

plot(W.obj, xlab="Time", ylab="", lty = 1)

## [1] "done"

title(main = "Smoothed Curves")
lines(SE_u, lwd = 3, lty = 3)
lines(SE_l, lwd = 3, lty = 3)
lines(W_mean,  lwd = 3)

Finally, we compute the covariance/correlation matrix for our sample of smoothed curves and use plotly to create an interactive plot of the three dimensional correlation surface along with a contour map.

days <- seq(0,100, by=2)
cov_W <- var.fd(W.obj)
var_mat <-  eval.bifd(days,days,cov_W)

fig <- plot_ly(x = days, y = days, z = ~var_mat) %>% 
  add_surface(contours = list(
    z = list(show=TRUE,usecolormap=TRUE, highlightcolor="#ff0000", project=list(z=TRUE))))

fig <- fig %>% 
  layout(scene = list(camera=list(eye = list(x=1.87, y=0.88, z=-0.64))))

fig

This covariance surface for the Brownian motion random walk we are using for data is interesting in its own right. There is a theoretical result that says that the estimate for the covariance function for this process is $\hat{c}(t,s) = min(t,s)$. You can find the proof in the book by Kokoszka and Reimherr referenced below. Using the mouse to hover over some points in the graph makes this result seem plausible.

References

Books

Kokoszka, P. and Reimherr, M. (2017). Introduction to Functional Data Analysis. CRC.
Ramsay, J.O. and Silverman, B.W. (2005). Functional Data Analysis. Springer.
Ramsay, J.0., Hooker, G. and Graves, S. (2009) Functional Data Analysis with R and MATLAB Springer.

Online Resources

Cao, J. (2019). Functional Data Analysis Course
Staicu, A. and Park, Y. (2016) Short Course on Applied Functional Data Analysis

Introduction to Functional Data Analysis with R

Tue, 04 May 2021 00:00:00 +0000

Suppose you have data that looks something like this.

This plot might depict 80 measurements for a participant in a clinical trial where each data point represents the change in the level of some protein level. Or it could represent any series of longitudinal data where the measurements are taken at irregular intervals. The curve looks like a time series with obvious correlations among the points, but there are not enough measurements to model the data with the usual time series methods. In a scenario like this, you might find Functional Data Analysis (FDA) to be a viable alternative to the usual multi-level, mixed model approach.

This post is meant to be a “gentle” introduction to doing FDA with R for someone who is totally new to the subject. I’ll show some “first steps” code, but most of the post will be about providing background and motivation for looking into FDA. I will also point out some of the available resources that a newcommer to FDA should find helpful.

FDA is a branch of statistics that deals with data that can be conceptualized as a function of an underlying, continuous variable. The data in FDA are smooth curves (or surfaces) in time or space. To fix a mental model of this idea, first consider an ordinary time series. For example, you might think of the daily closing prices of your favorite stock. The data that make up a time series are the individual points which are considered to be random draws from an underlying stochastic process.

Now, go up a level of abstraction, and consider a space where the whole time series, or rather an imaginary continuous curve that runs through all of your data points is the basic item of analysis. In this conceptual model, the curve comprises an infinite number of points, not just the few you observed. Moreover, unlike in basic time series analysis, the observed points do not need to be equally spaced, and the various curves that make up your data set do not need to be sampled at the same time points.

Mathematically, the curves are modeled as functions that live in an infinite dimensional vector space, what the mathematicians call a Hilbert Space. One way to think of this is that you are dealing with the ultimate large p small n problem. Each curve has infinitely many points, not just the 3 or 30 or 3,000 you happen to have.

The theory of Hilbert Spaces is part of the area of mathematical analysis called Functional Analysis, a subject usually introduced as part of a second or third course in mathematical analysis, or perhaps in a course on Quantum Mechanics.

It would be a heavy lift to expect someone new to Functional Data Analysis to start with the mathematics. Fortunately, this is not really necessary. The practical applications of FDA and the necessary supporting software have been sufficiently developed so that anyone familiar with the basics of ordinary vector spaces should have sufficient background to get started. The salient points to remember are:

Hilbert space is an infinite dimensional linear vector space
The vectors in Hilbert space are functions
The inner product of two functions in the Hilbert space is defined as the integral of two functions, but it behaves very much like the familiar dot product.

Moreover, for the last twenty years or so mathematical statisticians have been writing R packages to put FDA within the reach of anyone with motivation and minimal R skills. The CRAN Task View on Functional Data Analysis categorizes and provides brief explanations for forty packages that collectively cover most of the established work on FDA. The following graph built with functions from the cranly package shows part of the network for two core FDA packages.

The fda package emphasized in the network plot above is the logical place for an R user to begin investigating FDA. With thirty-two reverse depends, thirty-eight reverse imports and thirteen reverse suggest, fda is at the root of Functional Data Analysis software for R. Moreover, in a very real sense, it is at the root of modern FDA itself. fda was written to explicate the theory developed in the 2005 book by Ramsay and Silverman$^{1}$. Kokoszka and Reimnerr state that the first edition of this book published in 1997: “is largely credited with solidifying FDA as an official subbranch of statistics” (p xiv)$^{2}$. The refund package is used extensively throughout the book by Kokoszka and Reimnerr.

First Steps

The synthetic data in the figure above were generated by a Wiener, Brownian Motion process, which for the purposes of this post, is just a convenient way to generate a variety of reasonable looking curves. We suppose that the data points shown represent noisy observations generated by a smooth curve f(t). We estimate this curve with the model: $y_{i} = f(t_{i}) + \epsilon_{i}$ where the $\epsilon_{i}$ are normally distributed with mean 0 and variance $\sigma^{2}$.

Notice that the measurement times are randomly selected within the 100 day window and not uniformly spaced.

set.seed(999)
n_obs <- 80
time_span <- 100
time <- sort(runif(n_obs,0,time_span))
Wiener <- cumsum(rnorm(n_obs)) / sqrt(n_obs)
y_obs <- Wiener + rnorm(n_obs,0,.05)

Remember that the task ahead is to represent the entire curve of infinitely many points and not just the handful of observed values. Here is where the linear algebra comes in. The curve is treated as a vector in an infinite dimensional vector space, and what we want is something that will serve as a basis for this curve projected down into the subspace where the measurements live. The standard way to do this for non-periodic data is to construct a B-spline basis. (B-splines or basis splines are splines designed to have properties that make them suitable for representing vectors.) The code that follows is mostly borrowed from Jiguo Cao’s Youtube Video Course$^{3}$ which I very highly recommend for anyone just starting with FDA. In his first five videos, Cao explains B-splines and the placement of knots in great detail and derives the formula used in the code to calculates the number of basis elements from the number of knots and the order of the splines.

Note that we are placing the knots at times equally spaced over the 100 day time span.

times_basis = seq(0,time_span,1)
knots    = c(seq(0,time_span,5)) #Location of knots
n_knots   = length(knots) #Number of knots
n_order   = 4 # order of basis functions: cubic bspline: order = 3 + 1
n_basis   = length(knots) + n_order - 2;
basis = create.bspline.basis(c(min(times_basis),max(times_basis)),n_basis,n_order,knots)
n_basis

## [1] 23

and there are 23 basis vectors.

Next, we use the function eval.basis() to evaluate the basis functions at the times where our data curve was observed The matrix PHI contains the values of the 23 basis functions $\phi_j(t)$ evaluated at 80 points.

PHI = eval.basis(time, basis) 
dim(PHI)

## [1] 80 23

We plot the basis functions and locations of the knots.

matplot(time,PHI,type='l',lwd=1,lty=1, xlab='time',ylab='basis',cex.lab=1,cex.axis=1)
for (i in 1:n_knots)
{
  abline(v=knots[i], lty=2, lwd=1)
}

The plot shows that for interior points, four basis functions contribute to computing the value of any point. The endpoints, however, are computed from a single basis function.

Estimating the Basis Coefficients

As in ordinary regression, we express the function in terms of the coefficients $c_j$ and basis functions $\phi_j$ using the formula: $f(t) = \sum c_j \phi_j(t)$. Later we will see how to use built-in fda functions to estimate the coefficients, but now we follow Cao’s lead and calculate everything from first principles.

The following code uses matrix least squares equation $\hat{c} = (\Phi^t\Phi)^{-1} \Phi^{t}y$ to estimate the coefficients.

# Least squares estimate
# estimate basis coefficient
M = ginv(t(PHI) %*% PHI) %*% t(PHI)
c_hat = M %*% Wiener

We compute $\hat{y}$, the estimates of our observed values, and plot.

y_hat = PHI %*% c_hat
# Augment data frame for plotting
df <- df %>% mutate(y_hat = y_hat)
p2 <- df %>% ggplot() + 
      geom_line(aes(x = time, y = Wiener), col = "grey") +
      geom_point(aes(x = time, y = y_obs)) +
      geom_line(aes(x = time, y = y_hat), col = "red")
p2 + ggtitle("Original curve and least squares estimate") + 
      xlab("time") + ylab("f(time)")

The gray curve in the plot represents the underlying Brownian motion process, the dots are the observed values (the same as in the first plot), and the red curve represents the least squares “smoothed” estimates.

Now, we work through the matrix calculations to estimate the variance of the noise and the error bars for $\hat{y}$.

# estimate the variance of noise
## SSE = (Y - Xb)'(Y - Xb)
SSE = t(y_hat-y_obs)%*%(y_hat-y_obs)
sigma2 = SSE/(n_obs-n_basis)

# estimate the variance of the fitted curve
# H is the Hat matrix H
# H = X*inv(X'X)*X``
H = PHI %*% M
varYhat = diag(H %*% H * matrix(sigma2,n_obs,n_obs))

# 95% confidence interval

y_hat025 = y_hat-1.96*sqrt(varYhat)
y_hat975 = y_hat+1.96*sqrt(varYhat)

And, we plot. We have a satisfying smoothed representation of our original curve that looks like it would be and adequate starting point for further study. Note that process of using regression to produce a curve from the basis functions is often referred to as “regression smoothing”

df <- mutate(df, y_hat025 = y_hat025,
                 y_hat975 = y_hat975)
#names(df) <- c("time","Wiener","y_hat", "y_hat025", "y_hat975")
p3 <- df %>% ggplot() + 
      geom_line(aes(x = time, y = Wiener), col = "grey") +
      geom_point(aes(x = time, y = y_obs)) +
      geom_line(aes(x = time, y = y_hat), col = "red") +
      geom_line(aes(x = time, y_hat025), col = "green") +
      geom_line(aes(x = time, y_hat975), col = "green") 
p3 + ggtitle("Estimated curve with error bars") + 
     xlab("time") + ylab("f(time)")

We finish for today, by showing how to do the hard work of estimating coefficients and function values with a single line of code using the fda function smooth.basis(). The function takes the arguments argvals the times we want to use for evaluation as a vector (or matrix or array), y the observed values, and fdParobj, an fda object containing the basis elements.

Wiener_obj <- smooth.basis(argvals = time, y = y_obs, fdParobj = basis)

Here we plot our “hand calculated” curve in red and show the smooth.basis() curve in blue. They are reasonably close, except at the end points, where there is not much data to construct the basis.

plot(time, Wiener, type = "l", xlab = "time", ylab = "f(time)", 
     main = "Comparison of fda package and naive smoothing estimates", col = "grey")
lines(time,y_hat,type = "l",col="red")
lines(Wiener_obj, lwd = 1, col = "blue")

Note that we have shown the simplest use of smooth.basis() which is capable of computing penalized regression estimates and more. The examples of using the smooth.basis() function in the fda pdf are extensive and worth multiple blog posts. In general, the pdf level documentation for fda is superb. However, the package lacks vignettes. For a price, the book Functional Data Analysis with R and Matlab$^{4}$ supplies the equivalent of several the missing vignettes.

Next Steps

Once you have a basis representation, what’s next? You may be interested in the following:

More exploratory work such as Functional Principal Components Analysis, the analog of principal components analysis.
Clustering curves. See the funHDDC package.
Setting up regression models where either the dependent variable, or some of the independent variables, or both are functional objects. See the refund package and the book by Kokoszka and Reimnerr$^{2}$
Studying the shape of the curves themselves. For example, the shape of a protein concentration curve may convey some clinical meaning. FDA permits studying the velocity and acceleration of curves, offering the possibility of obtaining more information than the standard practice of looking at the area under the curves. You can explore this with the fda package (But be sure to check that the order of your basis functions is adequate to compute derivatives.). See the book by Ramsay, Hooker and Graves$^{4}$.
Learning what to do when you have sparse data. See the paper by Yao et al. below$^{5}$ and look into the fdapace package.
Working with two and three dimensional medical images$^{6}$.

I would like to make Functional Data Analysis a regular feature on R Views. If you are working with FDA and would like to post, please let me (joseph.rickert@rstudio.com) know.

References

Books

$^{2}$Kokoszka, P. and Reimherr, M. (2017). Introduction to Functional Data Analysis. CRC.
$^{1}$Ramsay, J.O. and Silverman, B.W. (2005). Functional Data Analysis. Springer.
$^{4}$Ramsay, J.0., Hooker, G. and Graves, S. (2009) Functional Data Analysis with R and MATLAB Springer.

Online Resources

$^{3}$Cao, J. (2019). Functional Data Analysis Course
Staicu, A. and Park, Y. (2016) Short Course on Applied Functional Data Analysis

March 2021: "Top 40" New CRAN Packages

Thu, 22 Apr 2021 00:00:00 +0000

By my count, two hundred twenty-one new packages stuck to CRAN in March 2021.¹ Here are my “Top 40” selections in twelve categories: Computational Methods, Data, Engineering, Genomics, Machine Learning, Medicine, Music, Networks, Science, Statistics, Utility, and Visualization. Two of these categories Engineering and Music have only one entry each. However, I decided to give them their own category in order to draw attention to the use of R outside of the mainstream, and I have always lamented the fate of the Miscellaneous. In the same spirit, note that the complete works of the Bard appear in the Data category and that due to tidypaleo Paleoenvironmental is now a thing in R.

Computational Methods

gamlss v1.0-5: Implements computationally intensive calculations for Generalized Additive Models for location, scale, and shape as described in Rigby & Stasinopoulos (2005).

waydown v1.1.0: Implements an algorithm based on the classical Helmholtz decomposition to obtain an approximate potential function for non gradient fields. See Rodríguez-Sánchez (2020) for background and the vignette for examples.

Data

aopdata v0.2.1: Provides functions to download data from the Access to Opportunities Project (AOP) which includes annual estimates of access to employment, health and education services by transport mode, as well as data on the spatial distribution of population, schools and health-care facilities at a fine spatial resolution for all cities included in the study. There is an Introduction to the package, and there are vignettes on Analyzing Inequality, Mapping Urban Accessibility, and Mapping Pooulation and Land Use.

bardr v0.0.9: Provides R data structures for Shakespeare’s complete works, as provided by Project Gutenberg. See README.

metro v0.9.1: Provides access to the Metro Transparent Data Sets API published by the Washington Metropolitan Area Transit Authority, the government agency operating light rail and passenger buses in the Washington D.C. area. See README.

RAQSAPI v2.0.1: Provides functions to retrieve air monitoring data and associated metadata from the US Environmental Protection Agency’s Air Quality System Service. There are several short vignettes including an Introduction and a vignette on Usage tips and precautions.

troopdata v0.1.3: Provides access to U.S. Department of Defense data on overseas military deployments and includes functions for pulling country-year troop deployment and basing data. See README to get started

Engineering

pipenostics v0.1.7: Implements empirical and data-driven models of heat losses, corrosion diagnostics, reliability and predictive maintenance of pipeline systems which should be of interest to the engineering departments of heat generating and heat transferring companies. See Timashev et al. (2016) and Reddy (2017) for the methods used and README to get started.

Genomics

glmmSeq v0.1.0: Provides functions to fit negative binomial mixed effects models with matched samples to model expression data. See the vignette for examples.

ondisc v1.0.0: Implements a method to allow researchers to analyze large-scale single-cell data as and R object stored on disk. There is a tutorial on the the ondisc matrix class and another on Metadata.

SignacX v2.2.0: Implements a neural network trained with flow-sorted gene expression data to classify cellular phenotypes in single cell RNA-sequencing data. See Chamberlain et al. (2021) for background. There are seven vignettes including an Analysis of Kidney Lupus Data and an Analysis of PBMCs from 10X Genomics.

Machine Learning

opitools v1.0.3: Implements a tool to analyze opinions inherent in a text document relating to a specific subject (A) and assess how opinions expressed with respect to another subject (B) may affect the opinions on subject A. This package has been designed specifically for application to social media datasets, such as Twitter and Facebook. See Adepeju and Jimoh (2021) for an extended example that demonstrates the utility of the approach and the vignette to get started.

poems v1.0.1: Provides a framework of interoperable R6 classes for building ensembles of viable models via the pattern-oriented modeling (POM) approach. The package includes classes for encapsulating and generating model parameters, and managing the POM workflow which includes: model setup; generating model parameters via Latin hyper-cube sampling; running multiple sampled model simulations; collating summary results; and validating and selecting an ensemble of models that best match known patterns. There are two vignettes: Simple Example and Thylacine Example.

Medicine

dampack v1.0.0: Implements a suite of functions for analyzing and visualizing the health economic outputs of mathematical models. See Hunink et al. (2014) for the theoretical underpinnings. There are five vignettes including Basic Cost Effectiveness Analysis, Probabilistic Sensitivity Analysis: Analysis and Value of Information Analysis.

rdecision v1.0.3: Provides classes and functions for using decision trees to model health care interventions using cohort models. See Briggs et al. for theory and terminology. There are five vignettes including Elementary decision tree (Evans 1997) and Decision tree with PSA.

Music

gm v1.0.2: Implements a high-level language to create music including converting your music to musical scores and audio files. It works with R Markdown, R Jupyter Notebooks, and RStudio. There vignette is available in English and in Chinese.

Networks

sfnetworks v0.5.1: Provides a tidy approach to spatial network analysis in the form of classes and functions that enable a seamless interaction between the network analysis package tidygraph and the spatial analysis package sf. There are vignettes on sf network structure, Preprocessing, Spatial joins and filters, Routing, and Spatial morphers.

valhallr v0.1.0: Implements an interface to the Valhalla routing engine’s API for turn-by-turn routing, isochrones, and origin-destination analyses. See the vignette for examples.

Science

asteRisk v0.99.4: Provides functions to calculate the positions of satellites given a known state vector. It includes implementations of the SGP4 and SDP4 simplified perturbation models to propagate orbital state vectors. See Hoots et al. (1988), Vallado et al. (2012), and Hoots et al. (2014) for background and the vignette for examples.

forImage v0.1.0: Implements a tool to measure the size of foraminifera and other unicellulars and includes functions to guide foraminiferal test biovolume calculations and cell biomass estimations. The volume function includes several microalgae models geometric adaptations based on Hillebrand et al. (1999), Sun & Liu (2003), and Vadrucci et al. (2007). See the vignette to get started.

OpenSpecy v0.9.1: Provides functions to analyze, process, identify and share Raman and (FT)IR spectra with functions to implement Savitzky-Golay smoothing in accordance with Zhao et al. (2007) and identify spectra using an onboard reference library, see Cowger et al. 2020. Analyzed spectra can be shared via Shiny App. There is a vignette.

tidypaleo v0.1.1: Provides functions with a common framework for age-depth model management, stratigraphic visualization, and common statistical transformations with a focus on stratigraphic visualization using ggplot2. There are vignettes on Age-depth Models, Nested Analyses, and Stratigraphic Diagrams.

VulnToolkit v1.1.2: Provides functions to analyze and summarize tidal data sets and to access to NOAA mean sea level data. See Hill & Anisfeld (2015) for background and the vignette for examples.

Statistics

corncob v0.2.0: Implements functions for modeling correlated count data using the beta-binomial distribution, described in Martin et al. (2020). See the vignette for an introduction.

hawkesbow v1.0.2: Implements an estimation method for Hawkes processes when count data are only observed in discrete time, using a spectral approach derived from the Bartlett spectrum. See Cheysson and Lang (2020) for background and the vignette for examples.

LMMELSM v0.1.0: Implements two-level mixed effects location scale models on multiple observed or latent outcomes, and between-group variance modeling. See Williams et al. (2020) and Hedeker et al. (2008) for background and README for an example.

mixpoissinreg v1.0.0: Provides functions to fit mixed Poisson regression models (Poisson-Inverse Gaussian or Negative-Binomial) with count data response variables. See Barreto-Souza and Simas (2016) for background. There are five vignettes on Global and Local Influence, Confidence and Prediction Intervals, MLE, Tidy Methods, and Overdispersed Count Data.

ppdiag v0.1.0: Provides a suite of diagnostic tools for univariate point processes including tools for simulating and fitting both common and more complex temporal point processes and the diagnostic tools described in Brown et al. (2002) and Wu et al. (2020). There is a vignette on Markov Modulated Point Processes and another on Diagnostic Tools.

robustlm v0.1.0: Implements a computationally efficient exponential squared loss algorithm for variable selection proposed by Wang et al.(2013). See the vignette.

smmR v1.0.2: Provides functions to estimate and simulate multi-state semi-Markov models. The methods implemented are described in Barbu & Limnios (2008) and Trevezas & Limnios (2011). The vignette contains an extended example.

spotoroo v0.1.1: Implements an algorithm to cluster satellite hot spot data spatially and temporally. See the vignette.

Utilities

clock v0.2.0: Provides a comprehensive library for date-time manipulations using a new family of orthogonal date-time classes (duration, time points, zoned-times, and calendars) that partition responsibilities so that the complexities of time zones are only considered when they are really needed. There is a Getting Started guide, as well as vignettes on FAQ, and Examples and Recipies.

crosstable v0.2.1: Provides functions to create descriptive tables for continuous and categorical variables, apply summary statistics, and create reports using rmarkdown or officer. There is an Introduction, and vignettes on Troubleshooting, Making Automatic Reports, and Selecting Variables.

pkgdepends v0.1.0: Provides functions to find recursive dependencies for R packages from various sources including CRAN, Bioconductor, and GitHub enabling users to obtain a consistent set of packages to install. See README to get started.

pkglite v0.1.1: Implements a tool, grammar, and standard to represent and exchange R package source code as text files. Converts one or more source packages to a text file and restores the package structures from the file. There are vignettes on Generating File Specifications, Representing Packages, and Compact Package Representation.

Visualization

datplot v1.0.0: Provides tools to process and prepare data for visualization and employs the concept of aoristic analysis. See aorist and the vignettes Data Preparation and Visualization and Visualizing Chronological Distribution.

ferrn v0.0.1: Implements diagnostic plots for optimization, with a focus on projection pursuit which show paths the optimizer takes in the high-dimensional space. See README for examples.

funcharts v1.0.0: Provides functional control charts for statistical process monitoring of functional data, using the methods of Capezza et al. (2020) and Centofanti et al. (2020). There are vignettes on Capezza 2020, Centofanti 2020 and on the mfd class.

gghilbertstrings v0.3.3: Provides functions to plot Hilbert curves which are used to map one dimensional data into the 2D plane. A specific use case maps a character column in a data frame into 2D space allowing visually comparing long lists of URLs, words, genes or other data that has a fixed order and position. See README for examples.

mapsf v0.1.1: Provides functions to create and integrate thematic maps including functions to design various cartographic representations such as proportional symbols, choropleth or typology maps. Look here for examples.

¹ I have used phrases like By my count and stuck to CRAN in the past, but I do not believe that I have explained what I mean. For some time now, but I believe more frequently in recent months, packages will appear as new on CRAN, only to be removed within a relatively short period of time for failing to resolve check problems. If you happen to know about these packages and search for them by name on CRAN you will receive the message:

Package XXXX was removed from the CRAN repository. Formerly available versions can be obtained from the archive. Archived on 2021-04-17 as check problems remained after update. A summary of the most recent check results can be obtained from the check results archive. Please use the canonical form https://CRAN.R-project.org/package=XXXX to link to this page.

I did not include the ten packages that were identified as being new for March when I created my list of March packages on April 10, 2021, but were removed by the time I finalized my list for this post a week later, in my total count of new CRAN packages. So, there is some instability with the notion of counting new packages in a given month.

An Alternative to the Correlation Coefficient That Works For Numeric and Categorical Variables

Thu, 15 Apr 2021 00:00:00 +0000

Dr. Rama Ramakrishnan is Professor of the Practice at MIT Sloan School of Management where he teaches courses in Data Science, Optimization and applied Machine Learning.

When starting to work with a new dataset, it is useful to quickly pinpoint which pairs of variables appear to be strongly related. It helps you spot data issues, make better modeling decisions, and ultimately arrive at better answers.

The correlation coefficient is used widely for this purpose, but it is well-known that it can’t detect non-linear relationships. Take a look at this scatterplot of two variables $x$ and $y$.

set.seed(42)
x <- seq(-1,1,0.01)
y <- sqrt(1 - x^2) + rnorm(length(x),mean = 0, sd = 0.05)

ggplot(mapping = aes(x, y)) +
  geom_point()

It is obvious to the human eye that $x$ and $y$ have a strong relationship but the correlation coefficient between $x$ and $y$ is only -0.01.

Further, if either variable of the pair is categorical, we can’t use the correlation coefficient. We will have to turn to other metrics. If $x$ and $y$ are both categorical, we can try Cramer’s V or the phi coefficient. If $x$ is continuous and $y$ is binary, we can use the point-biserial correlation coefficient.

But using different metrics is problematic. Since they are derived from different assumptions, we can’t compare the resulting numbers with one another. If the correlation coefficient between continuous variables $x$ and $y$ is 0.6 and the phi coefficient between categorical variables $u$ and $v$ is also 0.6, can we safely conclude that the relationships are equally strong? According to Wikipedia,

The correlation coefficient ranges from −1 to +1, where ±1 indicates perfect agreement or disagreement, and 0 indicates no relationship. The phi coefficient has a maximum value that is determined by the distribution of the two variables if one or both variables can take on more than two values.

A phi coefficient value of 0.6 between $u$ and $v$ may not mean much if its maximum possible value in this particular situation is much higher. Perhaps we can normalize the phi coefficient to map it to the 0-1 range? But what if that modification introduces biases?

Wouldn’t it be nice if we had one uniform approach that was easy to understand, worked for continuous and categorical variables alike, and could detect linear and nonlinear relationships?

(BTW, when $x$ and $y$ are continuous, looking at a scatter plot of $x$ vs $y$ can be very effective since the human brain can detect linear and non-linear patterns very quickly. But even if you are lucky and all your variables are continuous, looking at scatterplots of all pairs of variables is hard when you have lots of variables in your dataset; with just 100 predictors (say), you will need to look through 4950 scatterplots and this obviously isn’t practical)

A Potential Solution

To devise a metric that satisfies the requirements we listed above, let’s invert the problem: What does it mean to say that $x$ and $y$ don’t have a strong relationship?

Intuitively, if there’s no relationship between $x$ and $y$, we would expect to see no patterns in a scatterplot of $x$ and $y$ - no lines, curves, groups etc. It will be a cloud of points that appears to be randomly scattered, perhaps something like this:

x <- seq(-1,1,0.01)
y <- runif(length(x),min = -1, max = 1)

ggplot(mapping = aes(x, y)) +
  geom_point()

In this situation, does knowing the value of $x$ give us any information on $y$?

Clearly not. $y$ seems to be somewhere between -1 and 1 with no particular pattern, regardless of the value of $x$. Knowing $x$ does not seem to help reduce our uncertainty about the value of $y$.

In contrast, look at the first picture again.

Here, knowing the value of $x$ does help. If we know that $x$ is around 0.0, for example, from the graph we will guess that $y$ is likely near 1.0 (the red dots). We can be confident that $y$ is not between 0 and 0.8. Knowing $x$ helps us eliminate certain values of $y$, reducing our uncertainty about the values $y$ might take.

This notion - that knowing something reduces our uncertainty about something else - is exactly the idea behind mutual information from Information Theory.

According to Wikipedia (emphasis mine),

Intuitively, mutual information measures the information that $X$ and $Y$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if $X$ and $Y$ are independent, then knowing $X$ does not give any information about $Y$ and vice versa, so their mutual information is zero.

Furthermore,

Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair $(X,Y)$ is to the product of the marginal distributions of $X$ and $Y$.

This is very promising!

As it turns out, however, implementing mutual information is not so simple. We first need to estimate the joint probabilities (i.e., the joint probability density/mass function) of $x$ and $y$ before we can calculate their Mutual Information. If $x$ and $y$ are categorical, this is easy but if one or both of them is continuous, it is more involved.

But we can use the basic insight behind mutual information – that knowing $x$ may reduce our uncertainty about $y$ – in a different way.

The X2Y Metric

Consider three variables $x$, $y$ and $z$. If knowing $x$ reduces our uncertainty about $y$ by 70% but knowing $z$ reduces our uncertainty about $y$ by only 40%, we will intuitively expect that the association between $x$ and $y$ will be stronger than the association between $z$ and $y$.

So, if we can quantify the reduction in uncertainty, that can be used as a measure of the strength of the association. One way to do so is to measure $x$’s ability to predict $y$ - after all, if $x$ reduces our uncertainty about $y$, knowing $x$ should help us predict $y$ better than if we didn’t know $x$.

Stated another way, we can think of reduction in prediction error $\approx$ reduction in uncertainty $\approx$ strength of association.

This suggests the following approach:

Predict $y$ without using $x$.
- If $y$ is continuous, we can simply use the average value of $y$.
- If $y$ is categorical, we can use the most frequent value of $y$.
- These are sometimes referred to as a baseline model.
Predict $y$ using $x$
- We can take any of the standard predictive models out there (Linear/Logistic Regression, CART, Random Forests, SVMs, Neural Networks, Gradient Boosting etc.), set $x$ as the independent variable and $y$ as the dependent variable, fit the model to the data, and make predictions. More on this below.
Calculate the % decrease in prediction error when we go from (1) to (2)
- If $y$ is continuous, we can use any of the familiar error metrics like RMSE, SSE, MAE etc. I prefer mean absolute error (MAE) since it is less susceptible to outliers and is in the same units as $y$ but this is a matter of personal preference.
- If $y$ is categorical, we can use Misclassification Error (= 1 - Accuracy) as the error metric.

In summary, the % reduction in error when we go from a baseline model to a predictive model measures the strength of the relationship between $x$ and $y$. We will call this metric x2y since it measures the ability of $x$ to predict $y$.

(This definition is similar to R-Squared from Linear Regression. In fact, if $y$ is continuous and we use the Sum of Squared Errors as our error metric, the x2y metric is equal to R-Squared.)

To implement (2) above, we need to pick a predictive model to use. Let’s remind ourselves of what the requirements are:

If there’s a non-linear relationship between $x$ and $y$, the model should be able to detect it
It should be able to handle all possible $x$-$y$ variable types: continuous-continuous, continuous-categorical, categorical-continuous and categorical-categorical
We may have hundreds (if not thousands) of pairs of variables we want to analyze so we want this to be quick

Classification and Regression Trees (CART) satisfies these requirements very nicely and that’s the one I prefer to use. That said, you can certainly use other models if you like.

Let’s try this approach on the ‘semicircle’ dataset from above. We use CART to predict $y$ using $x$ and here’s how the fitted values look:

# Let's generate the data again
set.seed(42)
x <- seq(-1,1,0.01)

d <- data.frame(x = x, 
                    y = sqrt(1 - x^2) + rnorm(length(x),mean = 0, sd = 0.05))

library(rpart)
preds <- predict(rpart(y~x, data = d, method = "anova"), type = "vector")

# Set up a chart
ggplot(data = d, mapping = aes(x = x)) +
  geom_point(aes(y = y), size = 0.5) +
  geom_line(aes(y=preds, color = '2')) +
  scale_color_brewer(name = "", labels='CART', palette="Set1")

Visually, the CART predictions seem to approximate the semi-circular relationship between $x$ and $y$. To confirm, let’s calculate the x2y metric step by step.

The MAE from using the average of $y$ to predict $y$ is 0.19.
The MAE from using the CART predictions to predict $y$ is 0.06.
The % reduction in MAE is 68.88%.

Excellent!

If you are familiar with CART models, it is straightforward to implement the x2y metric in the Machine Learning environment of your choice. An R implementation is here and details can be found in the appendix but, for now, I want to highlight two functions from the R script that we will use in the examples below:

x2y(u, v) calculates the x2y metric between two vectors $u$ and $v$
dx2y(d) calculates the x2y metric between all pairs of variables in a dataframe $d$

Two Caveats

Before we demonstrate the x2y metric on a couple of datasets, I want to highlight two aspects of the x2y approach.

Unlike metrics like the correlation coefficient, the x2y metric is not symmetric with respect to $x$ and $y$. The extent to which $x$ can predict $y$ can be different from the extent to which $y$ can predict $x$. For the semi-circle dataset, x2y(x,y) is 68.88% but x2y(y,x) is only 10.2%.

This shouldn’t come as a surprise, however. Let’s look at the scatterplot again but with the axes reversed.

ggplot(data = d, mapping = aes(x = y)) +
  geom_point(aes(y = x), size = 0.5)  +
  geom_point(data = d[abs(d$x) < 0.05,], aes(x = y, y = x), color = "orange" ) +
  geom_point(data = d[abs(d$y-0.6) < 0.05,], aes(x = y, y = x), color = "red" )

When $x$ is around 0.0, for instance, $y$ is near 1.0 (the orange dots). But when $y$ is around 0.6, $x$ can be in the (-0.75, - 1.0) range or in the (0.5, 0.75) range (the red dots). Knowing $x$ reduces the uncertainty about the value of $y$ a lot more than knowing $y$ reduces the uncertainty about the value of $x$.

But there’s an easy solution if you must have a symmetric metric for your application: just take the average of x2y(x,y) and x2y(y,x).

The second aspect worth highlighting is about the comparability of the x2y metric across variable pairs. All x2y values where the $y$ variable is continuous will be measuring a % reduction in MAE. All x2y values where the $y$ variable is categorical will be measuring a % reduction in Misclassification Error. Is a 30% reduction in MAE equal to a 30% reduction in Misclassification Error? It is problem dependent, there’s no universal right answer.

On the other hand, since (1) all x2y values are on the same 0-100% scale (2) are conceptually measuring the same thing, i.e., reduction in prediction error and (3) our objective is to quickly scan and identify strongly-related pairs (rather than conduct an in-depth investigation), the x2y approach may be adequate.

Application to the Iris Dataset

The iris flower dataset is iconic in the statistics/ML communities and is widely used to illustrate basic concepts. The dataset consists of 150 observations in total and each observation has four continuous variables - the length and the width of petals and sepals - and a categorical variable indicating the species of iris.

Let’s take a look at 10 randomly chosen rows.

iris %>% sample_n(10) %>% pander

Sepal.Length	Sepal.Width	Petal.Length	Petal.Width	Species
5.9	3	5.1	1.8	virginica
5.5	2.6	4.4	1.2	versicolor
6.1	2.8	4	1.3	versicolor
5.9	3.2	4.8	1.8	versicolor
7.7	2.6	6.9	2.3	virginica
5.7	4.4	1.5	0.4	setosa
6.5	3	5.2	2	virginica
5.2	2.7	3.9	1.4	versicolor
5.6	2.7	4.2	1.3	versicolor
7.2	3.2	6	1.8	virginica

We can calculate the x2y values for all pairs of variables in iris by running dx2y(iris) in R (details of how to use the dx2y() function are in the appendix).

dx2y(iris) %>% pander

x	y	perc_of_obs	x2y
Petal.Width	Species	100	94
Petal.Length	Species	100	93
Petal.Width	Petal.Length	100	80.73
Species	Petal.Length	100	79.72
Petal.Length	Petal.Width	100	77.32
Species	Petal.Width	100	76.31
Sepal.Length	Petal.Length	100	66.88
Sepal.Length	Species	100	62
Petal.Length	Sepal.Length	100	60.98
Sepal.Length	Petal.Width	100	54.36
Petal.Width	Sepal.Length	100	48.81
Species	Sepal.Length	100	42.08
Sepal.Width	Species	100	39
Petal.Width	Sepal.Width	100	31.75
Petal.Length	Sepal.Width	100	30
Sepal.Width	Petal.Length	100	28.16
Sepal.Width	Petal.Width	100	23.02
Species	Sepal.Width	100	22.37
Sepal.Length	Sepal.Width	100	18.22
Sepal.Width	Sepal.Length	100	12.18

The first two columns in the output are self-explanatory. The third column - perc_of_obs - is the % of observations in the dataset that was used to calculate that row’s x2y value. When a dataset has missing values, only observations that have values present for both $x$ and $y$ will be used to calculate the x2y metrics for that variable pair. The iris dataset has no missing values so this value is 100% for all rows. The fourth column is the value of the x2y metric and the results are sorted in descending order of this value.

Looking at the numbers, both Petal.Length and Petal.Width seem to be highly associated with Species (and with each other). In contrast, it appears that Sepal.Length and Sepal.Width are very weakly associated with each other.

Note that even though Species is categorical and the other four variables are continuous, we could simply “drop” the iris dataframe into the dx2y() function and calculate the associations between all the variables.

Application to a COVID-19 Dataset

Next, we examine a COVID-19 dataset that was downloaded from the COVID-19 Clinical Data Repository in April 2020. This dataset contains clinical characteristics and COVID-19 test outcomes for 352 patients. Since it has a good mix of continuous and categorical variables, having something like the x2y metric that can work for any type of variable pair is convenient.

Let’s read in the data and take a quick look at the columns.

df <- read.csv("covid19.csv", stringsAsFactors = FALSE)
str(df)

## 'data.frame':    352 obs. of  45 variables:
##  $ date_published               : chr  "2020-04-14" "2020-04-14" "2020-04-14" "2020-04-14" ...
##  $ clinic_state                 : chr  "CA" "CA" "CA" "CA" ...
##  $ test_name                    : chr  "Rapid COVID-19 Test" "Rapid COVID-19 Test" "Rapid COVID-19 Test" "Rapid COVID-19 Test" ...
##  $ swab_type                    : chr  "" "Nasopharyngeal" "Nasal" "" ...
##  $ covid_19_test_results        : chr  "Negative" "Negative" "Negative" "Negative" ...
##  $ age                          : int  30 77 49 42 37 23 71 28 55 51 ...
##  $ high_risk_exposure_occupation: logi  TRUE NA NA FALSE TRUE FALSE ...
##  $ high_risk_interactions       : logi  FALSE NA NA FALSE TRUE TRUE ...
##  $ diabetes                     : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ chd                          : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ htn                          : logi  FALSE TRUE FALSE TRUE FALSE FALSE ...
##  $ cancer                       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ asthma                       : logi  TRUE TRUE FALSE TRUE FALSE FALSE ...
##  $ copd                         : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ autoimmune_dis               : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ temperature                  : num  37.1 36.8 37 36.9 37.3 ...
##  $ pulse                        : int  84 96 79 108 74 110 78 NA 97 66 ...
##  $ sys                          : int  117 128 120 156 126 134 144 NA 160 98 ...
##  $ dia                          : int  69 73 80 89 67 79 85 NA 97 65 ...
##  $ rr                           : int  NA 16 18 14 16 16 15 NA 16 16 ...
##  $ sats                         : int  99 97 100 NA 99 98 96 97 99 100 ...
##  $ rapid_flu                    : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ rapid_flu_results            : chr  "" "" "" "" ...
##  $ rapid_strep                  : logi  FALSE TRUE FALSE FALSE FALSE TRUE ...
##  $ rapid_strep_results          : chr  "" "Negative" "" "" ...
##  $ ctab                         : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
##  $ labored_respiration          : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ rhonchi                      : logi  FALSE FALSE FALSE TRUE FALSE FALSE ...
##  $ wheezes                      : logi  FALSE FALSE FALSE TRUE FALSE FALSE ...
##  $ cough                        : logi  FALSE NA TRUE TRUE TRUE TRUE ...
##  $ cough_severity               : chr  "" "" "" "Mild" ...
##  $ fever                        : logi  NA NA NA FALSE FALSE TRUE ...
##  $ sob                          : logi  FALSE NA FALSE FALSE TRUE TRUE ...
##  $ sob_severity                 : chr  "" "" "" "" ...
##  $ diarrhea                     : logi  NA NA NA TRUE NA NA ...
##  $ fatigue                      : logi  NA NA NA NA TRUE TRUE ...
##  $ headache                     : logi  NA NA NA NA TRUE TRUE ...
##  $ loss_of_smell                : logi  NA NA NA NA NA NA ...
##  $ loss_of_taste                : logi  NA NA NA NA NA NA ...
##  $ runny_nose                   : logi  NA NA NA NA NA TRUE ...
##  $ muscle_sore                  : logi  NA NA NA TRUE NA TRUE ...
##  $ sore_throat                  : logi  TRUE NA NA NA NA TRUE ...
##  $ cxr_findings                 : chr  "" "" "" "" ...
##  $ cxr_impression               : chr  "" "" "" "" ...
##  $ cxr_link                     : chr  "" "" "" "" ...

#%>% pander

There are lots of missing values (denoted by ‘NA’) and lots of blanks as well - for example, see the first few values of the rapid_flu_results field above. We will convert the blanks to NAs so that all the missing values can be treated consistently. Also, the rightmost three columns are free-text fields so we will remove them from the dataframe.

df <- read.csv("covid19.csv", 
                  stringsAsFactors = FALSE,
                  na.strings=c("","NA") # read in blanks as NAs
                  )%>% 
  select(-starts_with("cxr"))  # remove the chest x-ray note fields

str(df)

## 'data.frame':    352 obs. of  42 variables:
##  $ date_published               : chr  "2020-04-14" "2020-04-14" "2020-04-14" "2020-04-14" ...
##  $ clinic_state                 : chr  "CA" "CA" "CA" "CA" ...
##  $ test_name                    : chr  "Rapid COVID-19 Test" "Rapid COVID-19 Test" "Rapid COVID-19 Test" "Rapid COVID-19 Test" ...
##  $ swab_type                    : chr  NA "Nasopharyngeal" "Nasal" NA ...
##  $ covid_19_test_results        : chr  "Negative" "Negative" "Negative" "Negative" ...
##  $ age                          : int  30 77 49 42 37 23 71 28 55 51 ...
##  $ high_risk_exposure_occupation: logi  TRUE NA NA FALSE TRUE FALSE ...
##  $ high_risk_interactions       : logi  FALSE NA NA FALSE TRUE TRUE ...
##  $ diabetes                     : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ chd                          : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ htn                          : logi  FALSE TRUE FALSE TRUE FALSE FALSE ...
##  $ cancer                       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ asthma                       : logi  TRUE TRUE FALSE TRUE FALSE FALSE ...
##  $ copd                         : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ autoimmune_dis               : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ temperature                  : num  37.1 36.8 37 36.9 37.3 ...
##  $ pulse                        : int  84 96 79 108 74 110 78 NA 97 66 ...
##  $ sys                          : int  117 128 120 156 126 134 144 NA 160 98 ...
##  $ dia                          : int  69 73 80 89 67 79 85 NA 97 65 ...
##  $ rr                           : int  NA 16 18 14 16 16 15 NA 16 16 ...
##  $ sats                         : int  99 97 100 NA 99 98 96 97 99 100 ...
##  $ rapid_flu                    : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ rapid_flu_results            : chr  NA NA NA NA ...
##  $ rapid_strep                  : logi  FALSE TRUE FALSE FALSE FALSE TRUE ...
##  $ rapid_strep_results          : chr  NA "Negative" NA NA ...
##  $ ctab                         : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
##  $ labored_respiration          : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ rhonchi                      : logi  FALSE FALSE FALSE TRUE FALSE FALSE ...
##  $ wheezes                      : logi  FALSE FALSE FALSE TRUE FALSE FALSE ...
##  $ cough                        : logi  FALSE NA TRUE TRUE TRUE TRUE ...
##  $ cough_severity               : chr  NA NA NA "Mild" ...
##  $ fever                        : logi  NA NA NA FALSE FALSE TRUE ...
##  $ sob                          : logi  FALSE NA FALSE FALSE TRUE TRUE ...
##  $ sob_severity                 : chr  NA NA NA NA ...
##  $ diarrhea                     : logi  NA NA NA TRUE NA NA ...
##  $ fatigue                      : logi  NA NA NA NA TRUE TRUE ...
##  $ headache                     : logi  NA NA NA NA TRUE TRUE ...
##  $ loss_of_smell                : logi  NA NA NA NA NA NA ...
##  $ loss_of_taste                : logi  NA NA NA NA NA NA ...
##  $ runny_nose                   : logi  NA NA NA NA NA TRUE ...
##  $ muscle_sore                  : logi  NA NA NA TRUE NA TRUE ...
##  $ sore_throat                  : logi  TRUE NA NA NA NA TRUE ...

#%>% pander

Now, let’s run it through the x2y approach. We are particularly interested in non-zero associations between the covid_19_test_results field and the other fields so we zero in on those by running dx2y(df, target = "covid_19_test_results") in R (details in the appendix) and filtering out the zero associations.

dx2y(df, target = "covid_19_test_results") %>% 
  filter(x2y >0) %>% 
  pander

x	y	perc_of_obs	x2y
covid_19_test_results	loss_of_smell	21.88	18.18
covid_19_test_results	loss_of_taste	22.73	12.5
covid_19_test_results	sats	92.9	2.24

Only three of the 41 variables have a non-zero association with covid_19_test_results. Disappointingly, the highest x2y value is an unimpressive 18%. It is based on just 22% of the observations (since the other 78% of observations had missing values) and makes one wonder if this modest association is real or if it is just due to chance.

If we were working with the correlation coefficient, we could easily calculate a confidence interval for it and gauge if what we are seeing is real or not. Can we do the same thing for the x2y metric?

We can, by using bootstrapping. Given $x$ and $y$, we can sample with replacement a 1000 times (say) and calculate the x2y metric each time. With these 1000 numbers, we can construct a confidence interval easily (this is available as an optional confidence argument in the R functions we have been using; please see the appendix).

Let’s re-do the earlier calculation with “confidence intervals” turned on by running dx2y(df, target = "covid_19_test_results", confidence = TRUE) in R.

dx2y(df, target = "covid_19_test_results", confidence = TRUE) %>% 
  filter(x2y >0) %>% 
  pander(split.tables = Inf)

x	y	perc_of_obs	x2y	CI_95_Lower	CI_95_Upper
covid_19_test_results	loss_of_smell	21.88	18.18	-8.08	36.36
covid_19_test_results	loss_of_taste	22.73	12.5	-11.67	25
covid_19_test_results	sats	92.9	2.24	-1.85	4.48

The 95% confidence intervals all contain 0.0, so none of these associations appear to be real.

Let’s see what the top 10 associations are, between any pair of variables.

dx2y(df) %>%head(10) %>% pander

x	y	perc_of_obs	x2y
loss_of_smell	loss_of_taste	20.17	100
loss_of_taste	loss_of_smell	20.17	100
fatigue	headache	40.06	90.91
headache	fatigue	40.06	90.91
fatigue	sore_throat	27.84	89.58
headache	sore_throat	30.4	89.36
sore_throat	fatigue	27.84	88.89
sore_throat	headache	30.4	88.64
runny_nose	fatigue	25.57	84.44
runny_nose	headache	25.57	84.09

Interesting. loss_of_smell and loss_of_taste are perfectly associated with each other. Let’s look at the raw data.

with(df, table(loss_of_smell, loss_of_taste))

##              loss_of_taste
## loss_of_smell FALSE TRUE
##         FALSE    55    0
##         TRUE      0   16

They agree for every observation in the dataset and, as a result, their x2y is 100%.

Moving down the x2y ranking, we see a number of variables - fatigue, headache, sore_throat, and runny_nose - that are all strongly associated with each other, as if they are all connected by a common cause.

When the number of variable combinations is high and there are lots of missing values, it can be helpful to scatterplot x2y vs perc_of_obs.

ggplot(data = dx2y(df), aes(y=x2y, x = perc_of_obs)) +
         geom_point()

## Warning: Removed 364 rows containing missing values (geom_point).

Unfortunately, the top-right quadrant is empty: there are no strongly-related variable pairs that are based on at least 50% of the observations. There are some variable pairs with x2y values > 75% but none of them are based on more than 40% of the observations.

Conclusion

Using an insight from Information Theory, we devised a new metric - the x2y metric - that quantifies the strength of the association between pairs of variables.

The x2y metric has several advantages:

It works for all types of variable pairs (continuous-continuous, continuous-categorical, categorical-continuous and categorical-categorical)
It captures linear and non-linear relationships
Perhaps best of all, it is easy to understand and use.

I hope you give it a try in your work.

(If you found this note helpful, you may find these of interest)

Acknowledgements

Thanks to Amr Farahat for helpful feedback on an earlier draft.

Appendix: How to use the R script

The R script depends on two R packages - rpart and dplyr - so please ensure that they are installed in your environment.

The script has two key functions: x2y() and dx2y().

Using the `x2y()` function

Usage: x2y(u, v, confidence = FALSE)

Arguments:

u, v: two vectors of equal length
confidence: (OPTIONAL) a boolean that indicates if a confidence interval is needed. Default is FALSE.

Value: A list with the following elements:

perc_of_obs: the % of total observations that were used to calculate x2y. If some observations are missing for either $u$ or $v$, this will be less than 100%.
x2y: the x2y metric for using $u$ to predict $v$

Additionally, if x2y() was called with confidence = TRUE:

CI_95_Lower: the lower end of a 95% confidence interval for the x2y metric estimated by bootstrapping 1000 samples
CI_95_Upper: the upper end of a 95% confidence interval for the x2y metric estimated by bootstrapping 1000 samples

Using the `dx2y()` function

Usage: dx2y(d, target = NA, confidence = FALSE)

Arguments:

d: a dataframe
target: (OPTIONAL) if you are only interested in the x2y values between a particular variable in d and all other variables, set target equal to the name of the variable you are interested in. Default is NA.
confidence: (OPTIONAL) a boolean that indicates if a confidence interval is needed. Default is FALSE.

Value: A dataframe with each row containing the output of running x2y(u, v, confidence) for u and v chosen from the dataframe. Since this is just a standard R dataframe, it can be sliced, sorted, filtered, plotted etc.

Update on April 16, 2021: I learned from a commenter that a similar approach was proposed in April 2020, and that the R package ppsr which implements that approach is now available on CRAN.

What does it take to do a t-test?

Mon, 29 Mar 2021 00:00:00 +0000

In this post, I examine the fundamental assumption of independence underlying the basic Independent two-sample t-test for comparing the means of two random samples. In addition to independence, we assume that both samples are draws from normal distributions where the population means and common variance are unknown. I am going to assume that you are familiar with this kind of test, but even if you are not you are still in the right place. The references at the end of the post all provide rigorous, but gentle explanations that should be very helpful.

The two sample t-test

Typically, we have independent samples for some numeric variable of interest (say the concentration of a drug in the blood stream) from two different groups, and we would like to know whether it is likely that two groups differ with respect to this variable. The formal test of the null hypothesis, $H_0$, that the means of the underlying populations from which the samples are drawn are equal, proceeds making some assumptions:

$H_0$ is true
The samples are independent
The data are normally distributed
The variances of the two samples are equal (This is the simplest test.)

Next, a test statistic that includes the difference between the two sample means is calculated, and a decision is made to establish a “rejection region” for the test statistic. This region depends on the particular circumstances of the test, and is selected to balance the error of rejecting $H_0$ when it is true against the error of not rejecting $H_0$ when it is false. If we compute the test statistic and its value does not fall in the rejection region, then we do not reject $H_0$ and we conclude that we have found nothing. On the other hand, if the test statistic does fall in the rejection region, then we reject the $H_0$ and conclude that our data along with the the bundle of assumptions we made in setting up the test, and the “steel trap” logic of the t-test itself provide some evidence that the population means are different. (Page 6 of the MIT Open Courseware notes Null Hypothesis Significance Testing II contains an elegantly concise mathematical description of the t-test.)

All of the above assumptions must hold, or be pretty close to holding for the test to give an accurate result. However in my opinion, from the point of view of statistical practice, assumption 2. is fundamental. There are other tests and workarounds for the situations where 4. doesn’t hold. Assumption 3. is very important, but it is relatively easy to check, and the t-test is robust enough to deal with some deviation from normality. Of course, assumption 1. is important. The whole test depends on it, but this assumption is baked into the software that will run the test.

Independence

Independence, on the other hand can be a show stopper. Checking for independence is the difference between doing statistics and carrying out a mathematical or maybe just a mechanical exercise. It often involves considerable creative thinking and tedious legwork.

So, what do we mean by independent samples or independent data, and how do we go about verifying it? Independence is a mathematical idea, an abstraction from probability theory. Two events A and B are said to be independent events if the probability of both A and B happening equals the product of the probabilities of A and B happening. That is: P(AB) = P(A)P(B).

A more intuitive way to think about it is in terms of conditionally probability. In general, the probability of A happening given that B happens is defined to be:

P(A|B) = P($A\bigcap B$) / P(B)

If A and B are independent then P(A|B) = P(A). That is: B has no influence on whether A happens.

“Independent data” or “independent samples” are both shorthand for data sampled or otherwise resulting from independent probability distributions. Relating the mathematical concept to a real world situation requires a clear idea of the population of interest, considerable domain expertise, and a mental slight of hand that is nicely exposed in the short article What are independent samples?, by the Minitab® folks. They write:

Independent samples are samples that are selected randomly so that its observations do not depend on the values other observations.

Notice what is happening here: what started out as a property of probability distributions has now become a prescription for obtaining data in a way that makes it plausible that we can assume independence for the probability distributions that we imagine govern our data. This is a real magic trick. No procedure for selecting data is ever going to guarantee the mathematical properties of our models. Nevertheless, the statement does show the way to proceed. By systematically tracking down all possibilities for interaction within the sampling process and eliminating the possibilities for one sample to influence another it may be possible to reach confidence that it is plausible to assume that the samples are independent. Because the math says that independent data are not correlated much of the exploratory data analysis involves looking for correlations that would signal dependent data. The Minitab® authors make this clear in the example they offer to illustrate their definition.

For example, suppose quality inspectors want to compare two laboratories to determine whether their blood tests give similar results. They send blood samples drawn from the same 10 children to both labs for analysis. Because both labs tested blood specimens from the same 10 children, the test results are not independent. To compare the average blood test results from the two labs, the inspectors would need to do a paired t-test, which is based on the assumption that samples are dependent.

To obtain independent samples, the inspectors would need to randomly select and test 10 children using Lab A and then randomly select and test a different group of 10 different children using Lab B. Then they could compare the average blood test results from the two labs using a 2-sample t-test, which is based on the assumption that samples are independent.

Nicely said, and to further make their point, I am sure that the authors would agree that if it somehow turned out that the children from lab B happened to be the identical twins of the children in Lab A, they still would not have independent samples.

What happens when samples are not independent

The following example illustrates the consequences of performing a t-test when the independence assumption does not hold. We adapt a method of simulating a bivariate normal distribution with a specified covariance matrix that produces two dependent samples with a specified correlation matrix.

library(tidyverse)
library(ggfortify)
set.seed(9999)

First, we simulate a two uncorrelated samples with 20 observations each and run a two-sided t-test with equal variances. As you would expect, test output shows that there are 38 degrees of freedom and the p-value is large.

rbvn_t<-function (n=20, mu1=1, s1=4, mu2=1, s2=4, rho=0)
{
  X <- rnorm(n, mu1, s1)
  Y <- rnorm(n, mu2 + (s2/s1) * rho *
                (X - mu1), sqrt((1 - rho^2)*s2^2))
  t.test(X,Y, mu=0, alternative = "two.sided", var.equal = TRUE)
}
rbvn_t()

## 
##  Two Sample t-test
## 
## data:  X and Y
## t = 2.1, df = 38, p-value = 0.04
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.06266 5.06516
## sample estimates:
## mean of x mean of y 
##    2.9333    0.3694

Now we simulate 10,000 two-sided t-tests with independent samples having 20 observations in each sample.

ts <- replicate(10000,rbvn_t(n=20, mu1=1, s1=4, mu2=1, s2=4, rho=0)$statistic)

Plotting the simulated samples shows that the empirical density curve nicely overlays the theoretical density for the t-distribution.

p <- ggdistribution(dt, df = 38, seq(-4, 4, 0.1)) 
autoplot(density(ts), colour = 'blue', p = p,  fill = 'blue') +
  ggtitle("When variables are independent")

Moreover, the 0.975 quantile, the value that would indicate the upper boundary for the acceptance region for an $\alpha$ value of 0.05 is very close to the theoretical value of 2.024.

quantile(ts,.975)

## 97.5% 
## 1.996

qt(.975,38)

## [1] 2.024

Next, we simulate 10,000 small samples of 20 with a correlation of 0.3.

ts_d <- replicate(10000,rbvn_t(n=20, mu1=1, s1=4, mu2=1, s2=4, rho=.3)$statistic)

We see that now the fit is not so good. There simulated distribution has noticeably less probability in the tails.

pd <- ggdistribution(dt, df = 38, seq(-4, 4, 0.1)) 
autoplot(density(ts_d), colour = 'blue', p = pd,  fill = 'blue') +
  ggtitle("When variables are NOT independent")

The .975 quantile is much lower than the theoretical value of 2.024 showing that dependent data would lead to very misleading p-values.

quantile(ts_d,.975)

## 97.5% 
##  1.73

Summary

Properly performing a t-test on data obtained from an experiment could mean doing a whole lot of up front work to design the experiment in a way that will make the assumptions plausible. One could argue that the real practice of statistics begins even before making exploratory plots. Doing statistics with found data is much more problematic. At a minimum, doing a simple t-test means acquiring more that a superficial understanding of how the data were generated.

Finally, when all is said and done, and you have a well constructed t-test that results in a sufficiently small p-value to reject the null hypothesis, you will have attained what most people call a statistically significant result. However, I think this language misleadingly emphasizes the mechanical grinding of the “steel trap” logic of the test that I mentioned above. Maybe instead we should emphasize the work that went into checking assumptions, and think about hypothesis tests as producing “plausibly significant” results.

Some resources for doing t-tests in R

Holmes and Huber (2019) Modern Statistics for Modern Biology, Chapter 6,
Orloff and Bloom (2014) Null Hypothesis Significance Testing II
Poldrack (2018) Statistical Thinking for the 21st century, Chapter 9
Spector Using t-tests in R
Wetherill (2015) How to Perform T-tests in R

February 2021: "Top 40" New CRAN Packages

Fri, 19 Mar 2021 00:00:00 +0000

In February, two hundred forty-three new packages made it to CRAN, many of them very interesting and at least one entertaining. It was exceptionally difficult to pick the “Top 40”, but here they are, more or less, in eleven categories: Computational Methods, Data, Finance, Games, Genomics, Machine Learning, Mathematics, Medicine, Networks and Graphs, Statistics, Utilities, and Visualization. iconr in the Networks and Graphs section is a package for doing computational archaeology, a relatively new field that I hope will dig R. I also hope that sassy in the Statistics sections helps some statisticians find their way to R.

Computational Methods

blaster v1.0.3: Implements an efficient BLAST-like sequence comparison algorithm, written in C++11 and using native R data types. See Schmid et al. (2018) for background and README for an example.

rando v0.2.0: Provides random number generating functions that are much more context aware than the built-in functions. The functions are also safer, as they check for incompatible values, and reproducible.

Data

AWAPer 0.1.46: Provides catchment area weighted climate data NetCDF files from the Bureau of Meteorology Australian Water Availability Project for all of Australia. There is a vignette on Daily Area Weighted PET and Precipitation and another on Daily Point Precipitation

caRecall v0.1.0: Provides API access to the Government of Canada Vehicle Recalls Database used by the Defect Investigations and Recalls Division for vehicles, tires, and child car seats. See the vignette.

geofi v1.0.0: Provides tools for reading Finnish open geospatial data in R. There are vignettes on Datasets, Joining Attributes, Making Maps, Data Manipulation, and Color-coded Maps.

hockeystick v0.4.0: Provides easy access to essential climate change data sets for non-climate experts. Users can download the latest raw data from authoritative sources and view it via pre-defined ggplot2 charts. Data sets include atmospheric CO2, instrumental and proxy temperature records, sea levels, Arctic/Antarctic sea-ice, and Paleoclimate data. Sources include: NOAA Mauna Loa Laboratory, NASA GISTEMP, National Snow and Sea Ice Data Center, CSIRO, NOAA Laboratory for Satellite Altimetry, and Vostok Paleo carbon dioxide and temperature data. See README for examples.

votesmart v0.1.0: Implements a wrapper to the Project VoteSmart API. See the vignette.

Finance

PriceIndices v0.0.3: Provides functions to compute bilateral and multilateral indexes. For details, see: de Haan and Krsinich (2017) and Diewert and Fox (2020). The vignette offers examples.

treasuryTR v0.1.1: Generates Total Returns (TR) from bond yield data with fixed maturity (e.g. reported treasury yields) which may provide an alternative to commercial products. See Swinkels (2019) for background and the vignette for examples.

Games

pixelpuzzle v1.0.0: Implements a puzzle game that can be played in the R console. Restore the pixel art by shifting rows. Learn how to play here.

Genomics

CDSeq v1.0.8: Provides functions to estimate cell-type-specific gene expression profiles and sample-specific cell-type proportions simultaneously using bulk sequencing data. See Kang et al. (2019) for the theory and the vignette for examples.

ClusTorus v0.0.1: Provides various tools for clustering multivariate angular data on the torus including angular adaptations of usual clustering methods such as the k-means clustering, pairwise angular distances. See the vignette for examples.

dsb v0.1.0: Provides a method for normalizing and denoising protein expression data from droplet based single cell experiments. See the vignette for tutorials on how to integrate dsb with Seurat, Bioconductor and the AnnData class in Python. The preprint Mulè et al. (2020) describes the details.

Machine Learning

besridge v1.0.4: Provides functions to perform ridge regression in complex situations on high dimensional data using the primal dual active set algorithm proposed in Wen et al. (2020). Functions support regression, classification, count regression and censored regression, group variable selection and nuisance variable selection. See the vignette for examples.

ROCket v1.0.1: Provides functions for estimating receiver operating characteristic (ROC) curves and area under the curve (AUC) calculation which distinguish two types of ROC curve representations: 1) parametric curves - the true positive rate (TPR) and the false positive rate (FPR) are functions of a score parameter and 2) function curves - TPR is a function of FPR. See Gonçalves et al. (2014) and Cai & Pepe (2004) for background and README to get started.

wordpiece v1.0.2: Provides functions to apply Wordpiece tokenization to input text, given an appropriate vocabulary. The BERT tokenization conventions are used by default. See the vignette for an example.

Mathematics

fractD v0.1.0: Estimates the of fractal dimension of a black area in 2D and 3D (slices) images using the box-counting method. See Klinkenberg (1994) for background and the vignette for examples.

spacefillr v0.2.0: Generates random and quasi-random space-filling sequences including Halton, Sobol and other sequences with errors distributed as various types of jittered blue noise. See Joe and Kuo (2018), Christensen et al. (2018) and Heitz et al. (2019) for background and look here for examples.

tensorsign v0.1.0: Provides an efficient algorithm for nonparametric tensor completion via sign series. The algorithm which employs the alternating optimization approach to solve the weighted classification problem is described in Lee and Wang (2021)

Medicine

bp v1.0.1: Provides functions to aid in the analysis of blood pressure data of all forms by providing both descriptive and visualization tools for researchers. There is a vignette.

CHOIRBM v0.0.2: Provides functions for visualizing body map data collected with the Collaborative Health Outcomes Information Registry CHOIR). See the vignette.

QDiabetes v1.0-2: Calculates the risk of developing type 2 diabetes using risk prediction algorithms derived by ClinRisk. Look here for information and examples.

Networks and Graphs

bnmonitor v0.1.0. Implements sensitivity and robustness methods for Bayesian networks including methods to perform parameter variations via a variety of co-variation schemes, to compute sensitivity functions and to quantify the dissimilarity of two Bayesian networks via distances and divergences. See Chan and Darwiche (2002), Cowell et al. (2007), and Goergen and Leonell (2020) for background and README for examples.

iconr v0.1.0: Provides formal methods for studying archaeological iconographic data sets (rock-art, pottery decoration, stelae, etc.) using network and spatial analysis See Alexander (2008) and Huet (2018) for background and the vignette for examples.

MLVSBM 0.2.1: Provides functions for simulation, inference and clustering of multilevel networks using a stochastic block model framework as described in Chabert-Liddell et al. (2021). There is a tutorial.

motifr v1.0.0: Provides tools to analyze motifs(small configurations of nodes and edges) in multi-level networks (networks which combine multiple networks in one, e.g. social-ecological networks.) See The motif zoo and Baseline model comparisons.

Statistics

cfda v0.9.9: Provides functions to encode categorical data as functional data and perform basis statistical analysis. See Preda et al. (2020) for background and the vignette to get started.

cvCovEst v0.3.4: Implements an efficient cross-validated approach for covariance matrix estimation, particularly useful in high-dimensional settings. See the vignette for background and examples.

flipr v0.2.1: Implements a permutation framework point estimation, confidence intervals or hypothesis testing for multiple data types. There is a Tour of Permutation Inference, and vignettes on Alternative Hypothesis Testing, the Exactness of Permutation Tests, and Calculating p-value Functions.

ipmr v0.0.1: implements integral projection models using an expression based framework that handles density dependence and environmental stochasticity and provides tools for diagnostics, plotting, simulations, and analysis. See Easterling et al. (2000) for an in depth description of integral projection models. There is an Introduction and vignettes on Age-Size IPMS, Density Dependent IPMS, Hierarchical Notation, and Data Structures.

metapack v0.1.1: Provides functions performing Bayesian inference for meta-analytic and network meta-analytic models through Markov chain Monte Carlo algorithm. See Yao et al. (2015) for the theory, the vignette for an introduction and the online documentation.

sassy v1.0.4: Loads a collection of packages that collectively aim to make R easier for SAS® programmers. Functions bring many familiar SAS® concepts to R, including data libraries, data dictionaries, formats and format catalogs, a data step, and a traceable log. There is an Introduction, and vignettes with example Figures, Listings, and Tables, as well as a few Disclaimers which include a statement indicating that the packages were developed in the context of the pharmaceutical industry but should be generally helpful.

Utilities

gargoyle v0.0.1: Implements an event-Based framework for building Shiny apps. Instead of relying on standard Shiny reactive objects, this package allow to relying on a lighter set of triggers, so that reactive contexts can be invalidated with more control. See the vignette.

multidplyr Provides simple multicore parallelism through functions that partition a data frame across multiple worker processes. See the vignette.

quarto v0.1: Provides an interface to the Quarto markdown publishing system and allows converting R Markdown documents and Jupyter Notebooks to a variety of output formats.

var v0.0.2: Provides functions to manage, provision and use virtual machines pre-configured for R, and develop, test and build package in a clean environment. Vagrant and a provider such as Virtualbox must be installed.

Visualization

ggh4x v0.1.2.1: Extends ggplot2 facets by setting individual scales per panel, resizing panels, providing nested facets, and allowing multiple colour and fill scales per plot. See the Introduction, and the vignettes Facets, Misc, Position Guides, and Statistics.

tastypie v0.0.3: Provides functions and templates for making pie charts even though you probably shouldn’t. See the vignettes available templates and Your favorite template, and look here for examples.

terrainr v0.3.1: Provides functions to retrieve, manipulate, and visualize geospatial data, with an aim towards producing ‘3D’ landscape visualizations in the Unity 3D rendering engine. Functions are also provided for retrieving elevation data and base map tiles from the USGS National Map. There is an Introduction and a vignette on importing terrain tiles.

Cheat Sheets

Wed, 10 Mar 2021 00:00:00 +0000

In a previous post, I described how I was captivated by the virtual landscape imagined by the RStudio education team while looking for resources on the RStudio website. In this post, I’ll take a look at Cheatsheets another amazing resource hiding in plain sight.

Apparently, some time ago when I wasn’t paying much attention, cheat sheets evolved from the home made study notes of students with highly refined visual cognitive skills, but a relatively poor grasp of algebra or history or whatever to an essential software learning tool. I don’t know how this happened in general, but master cheat sheet artist Garrett Grolemund has passed along some of the lore of the cheat sheet at RStudio. Garrett writes:

One day I put two and two together and realized that our Winston Chang, who I had known for a couple of years, was the same “W Chang” that made the LaTex cheatsheet that I’d used throughout grad school. It inspired me to do something similarly useful, so I tried my hand at making a cheatsheet for Winston and Joe’s Shiny package. The Shiny cheatsheet ended up being the first of many. A funny thing about the first cheatsheet is that I was working next to Hadley at a co-working space when I made it. In the time it took me to put together the cheatsheet, he wrote the entire first version of the tidyr package from scratch.

It is now hard to imagine getting by without cheat sheets. It seems as if they are becoming expected adjunct to the documentation. But, as Garret explains in the README for the cheat sheets GitHub repository, they are not documentation!

RStudio cheat sheets are not meant to be text or documentation! They are scannable visual aids that use layout and visual mnemonics to help people zoom to the functions they need. … Cheat sheets fall squarely on the human-facing side of software design.

Cheat sheets live in the space where human factors engineering gets a boost from artistic design. If R packages were airplanes then pilots would want cheat sheets to help them master the controls.

The RStudio site contains sixteen RStudio produced cheat sheets and nearly forty contributed efforts, some of which are displayed in the graphic above. The Data Transformation cheat sheet is a classic example of a straightforward mnemonic tool. It is likely that even someone who just beginning to work with dplyr will immediately grok that it organizes functions that manipulate tidy data. The cognitive load then is to remember how functions are grouped by task. The cheat sheet offers a canonical set of classes: “manipulate cases”, “manipulate variables” etc. to facilitate the process. Users that work with dplyr on a regular basis will probably just need to glance at the cheat sheet after a relatively short time.

The Shiny cheat sheet is little more ambitious. It works on multiple levels and goes beyond categories to also suggest process and workflow.

The Apply functions cheat sheet takes on an even more difficult task. For most of us, internally visualizing multi-level data structures is difficult enough, imaging how data elements flow under transformations is a serious cognitive load. I for one, really appreciate the help.

Cheat sheets are immensely popular. And even in this ebook age where nearly everything you can look at is online, and conference attending digital natives travel light, the cheat sheets as artifacts retain considerable appeal. Not only are they useful tools and geek art (Take a look at cartography) for decorating a workplace, my guess is that they are perceived as runes of power enabling the cognoscenti to grasp essential knowledge and project it in the world.

When in-person conferences resume again, I fully expect the heavy paper copies to disappear soon after we put them out at the RStudio booth.

2021 R Conferences

Wed, 03 Mar 2021 00:00:00 +0000

It is not yet clear what lasting impact the Covid-19 pandemic will ultimately have on R conferences. We are still adapting to our inability to attend large events, and trying to make the best of the “silver lining” of virtual events which permit worldwide participation. The following is an attempt to list 2021 conferences that are likely to have interesting R content. I suspect that it is incomplete. If you know of an R Conference that is not mentioned, please add it to the comments section for this post.

Upcoming Events

NICAR 2021 (March 3 - 5), the Investigative Reporters & Editors Conference on data journalism should be well attended by data journalists using R for their everyday reporting.

CascadiaRConf 2021 (June 4 - 5), a jewel of a regional R conference for its first three years, was canceled in 2020. It is back this year as a virtual event. The Call for Presentations is open.

PHUSE US Connect 2021 (June 14 - 18) - PHUSE is a non-profit organization with the mission: “Sharing ideas, tools and standards around data, statistical and reporting technologies to advance the future of life sciences.” The conference which is focused on clinical data science is likely to have some interesting R content this year. The Call for Papers is open.

PSI 2021 Online (June 21 - 23) usually attracts six hundred or so statisticians from the pharmaceutical industry when the conference is held in person. PSI statisticians bring you Wonderful Wednesdays.

useR! 2021 (July 5 - 9) has an outstanding lineup of keynote speakers. The program is very likely to make US based attendees night-owls.

BioC 2021 (August 4 - 6) is the must attend event for anyone doing computational biology. Peruse the slides of past events to get a “rear view preview” of what to expect.

JSM 2021 Seattle (August 7 - 12), the mother of all statistics conferences, usually draws between 4,000 and 6,000 statisticians to in-person events. This organizers appear to be following some pretty optimistic Covid-19 vaccination rate models.

R/Medicine 2021 (August 27 - 29) has the dates, but no website yet. Don’t worry, the clinicians are big come from behind organizers. Last year’s conference was outstanding, and I expect an amazing event again this year.

R/Pharma 2021 organizers like to give R / Medicine organizers a head start, but a well placed source tells me that the conference will take place in Q3 or Q4. For the past three years, R/Pharma has been a bright star among R conferences where some of the best Shiny developers in the world meet and discuss their work.

EARL Conference 2021 (September 6 - 10), the premier R in industry event, will be online this year. The call for abstracts is already open.

NY R Conference 2021 is usually the perfect way to spend a couple of Manhattan Spring days. This year, the organizers are hoping for and in-person event in August or September if things go really well, but planning to surpass their spectacular 2020 virtual event if things don’t.

BIOP 2021 Rockville, MD (September 21 - 23) may be an in-person event. This workshop was originally an event for FDA statisticians but is now open to all statisticians interested in statistical practices for all areas regulated by the FDA.

noRth 2021 (September 29 30) is a regional conference out of the “Twin Cities” that is looking to virtually expand their reach within the R Community. Gabriela de Queiroz heads the list of confirmed speakers which includes new faces from IBM, Google, and the Federal Reserve.

Foss4g for OSGEO Buenos Aires (September 27 - October 2) is the annual conference of OSGeo, the Open Source Geospatial Foundation. Given the prominence of R in geospatial analysis this is sure to be an R heavy event. The conference will be online.

PHUSE EU Connect 21 (November 15 - 19) See above.

R Government has a reasonable chance of pulling off an in-person event (at least for people in the DC area) sometime in December if the region gets a break from Covid.

Earlier Events

rstudio::global (January 21) - The talks from this unique 24 hour, worldwide event are on line.

Evidence Synthesis and Meta-Analysis in R - The talks from this conference and hackathon which attracted 514 participants from 26 countries are online here.

January 2020: "Top 40" New CRAN Packages

Wed, 24 Feb 2021 00:00:00 +0000

Two hundred thirty new packages made it to CRAN in January. Here are my “Top 40” selections in ten categories: Data, Finance, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Data

igoR v0.1.1: Provides tools to extract information from the Intergovernmental Organizations (‘IGO’) Database , version 3, provided by the Correlates of War Project. See Pevehouse et al. (2020) for information from 1815 to 2014, and get started with the vignette.

OTrecord v0.1.0: Uses optimal transportation theory as described in Gares, Guernec & Savy (2019) and Gares & Omer (2020) to solve recoding problems. Given two databases that share a subset of variables, package functions assist users in obtaining a unique synthetic database with complete information. See the vignette for examples.

pwt10 v10.0-0: Interfaces to the Penn World Table 10.x which provides information on relative levels of income, output, input, and productivity for 183 countries between 1950 and 2019.

trainR v0.0.1: Interfaces to the the National Rail Enquiries systems, including Darwin which provides real-time arrival and departure predictions, platform numbers, delay estimates, schedule changes and cancellations. Look here for examples.

Finance

LSMRealOptions v0.1.0: Provides an implementation of the least-squares Monte Carlo simulation method to value American option products and capital investment projects through real options analysis. Cash flows are modeled as being dependent upon underlying state variables that evolve stochastically. See the vignette for examples.

Genomics

AlleleShift v0.9-2: Provides methods for calibrating and predicting shifts in allele frequencies through redundancy analysis (vegan::rda()) and generalized additive models (mgcv::gam()) and functions to visualize the predicted changes in frequencies. See README for examples.

GenomeAdmixR v1.1.3: Provides tools to simulate how patterns in ancestry along the genome change after admixture. Se Janzen (2020) for the details and the vignettes Isofemales, Joyplot, Visualization, and Walkthrough.

MOSS v0.1.0: Implements an omics integration method based on sparse singular value decomposition to deal with the challenges of high dimensionality, noise and heterogeneity among samples and features in omics data. See (Gonzalez-Reymundez & Vazquez, 2020) for background and the vignette for examples.

Machine Learning

autoMrP v0.98: Implements a tool that improves the prediction performance of multilevel regression with post-stratification (MrP) by combining a number of machine learning methods. For information on the method, refer to Broniecki, Wüest, Leemann (2020) and the vignette.

aweSOM v1.1: Implements Self-organizing maps, a method for dimensionality reduction and clustering of continuous data, as well as interactive graphics to assist analysis. See Kohonen (2001) for background and the vignette for an overview of the package.

RandomForestsGLS v0.1.2: Fits non-linear generalized least square regression models with Random Forests as described in Saha, Basu & Datta (2020).

torchaudio v0.1.1.0: Provides access to datasets, models and preprocessing facilities for deep learning in audio. See the vignette.

vimpclust v0.1.0: Implements functions to perform sparse k-means clustering with a group penalty and variable selection on mixed categorical and numeric data. See Chavet et al. (2020) for background. There are vignettes on numeric and mixed data sparse k-means clustering.

Medicine

dataQuieR v1.0.4: Provides functions to assess data quality issues in studies. See the TMF Guideline and the DFG Project for background, and the vignette for examples.

Science

LPDynR v1.0.1: Implements methods that use phenological and productivity-related variables derived from time series of vegetation indexes to assess ecosystem dynamics. Functions compute an indicator with five classes of land productivity dynamics. Look here for background. See the vignette for an example.

rgee v1.0.8: Provides an Earth Engine client library for R that includes all Earth Engine API classes, modules, and functions, as well as additional functions for importing spatial objects, extracting time series, and displaying metadata and interactive maps. Look here for further details. Read the Introduction and the vignette on Best Practices to get started.

SAMtool v1.1.1: Provides tools for simulating the MSEtool operating model to inform data-rich fisheries. It includes a conditioning model, tools for assessing models of varying complexity and comparing models, and diagnostic tools for evaluating assessments inside closed-loop simulations. There is a User Guide and a series of seven more vignettes including an overview of the Rapid Conditioning Model (RCM) for conditioning MSEtool operating models, and a mathematical description of RCM.

Statistics

circularEV v0.1.0: Provides functions for performing extreme value analysis on a circular domain. See the local methods example and the spline example.

ghcm v1.0.0: Implements a statistical hypothesis test for conditional independence which can be applied to both discretely observed functional data and multivariate data. See Lundborg et al. (2020) for details and the vignette for an overview of the generalized Hilbert Covariance measure with examples.

gplite v0.11.1: Implements the most common Gaussian process models using Laplace and expectation propagation approximations, maximum marginal likelihood inference for the hyperparameters, and sparse approximations for larger datasets. See the vignette for a quick start.

multibridge v1.0.0: Implements functions to evaluate hypotheses concerning the distribution of multinomial proportions using bridge sampling. Functions are able to compute Bayes factors for hypotheses that entail inequality constraints, equality constraints, free parameters, and mixtures of all three. See Sarafoglou et al. (2020) for background and the examples: Memory of Lifestresses, Mendelian Laws of Inheritance, and Prevalence of Statistical Reporting Errors.

partR2 v0.9.1: Provides functions to to partition the variance explained in generalized linear mixed models (GLMMs) into variation unique to predicators and variation shared among predictors. This can be done using semi-partial R² and inclusive R². See Nakagawa & Schielzeth (2013) and Nakagawa, Johnson & Schielzeth (2017) for the theory and the vignette for examples.

spNetwork v0.1.1: Provides tools to perform spatial analysis on network including estimating network kernel density, building spatial matrices. See Okabe et al. (2019) for background and the vignettes: Network k Functions, Network Kernel Density Estimate, Details about NKDE, and Spatial Weight Matrices.

ubms v1.0.2: Provides functions to fit Bayesian hierarchical models, including single-season occupancy, dynamic occupancy, and N-mixture abundance models, of animal abundance and occurrence with the rstan package. See Carpenter et al. (2017) and Fiske and Chandler (2011) for background. There is a package Overview, a vignette on Random Effects, and another on Comparing ubms with JAGS.

Time Series

autostsm v1.2: Provides functions to automate the decomposition of structural time series into trend, cycle, and seasonal components using the Kalman filter. See Koopman et al. (2012) for the theory and the vignette for an overview with examples.

bayesforecast v0.0.1: Provides functions to fit Bayesian time series models using Stan for full Bayesian inference. It includes seasonal ARIMA, ARIMAX, dynamic harmonic regression, GARCH, t-student innovation GARCH models, asymmetric GARCH, random walks, stochastic volatility models for univariate time series. See Hyndman (2017) and Carpenter et al. (2017) for background and README for examples.

Utilities

autoharp v0.0.5: Implements customizable tools for assessing and grading R or R-markdown scripts from students which allow for checking correctness of code output, runtime statistics and static code analysis. There is a User Manual and a vignettes on the S4 class treeharp.

cachem v1.0.4: Provides functions to cache R objects with automated pruning. Caches can limit either their total size or the age of the oldest object (or both), automatically pruning objects to maintain the constraints. See README for examples.

eList v0.2.0: Provides list compression functions to convert for loops into vectorized lapply() functions which support loops with multiple variables, parallelization, and loops across non-standard objects. See the vignette for examples.

Microsoft365R v1.0.0: Builds on AzureGraph to implement and interface to Microsoft365 and enables access to data stored in SharePoint Online and OneDrive. See the vignette to get started.

rtables v0.3.6: Provides a framework for declaring complex multi-level tabulations and then applying them to data. Tables are modeled as hierarchical, tree-like objects which support sibling sub-tables, arbitrary splitting or grouping of data in row and column dimensions, cells containing multiple values, and the concept of contextual summary computations. There is a Introduction and a series of vignettes on comparing against baseline or control, a clinical trials example, constructing tables manually, pruning and sorting tables, subsetting tables, Tabulation concepts, and a comparison with dplyr tabulation.

targets v0.1.0: Brings together function-oriented programming and make- like declarative workflows in toolkit for building statistics and data science pipelines in R. The methodology borrows from GNU make and drake. See the vignette and the reference website.

Visualization

ggmulti v0.1.0: Provides tools such as serial axes objects, Andrew’s plot, various scatter plot glyphs to visualize high dimensional data. There are vignettes on visualizing high dimensional data, adding glyphs to scatter plots, and creating histograms with density.

ggOceanMaps v1.0.9: Allows plotting data on bathymetric maps using ggplot2 using data that contain geographic information from anywhere around the globe. There is a User Manual and a vignette on pre-made shape files.

pacviz v1.0.0.5: Provides functions to map data onto a radial coordinate system and visualize the residual values of linear regression and Cartesian data in the defined radial scheme. See the pacviz documentation for more information.

parallelPlot v0.1.0: Provides functions to create parallel coordinates plots using the htmlwidgets package and d3.js. The vignette provides multiple examples.

thematic v0.1.1: Provides tools to “theme” ggplot2, lattice, and base graphics using a small set of choices that include foreground color, background color, accent color, and font family. See README for examples.

R Interface for MiniZinc

Mon, 15 Feb 2021 00:00:00 +0000

Akshit Achara is a medical device engineer and computer science enthusiast based in Bengaluru, Karnataka, India. You can connect with Akshit on LinkedIn.

Introduction

Constraint programming is a paradigm for solving combinatorial problems that draws on a wide range of techniques from artificial intelligence, computer science, and operations research. MiniZinc is a free and open-source constraint modeling language. Constraint satisfaction and discrete optimization problems can be formulated in a high-level modeling language. Models are compiled into an intermediate representation that is understood by a wide range of solvers. MiniZinc itself provides several solvers, for instance GeCode. The existing packages in R are not powerful enough to solve even mid-sized problems in combinatorial optimization.

Until recently, there were implementations of an Interface to MiniZinc in Python like MiniZinc Python and pymzn and JMiniZinc for Java but none for R.

rminizinc as a GSOC project

rminizinc started as a Google Summer of Code Project in 2020. Initially, the goal was to provide infrastructure/support for creating 15-20 commonly used MiniZinc problems by creating classes and functions for providing the basic syntax/constructs used in those MiniZinc models. However, it was decided that the libminizinc (MiniZinc C++ API) library can be leveraged using Rcpp for parsing various MiniZinc models and a mirror API can be created to construct the MiniZinc models. Using the library helped me in understanding MiniZinc more which in turn also helped to to provide more features and test the package on larger problems. The following objectives were achieved at the end of the GSOC period:

Parse a MiniZinc model into R.
Find the model parameters which have not been assigned a value yet.
Set the values of unassigned parameters. (Scope needs to be extended)
Solve a model and get parsed solutions as a named list in R.
Create a MiniZinc model in R using the R6 classes from MiniZinc API mirror.
Manipulate a model.

The development continued till the end of GSOC but the package was not yet submitted to CRAN.

Post GSOC

The package submission to CRAN was very challenging. Libminizinc and solver binaries were required in order to use the package because the Rcpp functions were using them to parse and solve the models. To tackle this, we leveraged the autotools to create a configure script for letting the users provide custom paths and configure the package during the installation, used #ifdef macros to provide alternative definitions in case libminizinc and/or solvers are not present on the system.

Examples

There are a lot of features provided by the package but let’s start with something simple say solving a knapsack problem. Knapsack problem is known by everyone who has interest in constraint programming. The knapsack problem is a problem in combinatorial optimization: Given a set of items, each with a weight and a value, determine the number of each item to include in a collection so that the total weight is less than or equal to a given limit and the total value is as large as possible. It derives its name from the problem faced by someone who is constrained by a fixed-size knapsack and must fill it with the most valuable items. The problem often arises in resource allocation where the decision makers have to choose from a set of non-divisible projects or tasks under a fixed budget or time constraint, respectively.

The knapsack() function can be used to directly solve the knapsack problem. Here, n is the number of items, capacity is the total capacity of carrying weight, profit is the profit corresponding to each item and weight is the weight/size of each item. The goal is to maximize the total profit. The solution is returned in the form of a named list with all the solutions along with the optimal solution if found.

Please find the installation instructions in the vignette or the github readme.

library(rminizinc)
# knapsack problem
print(knapsack(n = 3, capacity = 9, profit = c(15,10,7), size = c(4,3,2)))

$SOLUTION0
$SOLUTION0$x
[1] 0 0 0


$SOLUTION1
$SOLUTION1$x
[1] 0 0 1


$SOLUTION2
$SOLUTION2$x
[1] 0 0 2


$SOLUTION3
$SOLUTION3$x
[1] 0 0 3


$SOLUTION4
$SOLUTION4$x
[1] 0 0 4


$SOLUTION5
$SOLUTION5$x
[1] 0 1 3


$OPTIMAL_SOLUTION
$OPTIMAL_SOLUTION$x
[1] 1 1 1

A function to solve the assignment problem has also been provided. More common examples will be provided in the next package releases based on the user feedback. This will especially be useful for the users who don’t have any knowledge of MiniZinc.

Basic knowledge of MiniZinc is required in order to understand the next examples. MiniZinc tutorial would be a good place to start.

The users can also create a MiniZinc model using the API. Let’s compute the base of a right angled triangle given the height and hypotenuse. The Pythagoras theorem says that In a right-angled triangle, the square of the hypotenuse side is equal to the sum of squares of the other two sides i.e $a² + b² = c²$. The theorem give us three functions, $c = \sqrt{(a² + b²)}$, $a = \sqrt(c² - b²)$ and $b = \sqrt(c² - a²)$.

MiniZinc Representation of the model:

int: a = 4;
int: c = 5;
var int: b;

constraint b>0;
constraint a² + b² = c²;

solve satisfy;

Let’s create and solve the model using rminizinc.

a = IntDecl(name = "a", kind = "par", value = 4)
c = IntDecl(name = "c", kind = "par", value = 5)
b = IntDecl(name = "b", kind = "var")

# declaration items
a_item = VarDeclItem$new(decl = a)
b_item = VarDeclItem$new(decl = b)
c_item = VarDeclItem$new(decl = c)

# b > 0 is a binary operation
b_0 = BinOp$new(lhs = b$getId(), binop = ">", rhs = Int$new(0))
constraint1  = ConstraintItem$new(e = b_0)

# a ^ 2 is a binary operation
# a$getId() gives the variable identifier
a_2 = BinOp$new(lhs = a$getId(), binop = "^", Int$new(2))
b_2 = BinOp$new(lhs = b$getId(), binop = "^", Int$new(2))
a2_b2 = BinOp$new(lhs = a_2, binop = "+", rhs = b_2)
c_2 = BinOp$new(lhs = c$getId(), binop = "^", Int$new(2))
a2_b2_c2 = BinOp$new(lhs = a2_b2, binop = "=", rhs = c_2)
constraint2  = ConstraintItem$new(e = a2_b2_c2)

solve  = SolveItem$new(solve_type = "satisfy")

model = Model$new(items = c(a_item, b_item, c_item, constraint1, constraint2, solve))

cat(model$mzn_string())

int: a;

var int: b;

int: c;

constraint (b > 0);

constraint (((a ^ 2) + (b ^ 2)) = (c ^ 2));

solve  satisfy;

Creating the model required a lot of code which is more cumbersome that writing the model in MiniZinc itself. However, that was to give the users a taste of various classes that can be used to create items and expressions. This will especially be useful in modifying an existing model.

The items can directly be provided as strings.

a = VarDeclItem$new(mzn_str = "int: a = 4;")
c = VarDeclItem$new(mzn_str = "int: c = 5;")
b = VarDeclItem$new(mzn_str = "var int: b;")
constraint1 = ConstraintItem$new(mzn_str = "constraint b > 0;")
constraint2 = ConstraintItem$new(mzn_str = "constraint a^2 + b^2 = c^2;")
solve = SolveItem$new(mzn_str = "solve satisfy;")
model = Model$new(items = c(a, b, c, constraint1, constraint2, solve))
cat(model$mzn_string())

int: a = 4;

var int: b;

int: c = 5;

constraint (b > 0);

constraint (((a ^ 2) + (b ^ 2)) = (c ^ 2));

solve  satisfy;

The model can directly be parsed by providing the string representation as the argument to mzn_parse(). This method uses libminizinc to parse the model. The included mzn files are appearing because the parsed model is serialized back by libminizinc.

pythagoras_string = 
  " int: a = 4;
    int: c = 5;
    var int: b;
    
    constraint b > 0;
    constraint a^2 + b^2 = c^2;
    
    solve satisfy;
 "
model = mzn_parse(model_string = pythagoras_string)
cat(model$mzn_string())

int: a = 4;

int: c = 5;

var int: b;

constraint (b > 0);

constraint (((a ^ 2) + (b ^ 2)) = (c ^ 2));

solve  satisfy;

include "solver_redefinitions.mzn";

include "stdlib.mzn";

Let’s solve the model now.

solution = mzn_eval(r_model = model)
print(solution)

$SOLUTION_STRING
[1] "{\n  \"b\" : 3\n}\n----------\n==========\n"

$SOLUTIONS
$SOLUTIONS$OPTIMAL_SOLUTION
$SOLUTIONS$OPTIMAL_SOLUTION$b
[1] 3

In the next post, we will try another problem and/or use some other features of rminizinc. Please let me know what you think.

Some thoughts on rstudio::global talks

Thu, 04 Feb 2021 00:00:00 +0000

The fifty-five videos from last month’s rstudio::global conference are now available online. You will find them at the link above arranged in ten categories (Keynotes, Data for Good, Language Interop, Learning, Modeling, Organizational Tooling, Package Dev, Programming, Teaching, and Visualization) that reflect fundamental areas of technical and community infrastructure and R applications. Theses talks were selected from hundreds of submissions, many of which were really very good. I participated in the first selection round and found it impossible to make some choices, so I am certain that it must have been agonizingly difficult for the program committee to pare down to the final selections.

I believe that you will find the content of most of these talks to be nothing less than compelling. The themes and moods of the talks range from informative and deeply technical R issues to data science, journalism, art, education and public service. A few talks transcend the parochial concerns of the R community and address issues that are important to society at large. It is gratifying to see that in the hands of committed people R is helping to make the world just a little bit better. The videos themselves are high quality and a pleasure to watch. Unlike typical conference videos recorded in real time, all of these were produced with excellent lighting, good audio, and were rehearsed, pre-recorded, and edited. Except for the keynotes, the talks are shorter than twenty minutes.

In the remainder of this post, I will highlight just five talks that I personally found compelling. I have arranged them in an order that I think makes sense to view them. But, you might do just as well to sample talks from the categories listed above that organize the talks on the conference page. My selections do not cover the whole range of topics submitted, and they certainly do not include all of the good stuff. I do think, however, that they reflect the quality of the talks, and I hope that if you watch these five you will be motivated to watch the rest too.

My first three selections are by data journalists who are out to make the world a better place. The Opioid Files: Turning big pharmacy data over to the public by Washington Post data journalist Andrew Ba Tran demonstrates the scale of the opiod scandal. I had the opportunity to meet Andrew at the NICAR conference for Investigative Reporters and Editors in 2019 where he was speaking and teaching R. NICAR opened my eyes to the discipline and tradition of Data Journalism and the efforts of data journalists to harness technology for the public good. Andrew’s conference talk represents this tradition and illustrates the data crunching skills, persistence, and unvarnished storytelling necessary to illuminate a dark topic.

Next, I recommend watching Trial and Error in Data Viz at the ACLU. Sophie Beiers is a data journalist whose work for the ACLU involves discovering and visualizing data with sufficient rigor and clarity to support arguments that will hold up in court. Sophie describes the messy work of iterating through visualizations in a process built around candid feedback from colleagues and stakeholders. Driving the process is a determination to make charts that effectively communicate key points to the intended audience. Sophie’s talk hints at the emotional toll caused by striving to see the people behind the data, and the satisfaction that comes from making a difference. The ACLU analytics team coined a word for expressing the excitement at being able to prove terrible news with quantitative evidence.

The third talk on my list by John Burn Murdoch on Reporting on and visualizing the pandemic continues the theme of polishing visualizations until they work for the target audience. John is a data journalist with the Financial Times who has garnered quite a following for producing data visualizations that command attention. John’s talk dives deeply into his process of evolving a visualization until it not only illustrates what he wants to show but also shows that it is resonating with his mass audience.

In thinking about Sophie and John’s work, the Japanese word kaizen (making something good for the good of other people) comes to mind. There are many R visualization experts who can lay down the basic principles of making a good data visualization, but few I think, that have Sophie and John’s empathy and capacity to listen and process criticism.

The final two talks on my list are by R developers who are concerned with the big picture of sustaining the R package ecosystem. Hadley Wickham’s Keynote Maintaining the house the tidyverse built addresses a fundamental challenge encountered by all complex software projects that develop over time. How do you maintain functional stability while coping with growth and change? Hadley’s solution for the tidyverse encapsulated in the following figure:

may not be the right solution for all subsystems within the R ecosystem, but surely something like it must evolve to reach into all of the corners of the R Universe. For example, consider the CRAN Task Views. When they were assembled, each of these curated lists of R packages represented the cutting edge of software for a particular functional area. The Task View maintainers do a mostly unacknowledged, essential service in keeping these up to date. Nevertheless, it is not difficult to discover Task View packages that have not changed significantly in five or ten years or more. To my knowledge, there are no standards for retiring packages and integrating new work. The future of R depends on systems thinking and the development of new ideas and tools for open source development.

These considerations lead to Jeroen Ooms’ talk: Monitoring health and impact of open-source projects which describes how ROpenSci is taking on the immense challenge of measuring the quality of R packages according to technical, social and scientific indicators, while building out the infrastructure to improve the entire R package ecosystem. Some of the tools for monitoring the status and “health” of open source software are already in place. ROpenSci is offering R-universe, a platform based in git for managing personal R repositories. Once a “universe” of packages is registered with R-universe every time an author pushes an update the platform will automatically build binaries and documentation.

Enjoy the videos!!

Dec 2020: "Top 40" New CRAN Packages

Fri, 29 Jan 2021 00:00:00 +0000

One hundred twenty-three new packages made it to CRAN in December. Here are my “Top 40” selections in nine categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

FKF.SP v0.1.0: Provides a fast and flexible Kalman filtering implementation utilizing sequential processing, designed for efficient parameter estimation through maximum likelihood estimation. See the vignette.

rminizinc v0.0.4: Implements an interface to MiniZinc, a free and open-source constraint modeling language which is used to identify feasible solutions out of a very large set of candidates when the problem can be modeled in terms of arbitrary constraints. See the vignette.

nosiySBM v0.1.4: Implements the variational expectation-maximization algorithm to fit a noisy stochastic block model to an observed dense graph and to perform node clustering. See Rebafka & Villers (2020) for background and the vignette to get started.

qsimulatR v1.0: Implements a quantum computer simulator with up to 24 qubits which provides many common gates and allows users to define general single qubit gates and general controlled single qubit gates. The package supports plotting circuits and exporting circuits to Qiskit, a Python package which can be used to run on IBM’s Quantum hardware. There is an Introduction, and vignettes on Exponentiation modulo n, Addition by Fourier transform, the Deutsch-Sozsa Algorithm, the Phase Estimation Algorithm and Quantum Fourier Trafo.

Data

eyedata v0.1.0: Contains anonymized real life, open source data sets from patients treated in Moorfields Eye Hospital, London and includes data about people who received intravitreal injections with anti-vascular endothelial growth factor due to age-related macular degeneration or diabetic macular edema. See README for the list of medical publications associated with the data sets.

rgugik v0.2.1: Automates open data acquisition including raster and vector data from the Polish Head Office of Geodesy and Cartography. See the vignettes Digital Eelvation Model, Orthophotomap, and Topographic Database.

readrba v0.1.0: Provides tools to download current and historical statistical tables and forecasts from the Reserve Bank of Australia Data which comprise a broad range of Australian macroeconomic and financial time series. See the vignette to get started.

threesixtygiving v0.2.2: Provides access to open data from 360Giving, a database of charitable grant giving in the UK. See the vignette.

USgas Links to the US Energy Information Administration to provide and overview of natural gas demand at the county level. See the vignette.

Genomics

polyqtlR v0.0.4: Provides functions for quantitative trait loci (QTL) analysis in polyploid bi-parental F1 populations. See the vignette for background and examples.

RPPASPACE v1.0.7: Provides tools for the analysis of reverse-phase protein arrays (RPPAs), which are also known as tissue lysate arrays or simply lysate arrays. See Hu (2007) for background and the Guide to for examples.

RVA v0.0.3: Provides functions to automate downstream visualization & pathway analysis in RNAseq analysis. See the vignette.

Machine Learning

comparator v0.0.1: Implements functions for comparing strings, sequences and numeric vectors for clustering and record linkage applications. It includes generalized edit distances for comparing sequences/strings, Monge-Elkan similarity for fuzzy comparison of token sets, and L-p distances for comparing numeric vectors. See README to get started.

DoubleML v0.1.1: Implements the double/debiased machine learning framework of Chernozhukov et al. (2018) for partially linear regression models, partially linear instrumental variable regression models, interactive regression models and interactive instrumental variable regression models. There are guides on Installation and Getting Started.

functClust v0.1.6: Provides functions to cluster the components that make up an interactive system on the basis of their functional redundancy for one or more collective, systemic performances. There are six vignettes including and Overview, a simple Use Case, and Multi Fuctionality.

mlpack v3.4.2.1: Implements bindings to the mlpack C++ machine learning library. See Curtin et al (2018) for background and look here for documentation.

RFCCA v1.0.3: Implements Random Forest with Canonical Correlation Analysis, a method for estimating the canonical correlations between two sets of variables depending on the subject-related covariates. The method is described in Alakus et al. (2020). See the vignette for examples.

Medicine

metaSurvival v0.1.0: Provides a function to assess information from a summary survival curve and test the between-strata heterogeneity. See the GitHub repo for an example.

Science

cmcR v0.1.3: Implements the congruent matching cells method for cartridge case identification as proposed by Song (2013) as well as an extension of the method proposed by Tong et al. (2015). There is a vignette on Decision Rules and another vignette reproducing the study by Song et al.

envi v0.1.6: Provides tools for environmental interpolation using occurrence data, covariates, kernel density-based estimation, and spatial relative risk. See Davies et al. (2018) for details on spatial relative risk, Bithell (1990) for kernel density estimation and Bithell (1991) for estimating relative risk. The vignette provides background and examples.

PAMpal v0.9.14: Provides tools for loading and processing passive acoustic data, including functions to read Pamguard data, process, and export data. See Oswald et al (2007), Griffiths et al (2020), and Baumann-Pickering et al (2010) for background. Look here for the installation guide and tutorial.

Statistics

bpcs v1.0.0: Implements models for the analysis of paired comparison data using Stan including random effects, generalized model for predictors and order effect Bayesian versions of the Bradley-Terry model. See Bradley & Terry (1952), Davidson (1970), and Carpenter et al. (2017) for background and the vignette for an overview.

brolgar v0.1.0: Provides a framework of tools to summarise, visualise, and explore longitudinal data and includes methods for calculating features and summary statistics and sampling individual series. See Tierney, Cook & Prvan and the Getting Started Guide to get going. There are also vignettes exploratory modelling, finding features, identifying interesting observations, data structures, mixed effects models, and visualisation.

MASSExtra v1.0.2: Provides enhancements, extensions and additions (such as Gramm-Schmidt orthogonalisation and generalised eigenvalue problems) to the MASS package with convenient default settings and user interfaces. See the vignette.

motifr v1.0.0: Provides tools for motif analysis in multi-level networks to visualize multi-level networks, count multi-level network motifs and compare motif occurrences to baseline models. See the motif zoo and Baseline model comparisons to get started.

OptCirClust v0.0.3: Provides fast (runtime = O(K N log^2 N), optimal, reproducible clustering algorithms for circular, periodic, or framed data based on a core algorithm for optimal framed clustering. There are vignettes on Circular genome clustering, Performance, Circular Clustering, and Framed Clusterine.

pflamelet v0.1.1: Provides functions to compute the persistence flamelets, a statistical tool for exploring the Topological Invariants of Scale-Space families introduced in Padellini and Brutti (2017).

PRDA v1.0.0: Implements the Design Analysis proposed by Gelman & Carlin (2014) which combines the evaluation of Power-Analysis with other inferential-risks. See also Altoè et al. (2020) and Bertoldo et al. (2020) for background and the vignettes PRDA, Prospective and Retrospective.

puls v0.1.1: Supplements the fda and fda.use packages by providing a method for clustering functional data using subregion information of the curves. See the vignette for an example and references.

Utilities

coro v1.0.1: Provides coroutines, a family of functions that can be suspended and resumed later on. This includes async functions (which await) and generators (which yield). See the vignette.

dataReporter v1.0.0: Provides functions to auto generate a customizable data report showing potential errors in a data set. See Petersen & Ekstrøm (2019) for background, and the vignette for examples.

DescrTab2 v2.0.3: Provides functions to create descriptive statistics tables for continuous and categorical variables. There are vignettes on Maintenance, Usage, and Validation.

libr v1.1.1: Provides functions to create data libraries, generate data dictionaries, and simulate a data step. There is an Introduction, and vignettes on library operations and management, and Data Step operations and the enhanced equality operator.

outsider v0.1.1: Allows users to install and run external command-line programs in R through use of Docker and online repositories. Look here for package information.

srcr v1.0.0: Provides a simple tool to abstract connection details, including secret credentials, out of your source code and manage configurations for frequently-used database connections. See the vignette.

Visualization

ComplexUpset v1.0.3: Provides functions to create Upset plots which offer improvements over Venn Diagrams for set overlap visualizations.

nmaplateplot v1.0.0: Provides a graphical display of results from network meta-analysis (NMA) which is suitable for outcomes like odds ratios, risk ratios, risk differences, and standardized mean differences. See the vignette for examples. vignette for examples.

PantaRhei v0.1.2: Provides functions to produce Sankey diagrams which are used to visualize the flow of conservative substances through a system. See the vignette.

SEM Time Series Modeling

Fri, 22 Jan 2021 00:00:00 +0000

Andrea Luciani is a Technical Advisor for the Directorate General for Economics, Statistics and Research at the Bank of Italy, and co-author of the bimets package.

Structural Equation Models (SEM), which are common in many economic modeling efforts, require fitting and simulating whole system of equations where each equation may depend on the results of other equations. Moreover, they often require combining time series and regression equations in ways that are well beyond what the ts() and lm() functions were designed to do. For example, one might want to account for an error auto-correlation of some degree in the regression, or force linear restrictions modeling coefficients.

In this post, we will show how to do structural equation modeling in R by working through the Klein Model of the United States economy, one of the oldest and most elementary models of its kind.

These equations define the model:

$CN_t = \alpha_1 + \alpha_2 * P_t + \alpha_3 * P_{t-1} + \alpha_4 * ( WP_t + WG_t )$

$I_t = \beta_1 + \beta_2 * P_t + \beta_3 * P_{t-1} - \beta_4 * K_{t-1}$

$WP_t = \gamma_1 + \gamma_2 * ( Y_t + T_t - WG_t ) + \gamma_3 * ( Y_{t-1} + T_{t-1} - WG_{t-1} ) + \gamma_4 * Time$

$P_t = Y_t - ( WP_t + WG_t )$

$K_t = K_{t-1} + I_t$

$Y_t = CN_t + I_t + G_t - T_t$

Given:

$CN$ as private consumption expenditure;
$I$ as investment;
$WP$ as wage bill of the private sector (demand for labor);
$P$ as profits;
$K$ as stock of capital goods;
$Y$ as gross national product;
$WG$ as wage bill of the government sector;
$Time$ as an index of the passage of time, e.g. 1931 = zero;
$G$ as government expenditure plus net exports;
$T$ as business taxes.

$\alpha_i, \beta_j, \gamma_k$ are coefficient to be estimated.

This system has only 6 equations, three of which must be fitted in order to assess the coefficients. It may not seem so difficult to solve this system, but the real complexity emerges if you look at the incidence graph in the following figure, wherein endogenous variables are plotted in blue and exogenous variables are plotted in pink.

Each edge states a simultaneous dependence from a variable to another, e.g. the WP equation depends on the current value of the TIME time series; complexity arises because in this model there are several circular dependencies, one of which is plotted in dark blue.

A circular dependency in the incidence graph of a model implies that the model is a “simultaneous” equations model and that it must be estimated by using ad-hoc procedures; moreover it can be simulated, i.e. performing a forecast, only by using an iterative algorithm.

If we search for “simultaneous equations” inside the Econometrics Task View web page we can find two results: the systemfit and the bimets packages.

The systemfit package is a powerful tool for econometric estimation of simultaneous systems of linear and nonlinear equations, but it only provides fitting procedures, thus it cannot be used in our example in order to work out a forecast.

On the other hand, the bimets package implements, among others, simulation and forecasting procedures; as stated into the vignette it allows users to write down the model in a natural way, to test several strategies and to focus on the econometric analysis, without overly dealing with coding.

Time series projection, linear restrictions and error auto-correlation can be triggered directly in the model definition, so let us try to define a similar but more complex Klein model by using a bimets compliant syntax:

#load library
library(bimets)

#define the Klein model
kleinModelDef <- "
MODEL

COMMENT> Modified Klein Model 1 of the U.S. Economy with PDL, 
COMMENT> autocorrelation on errors, restrictions and conditional equation evaluations

COMMENT> Consumption with autocorrelation on errors
BEHAVIORAL> cn
TSRANGE 1923 1 1940 1
EQ> cn =  a1 + a2*p + a3*TSLAG(p,1) + a4*(wp+wg) 
COEFF> a1 a2 a3 a4
ERROR> AUTO(2)

COMMENT> Investment with restrictions
BEHAVIORAL> i
TSRANGE 1923 1 1940 1
EQ> i = b1 + b2*p + b3*TSLAG(p,1) + b4*TSLAG(k,1)
COEFF> b1 b2 b3 b4
RESTRICT> b2 + b3 = 1

COMMENT> Demand for Labor with PDL
BEHAVIORAL> wp 
TSRANGE 1923 1 1940 1
EQ> wp = c1 + c2*(y+t-wg) + c3*TSLAG(y+t-wg,1) + c4*time
COEFF> c1 c2 c3 c4
PDL> c3 1 2

COMMENT> Gross National Product
IDENTITY> y
EQ> y = cn + i + g - t

COMMENT> Profits
IDENTITY> p
EQ> p = y - (wp+wg)

COMMENT> Capital Stock with IF switches
IDENTITY> k
EQ> k = TSLAG(k,1) + i
IF> i > 0
IDENTITY> k
EQ> k = TSLAG(k,1) 
IF> i <= 0

END
"

#load the model
kleinModel <- LOAD_MODEL(modelText = kleinModelDef)

## Analyzing behaviorals...
## Analyzing identities...
## Optimizing...
## Loaded model "kleinModelDef":
##     3 behaviorals
##     3 identities
##    12 coefficients
## ...LOAD MODEL OK

The code is quite intuitive and uses explicit keywords in order to define equations, coefficients, parameters, etc. Users can easily:

change the TSRANGE in order to fit the model in a custom time range per equation;
modify an equation EQ without changing any user procedure or code;
add or remove one or more linear restriction on the coefficients by using the keyword RESTRICT, e.g.
RESTRICT> -1.23*b2 + 8.9*b3 = 0.34
RESRTICT> b4 – 1.2*b1 = 5
add or remove an error auto-correlation structure with an arbitrary order by using the keyword:
ERROR>

Equations can contain advanced expressions, e.g.:

EQ> TSDELTA(i) = b1 + b2*EXP(p/1000) + b3*TSDELTALOG(TSLAG(p,1)) + b4*MOVAVG(TSLAG(k,1),5)

Model estimation

Now, we define time series to be used in our example, and then we perform an estimation of the whole kleinModel by using the command ESTIMATE():

#define data
kleinModelData <- list(  
  cn  =TIMESERIES(39.8,41.9,45,49.2,50.6,52.6,55.1,56.2,57.3,57.8,
                  55,50.9,45.6,46.5,48.7,51.3,57.7,58.7,57.5,61.6,65,69.7,  
                  START=c(1920,1),FREQ=1),
  g   =TIMESERIES(4.6,6.6,6.1,5.7,6.6,6.5,6.6,7.6,7.9,8.1,9.4,10.7,
                  10.2,9.3,10,10.5,10.3,11,13,14.4,15.4,22.3,   
                  START=c(1920,1),FREQ=1),
  i   =TIMESERIES(2.7,-.2,1.9,5.2,3,5.1,5.6,4.2,3,5.1,1,-3.4,-6.2,
                  -5.1,-3,-1.3,2.1,2,-1.9,1.3,3.3,4.9,  
                  START=c(1920,1),FREQ=1),
  k   =TIMESERIES(182.8,182.6,184.5,189.7,192.7,197.8,203.4,207.6,
                  210.6,215.7,216.7,213.3,207.1,202,199,197.7,199.8,
                  201.8,199.9,201.2,204.5,209.4,    
                  START=c(1920,1),FREQ=1),
  p   =TIMESERIES(12.7,12.4,16.9,18.4,19.4,20.1,19.6,19.8,21.1,21.7,
                  15.6,11.4,7,11.2,12.3,14,17.6,17.3,15.3,19,21.1,23.5, 
                  START=c(1920,1),FREQ=1),
  wp  =TIMESERIES(28.8,25.5,29.3,34.1,33.9,35.4,37.4,37.9,39.2,41.3,
                  37.9,34.5,29,28.5,30.6,33.2,36.8,41,38.2,41.6,45,53.3,    
                  START=c(1920,1),FREQ=1),
  y   =TIMESERIES(43.7,40.6,49.1,55.4,56.4,58.7,60.3,61.3,64,67,57.7,
                  50.7,41.3,45.3,48.9,53.3,61.8,65,61.2,68.4,74.1,85.3, 
                  START=c(1920,1),FREQ=1),
  t   =TIMESERIES(3.4,7.7,3.9,4.7,3.8,5.5,7,6.7,4.2,4,7.7,7.5,8.3,5.4,
                  6.8,7.2,8.3,6.7,7.4,8.9,9.6,11.6, 
                  START=c(1920,1),FREQ=1),
  time=TIMESERIES(NA,-10,-9,-8,-7,-6,-5,-4,-3,-2,-1,0,
                  1,2,3,4,5,6,7,8,9,10, 
                  START=c(1920,1),FREQ=1),
  wg  =TIMESERIES(2.2,2.7,2.9,2.9,3.1,3.2,3.3,3.6,3.7,4,4.2,4.8,
                  5.3,5.6,6,6.1,7.4,6.7,7.7,7.8,8,8.5,  
                  START=c(1920,1),FREQ=1)
);

#load time series into the model object
kleinModel <- LOAD_MODEL_DATA(kleinModel,kleinModelData)

## Load model data "kleinModelData" into model "kleinModelDef"...
## ...LOAD MODEL DATA OK

#estimate the model
kleinModel <- ESTIMATE(kleinModel, quietly=TRUE)

In order to reduce this blog post length we only show the output for a single estimation; anyhow, for each estimated equation the output is similar to the following:

kleinModel <- ESTIMATE(kleinModel, eqList='cn')

## 
## Estimate the Model kleinModelDef:
## the number of behavioral equations to be estimated is 1.
## The total number of coefficients is 4.
## 
## _________________________________________
## 
## BEHAVIORAL EQUATION: cn
## Estimation Technique: OLS
## Autoregression of Order  2  (Cochrane-Orcutt procedure)
## 
## Convergence was reached in  6  /  20  iterations.
## 
## 
## cn                  =   14.83    
##                         T-stat. 7.608    ***
## 
##                     +   0.2589   p
##                         T-stat. 2.96     *
## 
##                     +   0.01424  TSLAG(p,1)
##                         T-stat. 0.1735   
## 
##                     +   0.839    (wp+wg)
##                         T-stat. 14.68    ***
## 
## ERROR STRUCTURE:  AUTO(2) 
## 
## AUTOREGRESSIVE PARAMETERS:
## Rho          Std. Error   T-stat.      
##  0.2542       0.2589       0.9817       
## -0.05251      0.2594      -0.2024       
## 
## 
## STATs:
## R-Squared                      : 0.9827   
## Adjusted R-Squared             : 0.9755   
## Durbin-Watson Statistic        : 2.256    
## Sum of squares of residuals    : 8.072    
## Standard Error of Regression   : 0.8201   
## Log of the Likelihood Function : -18.32   
## F-statistic                    : 136.2    
## F-probability                  : 3.874e-10
## Akaike's IC                    : 50.65    
## Schwarz's IC                   : 56.88    
## Mean of Dependent Variable     : 54.29    
## Number of Observations         : 18
## Number of Degrees of Freedom   : 12
## Current Sample (year-period)   : 1923-1 / 1940-1
## 
## 
## Signif. codes:   *** 0.001  ** 0.01  * 0.05  
## 
## 
## ...ESTIMATE OK

The ESTIMATE() function can fit also non-simultaneous system and a single equation. Several predefined time series transformations are available in bimets:

– Time series extension TSEXTEND()
– Time series merging TSMERGE()
– Time series projection TSPROJECT()
– Lag TSLAG()
– Lag differences: standard, percentage and logarithmic, i.e. TSDELTA(), TSDELTAP(), TSDELTALOG()
– Cumulative product CUMPROD()
– Cumulative sum CUMSUM()
– Moving average MOVAVG()
– Moving sum MOVSUM()
– Parametric (Dis)Aggregation YEARLY(), QUARTERLY(), MONTHLY(), DAILY()
– Time series data presentation TABIT()

Forecasting

The predict() function of the lm() or dyn$lm linear model framework produces predicted values, obtained by evaluating the regression function with new data: it is a popular function among the R users.

Unfortunately, it does not help in our example: as we said before, in order to forecast a simultaneous model that presents circular dependencies in the incidence graph, we cannot merely assess the right-hand side of the equations, as the predict.lm function does; in our case we need an iterative algorithm.

The predict.lm equivalent function that allows to forecast our simultaneous model is the SIMULATE() function. On the other hand the SIMULATE() function can also solve non-simultaneous models and gives the same results as the predict.lm function.

In addition, as in the Capital Stock k equation in our example, the SIMULATE() function can conditionally evaluate an identity during a simulation, depending on the value of a logical expression (e.g. for each simulation period the k equation changes depending on the i current value). Thus, it is possible to have a model alternating between two or more equation specifications for each simulation period, depending upon results from other equations.

Structural stability, multiplier analysis and endogenous targeting are additional capabilities coded in bimets but not described in this post.

In order to forecast the model up to 1944, we need to extend exogenous time series by using the TSEXTEND() function. In this example, we perform simple extensions:

#we need to extend exogenous variables up to 1944
kleinModel$modelData <- within(kleinModel$modelData,{
    wg    = TSEXTEND(wg,  UPTO=c(1944,1),EXTMODE='CONSTANT')
    t     = TSEXTEND(t,   UPTO=c(1944,1),EXTMODE='LINEAR')
    g     = TSEXTEND(g,   UPTO=c(1944,1),EXTMODE='CONSTANT')
    k     = TSEXTEND(k,   UPTO=c(1944,1),EXTMODE='LINEAR')
    time  = TSEXTEND(time,UPTO=c(1944,1),EXTMODE='LINEAR')
  })

A call to the SIMULATE() function will solve our simultaneous system of equations:

#forecast model
kleinModel <- SIMULATE(kleinModel
                      ,simType='FORECAST'
                      ,TSRANGE=c(1941,1,1944,1)
                      ,simConvergence=0.00001
                      ,simIterLimit=100
                      ,quietly=TRUE
  )

The historical GNP (original referred as “Net national income, measured in billions of 1934 dollars” , pg. 141 in “Economic Fluctuations in the United States 1921-1941” by L. R. Klein, Wiley and Sons Inc., New York, 1950) is shown in figure, along with the simulation and the forecast.

#get forecasted GNP
TABIT(kleinModel$simulation$y)

## 
##       DATE, PER, kleinModel$simulation$y
## 
##       1941, 1  ,  125.3      
##       1942, 1  ,  172.5      
##       1943, 1  ,  185.6      
##       1944, 1  ,  141.1

A Custom Forest Plot from Wonderful Wednesdays

Fri, 15 Jan 2021 00:00:00 +0000

Waseem Medhat is a Statistical Programmer and Computational Experimentalist who resides in Alexandria, Egypt

This post takes a closer look at the forest plot that was mentioned in a previous post introducing PSI’s Wonderful Wednesdays events. It describes a custom version of a forest plot with additional bands to visualize heterogeneity between studies in a meta-analysis that was part of a project submitted to the Wonderful Wednesdays challenge hosted by PSI and reviewed by statisticians in the organization. Find more information here. The plot is built with JavaScript using the D3.js library and wrapped in a Shiny app with the help of the R2D3 package.

Background problem

The problem around this visualization is specific to meta-analysis, which is the statistical pooling of the results of multiple studies (e.g. a multi-center clinical trial) to obtain a single, more powerful estimate. The choice of pooling model (fixed effect vs. random effects) depends on the heterogeneity of effect size between studies. So, the main question that I wanted to answer with this visualization is:

“What graphical tools can be used to assess heterogeneity?”

Like any statistical graphic, the purpose of the visualization is to complement the statistical measures of heterogeneity, like I^2^, to give a more complete picture.

Plot description

A lot of the plot components are directly comparable to the typical forest plot, which is very popular in the medical field as visualization tool in meta-analyses. Its main features are:

A square for each point estimate for a study. The square size is proportional to the sample size (i.e. weight) of that study.
A line for each confidence interval of the effect size in a study.
Diamonds that represent the pooled estimate using either fixed-effect or random-effects model. The diamond width represents the confidence interval around the pooled estimate.
The plot is usually combined with a tabular display of the numbers represented by the plot.

My own additions are:

Colored bands to give a better visualization of the heterogeneity between studies. There is a band for each study, with a width equal to its confidence interval. All the bands are semi-transparent and overlayed over each other so that more overlapping produces darker areas.
More attention to annotations than the typical plots, providing a title and a subtitle with the interventions and the outcome, respectively. Another label is also added to show which direction represents the “positive” effect.

Shiny app description

As a proof of concept for a viable product, I wrapped the plot in a Shiny app which provides additional interactive features:

Selection of summary measure, which can be expanded to include more than the odds ratio and risk ratio.
Control over the plot dimensions. Giving this control to the user allows the plot to be conveniently visible in different screen sizes and deliverable forms (e.g. a report or a dashboard).
Help button that shows a guide for interpretation. This makes the information available on-demand instead having it a separate tab.

Technologies and packages

D3.js (JavaScript)

The plot itself (and associated tabular display) was built using D3. Being a JavaScript library, D3 works with web technologies: HTML, CSS, and especially SVG. It has a lot of low-level tools that bind data to SVG shapes and change the shape properties accordingly. One particular advantage of using web technologies in this visualization is that CSS allows semi-transparent elements to “blend” colors in multiple ways, which allowed me to choose a blend mode that emphasizes the overlap.

R2D3

R2D3 was the main wrapper around the D3 visualization. Beside the obvious advantage of introducing an interface between R and D3 and allowing its rendering in Shiny, it makes some steps easier like giving the data to the plot and making the plot take as much space as possible inside its container. Because of this, initial variables like data, width, height, and (the container) svg are provided by R2D3 and are not declared in the JavaScript code.

Shiny

It does not need an introduction at this point: Shiny is the de facto standard for R-based web applications. With the R2D3 package, I have available d3Output() and renderD3() to render the D3 plot just like any typical output in Shiny. Other Shiny packages I used are shinyhelper, which provides a help button and rich modal dialogs for help content, and shinythemes to change the appearance of the app.

By wrapping the visualization in a Shiny app, this project became a prototype that can be taken in a lot of different directions to include more effect sizes and other meta-analysis techniques, or maybe even add another module that imports the data, performs the meta-analysis, and send the results to this module to be visualized.

Wonderful Wednesdays

Mon, 11 Jan 2021 00:00:00 +0000

For almost a year now, the PSI Visualization Special Interest Group (VIS SIG) has been conducting a monthly graduate-level seminar on creating effective statistical visualizations that is open to everyone. Wonderful Wednesdays is a unique collegial event. Every month the SIG publishes a link to a new data set and issues a challenge to produce visualizations that effectively illustrate some specific aspects of the data. Anyone can submit an entry coded in the language of their choice. Submissions that are received by the deadline are then critiqued in free webinar that takes place roughly thirty days later. You don’t have to make a submission to attend the webinar.

The process is well organized and straightforward. The figure below illustrates the process and time liness for the for the webinar that will happen this week on Wednesday, January 13th.

Click here to register for the webinar.

Here is the link to the December webinar where the challenge was to visualize the heterogeneity among the data used for a meta-analysis of seven studies undertaken to show a reduction in hypertension.

The following image is a traditional forest plot showing a comparison of the odds ratios for the seven studies. (If you are not familiar with this plot type look here for some tips on how to interpret it.)

This image comes from a shiny app that is critiqued at the by the VIS-SIG statisticians towards the beginning of the webinar. The experts liked the clean look and labeling on the plot, but had mixed feelings about the colored bands which are meant to show regions where the studies overlap. (Darker color indicates more overlap.) Here are the links to the Shiny App and the code and also to the blog post that reviews all of the submissions for the December challenge.

Visit the VIS-SIG Blog to finds posts and code for the submissions for all of Wonderful Wednesday events so far.

COVID-19 Data: The Long Run

Wed, 06 Jan 2021 00:00:00 +0000

The world seems to have moved to a new phase of paying attention to COVID-19. We have gone from pondering daily plots of case counts, to puzzling through models and forecasts, and are now moving on to the vaccines and the science behind them. For data scientists, however, the focus needs to remain on the data and the myriad issues and challenges that efforts to collect and curate COVID data have uncovered. My intuition is that not only will COVID-19 data continue to be important for quite some time in the future, but that efforts to improve the quality of this data will be crucial for successfully dealing with the next pandemic.

An incredible amount of work has been done by epidemiologists, universities, government agencies and data journalists to collect, organize, and reconcile data from thousands of sources. Nevertheless, the experts caution that there is much yet to be done.

Roni Rosenfeld, head of the Machine Learning Department of the School of Computer Science at Carnegie Mellon University and project lead for the Delphi Group put it this way in a recent COPSS-NISS webinar:

Data is a big problem in this pandemic. Availability of high quality, comprehensive, geographically detailed data is very far from where it should be.

There are over 6,000 hospitals in the United States, and over 160,000 hospitals worldwide. Many of these are collecting COVID-19 data yet there few standards for recording cases, dealing with missing data, updating case count data, and coping with the time lag between recording and reporting cases. Nowcasting epidemiological and heath care data has become a vital field of statistical research.

The following slide from the COPSS-NISS webinar shows a hierarchy of relevant COVID data organized on the Severity Pyramid that epidemiologists use to study disease progression.

The Delphi Group is making fundamental contributions to the long term improvement of COVID data by archiving the data shown in such a way that versions can be retrieved by date, and also by collecting massive data sets of leading indicators.

The webinar is well worth watching, and I highly recommend listening through the Q&A session at the end. The speakers explain the importance of nowcasting and Professor Rosenfeld presents a vision of making epidemic forecasting comparable to weather forecasting. It seems to me that this would be a worthwhile project to help advance.

Note that the Delphi’s COVID-19 indicators, probably the nation’s largest public repository of diverse, geographically-detailed, real-time indicators of COVID activity in the US, are freely available through the public API which is easily accessible to R and Python users.

Also note that R users can contribute to R Consortium sponsored COVID related projects that include the COVID-19 Data Hub an organized archive of global COVID-19 case count data, and the RECON COVID-19 Challenge, an open source project to improve epidemiological tools.

November: "Top 40" New CRAN Packages

Tue, 22 Dec 2020 00:00:00 +0000

Two hundred ninety-two new packages made it to CRAN in November. Picking forty was unusually difficult. Nevertheless, here are my “Top 40” selections in twelve categories: Archaeology, Computational Methods, Data, Epidemiology, Games, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Utilities, and Visualization. R developers continue to extend the reach of R. November featured a new package on Archaeology, one of only seventeen I could find on CRAN pkgsearch::pkg_search(query="Archaeology ",size=200), as well as a package that wraps Python’s chess package.

Looking back over the last twelve months my impression is that R continues to grow in the life sciences. Packages that I have classified as belonging to the categories Epidemiology, Genomics, or Medicine have comprised between ten and fourteen percent of the packages I have reviewed each month.

Archaeology

archeofrag v0.6.0: Implements methods based on graphs and graph theory for the stratigraphic analysis of fragmented objects in archaeology using “refitting” relationships between fragments scattered in stratigraphic layers. See the vignette.

Computational Methods

ADtools v0.5.4: Implements the forward-mode automatic differentiation for multivariate functions using the matrix-calculus notation from Magnus and Neudecker (2019). See the vignette for an introduction.

ML2Pvae v1.0.0: Provides functions to create a variational autoencoder (VAE) for parameter estimation in Item Response Theory (IRT) which allows straight-forward construction, training, and evaluation. Only minimal knowledge of tensorflow or keras is required. See Curi et al. (2019) for background and the vignette for an overview.

Data

campfin v1.0.4: Provides tools to explore and normalize American campaign finance data. This package was created by the Investigative Reporting Workshop to facilitate work on The Accountability Project. See the vignette to get started.

cpsvote v0.1.0: Provides automated methods for downloading, recoding, and merging selected years of the Current Population Survey’s Voting and Registration Supplement, a large national survey about registration, voting, and non-voting in United States federal elections. There are vignettes on basics, background, voting, and adding variables.

geogenr v1.0.0: Allows users to access geodatabasees and obtain information from the American Community Survey (ACS). See the vignette to get started.

openSkies v0.99.8: Provides a client interface to the OpenSky API that allows retrieval of flight information, as well as aircraft state vectors. See the vignette.

salem v0.2.0: Access data on all 152 accused witches from the 1692 Salem Witch Trials. There is an Introduction and a vignette reproducing an analysis.

Epidemiology

oxcgrt v0.1.0: Implements an interface to the Oxford COVID-19 Government Response Tracker (OxCGRT). There are vignettes on calculating incices and retrieving data.

PandemicLP v0.2.0: Implements the CovidL methodology for long-term epidemic and pandemic prediction. There is an Introduction and a case study.

SEIRfansy v1.1.0: Implements the Extended Susceptible-Exposed-Infected-Recovery Model for handling high false negative rate and symptom based administration of diagnostic tests. See Bhaduri et al. (2020) and the GitHub site for examples.

trendeval v0.0.1: Provides a coherent interface for evaluating models fit with the RECON trending package. See README to get started.

Games

chess v1.0.1: Implements an “opinionated” wrapper around the python-chess library allowing users to read and write PGN files as well as create and explore game trees such as the ones seen in chess books. See the vignettes chess, games, and advanced.

codebreaker v0.0.2: Inspired by Mastermind, the package implements a logic game in the style of the early 1980s home computers that can be played in the R console. Can you break the code? See README to start playing.

Machine Learning

fastai v2.0.2: Implements functions to simplify training neural networks based on best practices developed at fast.ai. See the website to get started and the twenty-three vignettes which include Audio Classification, Multilabel Classification and Medical Images.

mikropml v0.0.2: Implements the ML pipeline described in Topçuoğlu et al. (2020) For building machine learning models for classification and regression problems. There is an Introduction and an Overview.

stacks v0.1.0: Implements a grammar of model stacking for tidymodels. There is a Getting Started Guide and a vignette on classification.

Mathematics

BaseSet v0.0.14: Implements a class and methods to work with sets, doing intersection, union, complementary sets, power sets, cartesian product and other set operations in a “tidy” way. See the Introduction, and the vignettes Advanced Examples and Fuzzy Sets.

viscomplexr v1.1.0: Provides functions to create phase portraits of functions in the complex number plane. See the vignette to get started.

Medicine

Statistics

ceser v1.0.0: Implements the Cluster Estimated Standard Errors method proposed in Jackson (2020) to compute clustered standard errors of linear coefficients in regression models with grouped data. See the vignette.

gfilmm v2.0.2: Implements generalized Fiducial inference for normal linear mixed models. Fiducial inference is similar to Bayesian inference in the sense that it represents the uncertainty about the parameters with a probability distribution. However, it does not require a prior. See Cisewski and Hannig (2012) for background and the vignette for examples.

hdpGLM v1.0.0: Implements MCMC algorithms to estimate the Hierarchical Dirichlet Process Generalized Linear Model presented in paper Ferrari (2020). See the vignette for examples.

latrend v1.0.1: Implements a framework for clustering longitudinal datasets in a standardized way. There is a Demo vignette and vignettes on implementing new models and validating cluster models.

mixComp v0.1-1: Implements methods to estimate the order of mixture distributions. See the vignette for an introduction to mixture models and and extended list of references.

monoClust v1.2.0: Implements the monothetic clustering algorithm for continuous data described in Chavent (1998). See the vignette.

potential v0.1.0: Implements the potential model for measuring social influences described in Stewart (1941). See the vignette for an introduction.

sftrack v0.5.2: Implements classes for tracking and movement data, building on sf spatial infrastructure, and early theoretical work from Turchin (1998), and Calenge et al. (2009). There is an Overview along with the vignettes Reading in an sftrack, Structure, Fantastic Groups, and Getting Spatial.

simrec v1.0.0: Provides functions to simulate recurrent event data with a non-constant baseline hazard and possibly risk-free intervals and competing events. See Jahn-Eimermacher et al. (2015) for background and the vignette for an introduction.

Time Series

modeltime.resample v0.1.0: A modeltime extension which implements forecast resampling tools to asses time-based model performance and stability for time series, panel data, and cross-sectional time series. There is a Getting Started guide and a vignette on Resampling.

tfarima v0.1.1: Provides tools to build customized transfer functions and ARIMA models with multiple operators and parameter restrictions. see Bell & Hilmer (1983) and Box & Tiao (1973) for background and the vignette for some theory and examples.

Utilities

getDTeval v0.0.1: Provides functions to translate statements that use get() or eval() to improve run-time efficiency. See the vignette.

lineup2 v0.2-5: Provides tools for detecting and correcting sample mix-ups between two sets of measurements, such as between gene expression data on two tissues. There is a vignette.

sdcLog v0.1.0: Tools for researchers to explicitly show that their results comply to rules for statistical disclosure control imposed by research data centers. The methods used are described in Bond et al. (2015). There is an Introduction and a vignette on options.

Visualization

leaflet.multiopacity v0.1.1: Extends leaflet by adding a widget to control the opacity of multiple layers. There are vignettes for using the package with leaflet and leafletProxy.

mapboxer v0.4.0: Provides access to Mapbox GL JS, an open source JavaScript library that uses WebGL to render interactive maps via the htmlwidgets package. Visualizations can be used from the R console, in R Markdown documents and in Shiny apps. See the vignette to get started.