Opinion on R Views

May 2023: "Top 40" New CRAN Packages

Wed, 28 Jun 2023 00:00:00 +0000

This will be my final R Views post offering my totally idiosyncratic selection of 40 CRAN packages. It has been a good run that began in August 2016. I would like to thank all of you who followed these posts. Your readership kept me going. I hope that the “Top 40” posts were somewhat useful for helping you to keep up with the explosion of creativity in the R Community.

I would also like to thank the guest authors who contributed posts to R Views over the years. It has been a pleasure and privilege to work with all of you. Please find a way to keep writing about R. When you do have something to share on R Views, I am certain you will enjoy working with R Views’ new senior editor, Isabella Velásquez as I have. It has been my great pleasure to collaborate with Isabella.

One hundred ninty-eight new packages stuck to CRAN in May. (I identified eight packages that were removed between the end of may and the 24th of June.) Here are my “Top 40” selections in fifteen categories Astronomy, Biology, Chemistry, Climate Science, Computational Methods, Data, Ecology, Engineering, Finance, Genomics, Mathematics, Medicine, Statistics, Utilities, and Visualization.

Astronomy

snvecR v3.7.7: Provides functions to calculate precession and obliquity from an orbital solution and assumed or reconstructed values for tidal dissipation and dynamical ellipticity. See Zeebe and Lourens (2022) for background and the vignette for examples.

Biology

dwctaxon v2.0.2: Provides tools to edit and validate taxonomic data in compliance with Darwin Core standards (Darwin Core Taxon class). There is an Indroduction, an Example, and there are vignettes on Editing and Validating texon data.

Chemistry

anabel v3.0.1: Implements tools suitable for high-throughput analysis of 1:1 molecular interaction studies. See Kraemer et al. (2019) and Norval et al. (2019) for background and the vignette for examples.

Climate Science

myClim v1.0.1: Provides functions for working with microclimate data including reading, converting data formats, sorting into localities, computing microclimate variables, and plotting. See the vignette.

Computational Methods

matrixset v0.1.1: Provides functions to create, store, manipulate and code with matrix ensembles, matrices that share common properties. See the Gentle Introduction, the example, and the vignettes on annotation, applying functions.

zonohedra v0.2-2: Provides functions to compute and plot zonohedra from vector generators. See Hecbert (1985) for the optimization methods. Vignettes include a User Guide, Matroids, Ray tracing zonohedron boundaries, Transition subcomplex and surface, and Zonotopes.

woyler v0.0.5: Provides functions for generating a tour path by interpolating between d-D frames in p-D space using Givens rotations. See Buja et al (2005) for background on high dimensional rotations, the Introduction and the vignette on Plotting.

Data

appeears v1.0: Implements a programmatic interface to the NASA Application for Extracting and Exploring Analysis Ready Samples services (AppEEARS) providing easy access to analysis ready earth observation data. See the vignette.

BRVM v5.0.0: Provide real-time access to data from the Regional Securities Exchange SA, commonly known as the BRVM stock exchange.

denguedatahub v1.0.4: Provides a weekly, monthly, yearly summary of dengue cases by state, province, or country. Look here for information on the data provided.

Ecology

forestat v1.0.1: Provides functions for evaluating the quality of a natural forest including tree height classification, tree height model establishment, sectional area growth and stock growth modeling, and for calculating actual and potential forest productivity. see the vignette.

hemispheR v0.2.0: Provides functions for processing hemispherical canopy images including functions to import and classify canopy fish-eye images, estimate angular gap fraction and derive canopy attributes like leaf area index and openness. See Chianucci & Macek (2023) for details and README for examples.

traitstrap v0.1.0: Calculates trait moments from trait and community data using the methods developed in Maitner et al (2021). See the vignette for an introduction and the playlist, a definite innovation in R package vignettes.

Engineering

navigation v0.0.1: Implements the framework presented in Cucci et al. (2023) which allows analyzing the impact of sensor error modeling on the performance of integrated navigation (sensor fusion) based on inertial measurement unit (IMU), Global Positioning System (GPS), and barometer data. See the vignettes Compare Models and Model Evaluation.

Finance

FinNet v0.1.1: Provides classes, methods. and functions to build and manipulate financial networks that include multiple types of relationships including ownership, board interlocks and legal entities. Look here for detailed information.

intradayModel v0.0.1: Provides functions to model, analyze, and forecast intraday signals via a univariate state-space model for intraday trading volume provided by Chen (2016). See the vignette.

StockDistFit v1.0.0: Provides functions for fitting probability distributions to stock price data using maximum likelihood estimation to find the best-fitting distribution for a given stock. See Siew et al. (2008) for background and the vignette for details.

Genomics

ApackOfTheCLones v0.1.2: Provides functions to visualize T-cell clonal expansion via circle-packing that can integrate single-cell RNA sequencing and T-cell receptor libraries from the 10X genomics Single Cell Immune Profiling and Cell Ranger pipeline, to produce a simple, publication-ready visualization of the clonal expansion. See README for examples.

baldur v0.0.1: Implements a hierarchical Bayesian model for statistical decisions in proteomics that includes two regression models for describing the mean-variance trend, a gamma regression or a latent gamma mixture regression. The regression model is then used as an Empirical Bayes estimator for the prior on the variance in a peptide. See Berg & Popescu (2023) for the theory and the UPS and Yeast tutorials.

Mathematics

resde v1.1: Implements maximum likelihood estimation for univariate reducible stochastic differential equation models for discrete, possibly noisy observations, not necessarily evenly spaced in time. See Garcia (2019) for the theory and the vignette for some math and examples.

Medicine

PatientProfiles v0.2.0: Provides methods to identify the characteristics of patients in data mapped to the Observational Medical Outcomes Partnership (OMOP) common data model. See the vignettes Get cohort intersections and Get cohort patients characteristics.

PANACEA v1.0.0: Provides a collection of personalized anti-cancer drug prioritization approaches utilizing network methods that utilize personalized “driverness” scores from driveR to rank drugs and map them onto a protein-protein interaction network. See the vignette to get started.

Statistics

boostingDEA v0.1.0: Provides functions to estimate the production frontier and predict ideal output in the Data Envelopment Analysis (DEA) context using both standard models from DEA and Free Disposal Hull (FDH) and boosting techniques. See Guillen et al. (2023) for background and the vignette for examples.

gFormulaMI v1.0.0: Implements the G-Formula method for causal inference with time-varying treatments and confounders using Bayesian multiple imputation methods, as described by Bartlett et al. (2023). See the vignette.

nestedLogit v0.3.2: Provides functions for specifying and fitting nested dichotomy logistic regression models for a multi-category response, and methods for summarizing and plotting those models. There are vignettes on Nested dichotomies Logit models, Plotting, and Standard Errors.

nlpsem v0.1.1: Provides computational tools for intrinsically nonlinear longitudinal models in multiple scenarios including multivariate models with time varying covariates, latent class, and a goal to assess correlation or causation among multiple longitudinal variables. Liu (2023) elaborates on the details. The vignettes provide examples for Latent Change Score Models, Latent Growth Curve Models, Multivariate Longitudinal Models, Longitudinal Mixture Models, Longitudinal Mediation Models, and Time-varying Covariate Models.

npcurePK v1.0-2: Provides functions to perform nonparametric estimation in mixture cure models when the cure status is partially known. See Safari et al. (2021), Safari et al. (2022), and Safari et al. (2023) and the vignette for an example.

semlbci v0.10.3: Implements likelihood-based confidence intervals for parameters in structural equation modeling. See Cheung and Pesigan (2023), Pek and Wu (2018), and Falk (2018) for details. There are vignettes: Get Started, Log Profile Likelihood of a Parameter, and Searching for One Bound.

ssMRCD v0.1.0: Provides functions to estimate the Spatially Smoothed Minimum Regularized Determinant (ssMRCD) estimator and use it for outlier detection as described in Puchhammer and Filzmoser (2023) See the Introduction.

survstan v0.0.2: Implements parametric survival regression models under the maximum likelihood approach via Stan including accelerated failure time models, proportional hazards models, proportional odds models, accelerated hazard models, and Yang and Prentice models. See Bennett (1982, Chen and Wang(2000), Demarqui and Mayrink (2021) for background, README for some math, and the vignette for an example.

voi v1.0: Implements methods to calculate the expected value of information from a decision-analytic model. See Jackson et al. (2022) for the theory, the Package Overview and the vignette on Plots.

Utilities

crc32c v0.0.2: Implements hardware-based support for CRC32C cyclic redundancy checksum function for x86_64 systems with SSE2 support as well as for the arm64. The functionality is exported at the C-language level for use by other packages. See README for some background.

glossary v1.0.0: Implements tools to add glossaries to markdown and quarto documents by tagging individual words. Definitions can be provided inline or in a separate file. See the vignette.

iterors v1.0: Attempts to improve the performance of iterators in R. The package is cross-compatible with iterators and includes a collection of iterator constructors and combinators ported and refined from the iterators, itertools, and itertools2 packages. There are several vignettes including an Introduction, a Guide on using the package, and a vignette on Writing Custom Iterators..

rebib v0.2.2: Implements tools to convert embedded LaTeX bibliographies into a close BibTeX equivalent that is useful for web publishing systems (such as the R Journal) that do not support embedded bibliographies. See the Introduction and the vignettes general usage, parser mechanics, and use with RJ article.

sasr v0.1.2: Provides a SAS interface that enables creating SAS sessions, executing SAScode on remote SAS servers, exchanging data sets between SAS and R and more. See the vignette.

tidytlg v0.1.2: Implements tools to generate tables, listings, and graphs (TLG) using the tidyverse. Tables can be created functionally, using a standard process, or by specifying table and column metadata to create generic analysis summaries. There are several vignettes including examples for Freauency Analysis, Table Manipulation, and a Get Started guide.

Visualization

ggautomap v0.3.2: Extends ggplot2 to convert place names to coordinates without having to work directly with the underlying geospatial tools. See the Getting Started Guide.

ggplend v0.1.0: v0.1.0: Implements an algebra of operations for blending, copying, adjusting, and compositing layers in ggplot2 that supports copying and adjusting the aesthetics or parameters of an existing layer, partitioning a layer into multiple pieces for re-composition, applying affine transformations to layers, and combining layers. See README for examples.

ggtricks v0.1.0: Provides collection of ggplot2() geoms to create graphics using Cartesian coordinate system and avoid the use of polar coordinates. Look here for examples.

isopleuros v1.0.0: Provides functions to display data in ternary space, to add or tune graphical elements and to display statistical summaries. It also includes common ternary diagrams which are useful for archaeologists. See README for examples.

tidygam v0.2.0: Provides functions that compute predictions from Generalized Additive Models (GAMs) fitted with mgcv and return them as a tibble. See the vignette to get started.

April 2023: "Top 40" New CRAN Packages

Thu, 25 May 2023 00:00:00 +0000

One hundred fifty-six new packages made it to CRAN in April. Here are my “Top 40” selections in twelve categories: Computational Methods, Data, Ecology, Economics, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

clarabel v0.4.1: Implements Clarabel, a versatile interior point solver that solves linear programs, quadratic programs, second-order cone programs, and problems with exponential and power cone constraints. See the vignette.

condor v1.0.0: Provides functions to access the Condor high performance computing environment. Files are first uploaded to a submitter machine and the resulting job is then passed on to Condor. Look here for the code.

GPUmatrix v0.1.0: Extends R to use GPUs for matrix computations. See the vignette.

hydroMOPSO v0.1-3: Implements a state-of-the-art Multi-Objective Particle Swarm Optimiser (MOPSO), based on the algorithm developed by Lin et al. (2018) with improvements described by Marinao-Rivas & Zambrano-Bigiarini (2020) which can be used for global optimization of non-smooth and non-linear R functions and other models that need to be run from the system console, e.g. SWAT+.

Data

dataverifyr v0.1.5: Provides a thin wrapper around dplyr, data.table, arrow, and DBI to allow users to define rules which can be used to verify a given dataset. See README to get started.

neotoma2 v1.0.0: Provides functions to access and manipulate data in the Neotoma Paleoecology Database. See the vignette.

rpaleoclim v1.0.0: Implements an interface to PaleoClim, a set of free, high resolution paleoclimate surfaces covering the whole globe that includes data on surface temperature, precipitation and the standard bioclimatic variables commonly used in ecological modelling. See Brown et al. (2019) for background and the vignette.

zctaCrosswalk v2.0.0: Contains the US Census Bureau’s 2020 ZCTA to County Relationship File, as well as convenience functions to translate between States, Counties and ZIP Code Tabulation Areas (ZCTAs). See the Introduction and the vignettes Workflow with tidycensus, and Developer Notes.

Ecology

EWSmethods v1.1.2: Implements methods for forecasting tipping points at the community level that include rolling and expanding window approaches to assessing abundance based early warning signals, non-equilibrium resilience measures, and machine learning. See Dakos et al. (2012), Deb et al. (2022), and Drake and Griffen (2010) for background and the vignette for an introduction.

fqacalc v1.0.0: Provides functions for calculating Floristic Quality Assessment (FQA) metrics using regional FQA databases that have been approved or approved with reservations as ecological planning models by the U.S. Army Corps of Engineers (USACE). For information on FQA see Spyreas (2019). There is an Introduction.

Economics

clptheory v0.1.0: Provides functions to compute the uniform rate of profit, the vector of price of production and the vector of labor values, and also compute measures of deviation between relative prices of production and relative values. See Basu and Moraltis (2023) for background and README for an introduction.

Genomics

BREADR v1.0.1: Implements a method for estimating degrees of relatedness for extreme low-coverage genotype data and includes functions to quantify and visualize the level of confidence in the estimated degrees of relatedness. See Rohrlach et al. (2023) for package details and README for examples.

crosshap v1.2.2: Implements a local haplotyping visualization toolbox to capture major patterns of co-inheritance between clusters of linked variants, while connecting findings to phenotypic and demographic traits across individuals. See Marsh et al. (2022) for a detailed example and README for an introduction.

DAISIEprep v0.3.2: Extracts colonization and branching times of island species for analysis with the DAISIE package. There is a Tutorial and there are vignettes on Performance and Sensitivity.

Machine Learning

CCMMR v0.1: Implements the convex clustering through majorization-minimization algorithm described in Touw, Groenen, and Terada (2022) to minimize the convex clustering loss function. See README for examples.

rcccd v0.3.2: Provides functions to fit class cover catch digraph classification models. Methods are explained in Priebe et al. (2001), Priebe et al. (2003), and Manukyan and Ceyhan (2016). README contains some description.

TheOpenAir v0.1.0: Implements a wrapper using the OpenAI API as a back end to integrate ChatGPTinto diverse data-related tasks, such as data cleansing and automating analytics scripts. See README to get started.

Mathematics

cyclotomic v1.1.0: Implements algorithms from the GAP project to work with cyclotomic numbers: complex numbers that can be thought of as the rational numbers extended with the roots of unity. They have applications in number theory, algebraic geometry, algebraic number theory, coding theory, in the theory of graphs and combinatorics, and in the theory of modular functions and modular curves. See README for examples.

markovmix v0.1.1: Provides functions to fit a mixture of Markov chains of higher orders from multiple sequences along with various utility functions to derive transition patterns, transition probabilities per component and component priors. See README for examples.

Medicine

DiDforBigData v1.0: Provides a big-data-friendly and memory-efficient difference-in-differences estimator for staggered (and non-staggered) treatment contexts. See the Get Started Guide the vignettes Background, Examples, and Theory.

predictNNB v0.1.0: Provides tools to estimate when and where a model-guided treatment strategy may outperform a treat-all or treat-none approach using Monte Carlo simulation and evaluation of the Net Monetary Benefit. See Parsons et al. (2023) for details, the Introduction, and the vignettes on creating functions, summarising results, and detailed example.

predRupdate v0.1.0: Provides functions to evaluate the predictive performance of existing clinical prediction model given a new dataset. See Su et al. (2018), Debray et al. (2014), and Martin et al. (2018) for background and the vignettes Introduction and Technical Background.

SPARRAfairness v0.0.0.1: Provides functions to analyse the behavior and performance of the Scottish Patients At Risk of admission and Re-Admission risk score which estimates yearly risk of emergency hospital admission using electronic health records for most of the Scottish population. Analysis focuses on differential performance over demographically-defined groups. See the vignette.

Science

kronos v1.0.0: Implements a framework to analyse circadian or otherwise rhythmic data using the familiar R linear modelling syntax, while taking care of the trigonometry under the hood. Look here for examples.

mpmsim v1.0.0: Provides functions to to simulate matrix population models with particular characteristics based on aspects of life history such as mortality trajectories and fertility trajectories, and allows the exploration of sampling error due to small sample size. See the vignettes on robustness, sampling error & propagation, and PCA.

Statistics

BGFD v0.1: Implements the probability density function, cumulative distribution function, quantile function, random numbers, survival function, hazard rate function, and maximum likelihood estimates for the family of Bell-G and Complementary Bell-G distributions. See Fayomi et al. (2022), Alanzi et al.(2023), and Algarni (2022) for details.

D3mirt v1.0.3: Provides functions for identifying, estimating, and plotting descriptive multidimensional item response theory models, restricted to 3D and dichotomous or polytomous data that fit the two-parameter logistic model or the graded response model. See the vignette for an extensive introduction.

funStatTest v1.0.2: Implements two sample comparison procedures based on median-based statistical tests for functional data, described in Smida et al. (2022), Chakraborty and Chaudhuri (2015), Horvath et al. (2013, and Cuevas et al. (2004). See the vignette for examples.

lessSEM v1.4.16: Provides regularized structural equation modeling (regularized SEM) with non-smooth penalty functions (e.g., lasso) building on lavaan. There are nine vignettes including: lessSEM, The Structural Equation Model, and Mixed Penalties.

panelhetero v1.0.0: Provides tools for estimating the degree of heterogeneity across cross-sectional units in the panel data analysis using the methods developed by Okui and Yanagi (2019) and Okui and Yanagi (2020). See the vignette.

tdsa v1.0-1: Provides functions to perform time-dependent sensitivity analysis by calculating time-dependent state and parameter sensitivities for both continuous- and discrete-time deterministic models. See Ng et al. (in review) for background and the vignette to get started.

Utilities

crew.cluster v0.1.0: Extends the mirai-powered crew package with worker launcher plugins for traditional high-performance computing systems to enable statisticians and data scientists to asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. Look here to get started.

duke v0.0.1: Provides functions to generate visualizations with Duke’s official suite of colors in a color blind friendly way. There is an Overview and four additional vignettes including one on the theme_duke() function.

grateful v0.2.0: Facilitates the citation of R packages used in analysis projects by providing functions to scan projects for packages used and produces documents with citations in the preferred bibliography format. Functions may be used within rarkdownor quarto documents. See README for examples.

hightR v0.3.0: Implements the HIGHT block cipher encryption algorithm developed to provide confidentiality in low power consumption computing environments such Radio-Frequency Identification and Ubiquitous Sensor Network. Look here for more information.

myCRAN v1.0: Provides functions to plot the daily and cumulative number of downloads of R packages, obtaining daily and cumulative counts in one run. See the vignette.

woodendesc v0.1.0: Provides functions to simplify obtaining available packages, their version codes and dependencies from any R repository. Uses extensive caching for repeated queries. See READMEfor examples.

Visualization

fxl v1.6.3: Provides functions to prepare and design single case design figures that are typically prepared in spreadsheet software. See the vignette for theory and examples.

ggragged v0.1.0: Extends ggplot2 facets to panel layouts arranged in a grid with ragged edges with rows and columns of potentially varying lengths. These may be useful in representing nested or partially crossed relationships between faceting variables. See README for examples.

nndiagram v1.0.0: Generates LaTeX code for drawing well-formatted neural network diagrams with TikZ. Users define the number of neurons on each layer, neuron connections to keep or omit, layers considered to be oversized, and neurons to draw with lighter color. See README for instructions.

PlotTools v0.2.0: Provides functions to manipulate irregular polygons and annotate plots with legends for continuous variables and color spectra using the base graphics plotting tools. See README for an example.

March 2023: "Top 40" New CRAN Packages

Fri, 28 Apr 2023 00:00:00 +0000

Accounting

debkeepr v0.1.1: Provides tools to analyze historical, non-decimal currencies and value systems that use tripartite or tetrapartite systems such as pounds and shillings in the context of double-entry bookkeeping. See the Getting Started guide and the vignettes Analysis of Richard Dafforne’s Journal and Ledger and Transactions in Richard Dafforne’s Journal.

Computational Methods

ABM v0.3: Implements a high-performance, flexible and extensible framework to develop continuous-time agent based models capable of simulating millions of agents in which state transitions may be either spontaneous or caused by agent interactions. See README for multiple examples including Simulate an agent based SEIR model and Simulate contact tracing on an SIR model.

rvMF v0.0.7: Provides functions to generate pseudo-random vectors that follow an arbitrary von Mises-Fisher distribution on a sphere including functions to generate random variates, compute the density for the distribution of an inner product between von Mises-Fisher random vector and its mean direction. Look here for an example.

Data

FertNet v0.1.1: Provides tools to processes data from The Social Networks and Fertility Survey including functions for correcting respondent errors and for transforming network data into network objects to facilitate analyses and visualization. See README to get started.

oldbailey v1.0.0: Provides functions to fetch trial data from the Old Bailey Online API. Data includes the names of the first person speakers, defendants, victims, their recorded genders, verdicts, punishments, crime locations, and dates. Look here for an example.

webtrackR v0.0.1: Implements data structures and methods to work with web tracking data, including data preprocessing steps, methods to construct audience networks as described in Mangold & Scharkow (2020) and metrics of news audience polarization described in Mangold & Scharkow (2022). Look here to get started.

Ecology

GIFT v1.0.0: Provides functions to retrieve regional plant checklists, species traits and distributions, and environmental data from the Global Inventory of Floras and Traits database and to visualize the map of available flora. There is an introductory Tutorial, an Advanced Tutorial, and a vignette on Queries.

rTLsDeep v0.0.5: Uses terrestrial laser scanning and deep learning to classify post-hurricane damage severity at the individual tree level. See Klauberg et al. (2023) for details, and look here for an example.

Finance

HSRFA v0.1.1: Implements two algorithms to do robust factor analysis by considering the Huber loss: one is based on minimizing the Huber loss of the idiosyncratic error’s L2 norm, the other is based on minimizing the element-wise Huber loss. See He et al. (2023) for background, Bai (2003) for PCA code, and He et al. (2022), and Chen et al. (2021) for the Quantile Factor Analysis method.

PCRA v1.0: Provides a collection of functions and several real-world data sets that support teaching a quantitative finance MS level course on Portfolio Construction and Risk Analysis. See the vignette: Introduction to CRSP Stocks and SPGMI Factors in PCRA.

Genomics

GESciLiVis v1.1.0: Provides tools to visualize publication activity per gene based on a gene list and a user-defined set of keywords to perform an NCBI database search as in PubMed. See the vignette.

ggpicrust2 v1.6.0: Provides tools to analyze and visualize PICRUSt2 output with pre-defined plots and functions, including a one-click option for creating publication-level plots. For more details, see Yang et al. (2023). Look here for examples.

gsdensity v0.1.2: Implements a computational tool for pathway centric analysis of single-cell data including scRNA-seq data and spatial genomics data. Given a gene set and a cell-by-gene matrix, ask the question: is this gene set somehow enriched by a subpopulation of the cells? See README for examples.

metaGE v1.0.0: Provides tools for conducting genome-wide association studies for studying Genotype x Environment interactions, including functions to collect the results of GWAS data from different files, infer the inter-environment correlation matrix, perform global test procedure for quantitative trait loci detection, and perform tests of contrast or meta-regression. See De Walsche et al. (2023) for the details.

Machine Learning

FACT v0.1.0: Implements an algorithm agnostic framework for feature attribution while preserving the integrity of the data and facilitating the understand of the mapping procedure of an algorithm that assigns instances to clusters. See README to get started.

lpda v1.0.1: Implements the linear programming classification method described by Nueda, et al. (2022) which is advantageous when variable distributions are unknown or when the number of variables is much greater than the number of observations. See the vignette.

UBayFS v1.0: Implements the user-guided Bayesian framework proposed by Jenul et al. (2022) for ensemble feature selection. See the Introduction and the vignette on Block feature selection.

Mathematics

qfratio v1.0.1: Provides functions to evaluate moments of ratios and products of quadratic forms in normal variables using recursive algorithms developed by Bao and Kan (2013) and and Hillier et al. (2014). See README for examples.

Medicine

gsDesign2 v1.0.7: Provides tools to enable fixed or group sequential design under non-proportional hazards assumptions that support flexible enrollment, time-to-event and time-to-dropout assumptions. Design methods include average hazard ratio, the weighted logrank tests in Yung and Liu (2019), and MaxCombo tests. See the vignette to get started.

NCC v1.0: Provides functions to simulate and analyze platform trials with non-concurrent controls. See Bofill Roig et al. (2022), Saville et al. (2022), and Schmidli et al. (2014) for background. There is a brief Introduction and there are vignettes on simulating binary data, continuous data, and How to run a simulation study.

Pharma

DrugExposureDiagnostics v0.4.1: Provides ingredient specific diagnostics for drug exposure records in the Observational Medical Outcomes Partnership (OMOP) common data model. See the Introduction and the Summary of checks vignette.

rlistings v0.1.1: Provides functions to create and display listings for clinical trials. See the Getting Started Guide.

Science

LCMSQA v1.0.0: Provides functions to check the quality of liquid chromatograph/mass spectrometry (LC/MS) experiments using an interactive shiny application. Tests include total ion current chromatogram, base peak chromatogram, mass spectrum, and extracted ion chromatogram. See the Introduction.

Statistics

lmw v0.0.1: Provides functions to compute the implied weights of linear regression models for estimating average causal effects and provides diagnostics based on these weights. See Chattopadhyay and Zubizarreta (2022) where several regression estimators are represented as weighting estimators, in connection with inverse probability weighting. Look here for examples.

ptable v1.0.0: Implements the cell-key statistical disclosure control perturbation technique to protect confidential information. See Giessing (2016) for the technical details and the vignette for examples.

snha v0.1.3: Implements the St. Nicolas House Analysis to explore interacting variables and create correlation networks. See the vignette.

satdad v1.1: Implements theoretical and non-parametric tools to analyze tail dependence in sample based or theoretical models. A goal is to generate multivariate extreme value models in any dimension. See the extensive vignette.

sr v0.1.0: Implements the Gamma test based smooth regression method for measuring smoothness in multivariate relationships, finding causal connections in precision data, finding lags and embeddings in time series, and training neural networks. See Evans & Jones (2002) and Jones (2004) for details and the vignette for examples.

wqspt v1.0.1: Implements a permutation test method for the weighted quantile sum (WQS) regression used to evaluate the effect of complex exposure mixtures on an outcome. See Carrico et al. (2015) and Day et al. (2022) for the theory and the vignette for examples.

Time Series

coconots v1.1.1: Provides tools for fitting, validating, and forecasting practical convolution-closed time series models for low counts. The models are described in Jung and Tremayne (2011), and the model assessment tools are presented in Czado et al. (2009), Gneiting and Raftery (2007), and, Tsay (1992). See README for examples.

sparseDFM v1.0: Implements various estimation methods for dynamic factor models (DFMs) including PCA, see Stock and Watson (2002), and EM, see Banbura and Modugno (2014) and DFMs Mosley et al. (2023). There are vignettes on Nowcasting UK Trade in Goods and Inflation.

Utilities

askgpt v0.0.2: Implements a connection to the OpenAI API to answer questions about R. See the vignette.

cellKey v1.0.1: Implements a method to protect statistical data by computing cell keys for individual cells in statistical tables. The theory behind the method is described in Thompson, Broadfoot and Elazar (2013) and Giessing and Tent (2019).

occupationMeasurement v0.2.0: Implements an interface for performing interactive occupation coding during interviews as described in Peycheva et al. (2021) and Schierholz et al. (2018). There are several vignettes including a Getting Started guide, Using the API, and Custom Questionnaires.

pracpac v0.1.0: Provides functions to streamline the creation of Docker images with R packages and dependencies embedded. See Nagraj and Turner (2023) for details and the vignettes Basic usage and Use cases.

RmdConcord v0.1.6: Supports concordances in R Markdown documents to easily find the source in the .Rmd file of errors detected by HTML tidy. See README for details and note that the vignette serves as a practice file.

symbol.equation.gpt v1.1.1: Provides an interface for adding symbols, smileys, arrows, and building mathematical equations using LaTeX or r2symbols for Markdown and Shiny development. See the vignette.

tinysnapshot v0.0.3: Provides snapshots for unit tests using the tinytest framework and includes expectations to test base R and ggplot2 plots as well as console output from print(). See README for usage.

Visualization

PlotBivInvGaus v0.1.0: Provides functions to create bivariate inverse Gaussian distribution contour plots for non-negative random variables. See the vignette.

textBoxplacement v1.0: Provides functions to compute a non-overlapping layout of text boxes to label multiple overlaying curves. See the vignette.

February 2023: "Top 40" New CRAN Packages

Tue, 28 Mar 2023 00:00:00 +0000

One hundred seventy-three new packages made it to CRAN in February. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Ecology, Economics, Machine Learning, Mathematics, Medicine, Pharma, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

dcTensor v1.0.1: Implements semi-binary and semi-ternary matrix methods based on non-negative matrix factorization (NMF) and singular value decomposition (SVD). For the details see the reference section of GitHub README.md. There are seven vignettes including Discretized Singular Value Decomposition, Discretized Partial Least Squares, and Discretized Non-negative Tucker Decomposition.

Data

dhis2r v0.1.1: Implements a connection to DHIS2, a global open-source project coordinated by the HISP Centre at the University of Oslo. See the vignette to get started.

ispdata v1.1: Provides access to data from the Rio de Janeiro Public Security Institute including criminal statistics, data on gun seizures and femicide. See README to get started.

ohsome v0.2.1: Implements a client for Heidelberg Institute for Geo information Technology’s OpenStreatMap API and provides functions to analyze the rich data source of OpenStreetMap (OSM) history. See the vignette for examples.

OlympicRshiny v1.0.0: Implements a Shiny App to visualize Olympic Data from 1896 to 2016 residing in a Kaggle Dataset. Look here to get started.

whitewater v0.1.2: Provides methods for retrieving United States Geological Survey (USGS) water data using sequential and parallel processing. See Bengtsson (2022) for background on the parallel methods and README for examples.

Ecology

birdscanR v0.1.2: Provides functions to extract bird and insect data from Birdscan MR1 SQL vertical-looking radar databases, filter, and process the data into Migration Traffic Rates, e.g. #objects per hour and per km, etc. See Haest et al. (2021) and Schmid et al. (2019) for background and the vignette for examples.

Economics

echoice2 v0.2.3: Implements choice models based on economic theory, including MCMC based estimation prediction, Hierarchical Multinomial Logit and Multiple Discrete-Continuous (Volumetric) models. See Allenby, Hardt and Rossi (2019), Kim, Hardt, Kim and Allenby (2022), and Hardt and Kurz (2020) for the underlying theory, and the vignettes on Importing lists of lists and Volumetric Demand and Conjunctive Screening.

Machine Learning

ODRF v0.0.3: Implements oblique decision random random forests as an ensemble of oblique decision trees which use linear combinations of predictors for partitioning trees. See the vignette.

spinner v1.1.0: Provides a torch implementation of Graph Net architecture allowing different options for message passing and feature embedding. Look here for examples.

sMTL v0.1.0: Implements L0-constrained Multi-Task Learning and domain generalization algorithms which are coded in Julia allowing for fast coordinate descent and local combinatorial search algorithms. See Loewinger et al. (2022) for details and look here for an introduction.

tfevents v0.0.2: Provides a convenient way to log scalars, images, audio, and histograms in the tfevent record file format. Logged data can be visualized on the fly using TensorBoard, a web based tool that focuses on visualizing the training progress of machine learning models. Look here for an example.

tidyAML v0.0.1: Implements a simple interface for automatic machine learning that uses the fits tidymodels framework to fit models. Look here for examples.

Mathematics

deFit v0.1.2: Provides functions that use numerical optimization to fit first and second order ordinary differential equations to time series data in order to examine the dynamic relationships between variables or the characteristics of a dynamical system. Look here for examples.

fitlandr v0.1.0: Provides a toolbox for estimating vector fields from intensive longitudinal data and construct potential landscapes. The vector fields can be estimated with two nonparametric methods: the Multivariate Vector Field Kernel Estimator by Bandi & Moloche (2018) and the Sparse Vector Field Consensus algorithm by Ma et al. (2013). The potential landscapes are simulated with the simlandr package of Cui et al. (2021) or with the method of Bhattacharya et al. (2011). See README for examples.

Medicine

CodelistGenerator v1.0.0: Provides functions to generate a candidate code list for the Observational Medical Outcomes Partnership common data model based on string matching. For a given search strategy, a candidate code list will be returned. See the Introduction and the vignettes on Candidate codes, Options, and Codelists for medicaitons.

simaerep v0.4.3: Implements bootstrap based simulation methods to detect clinical trials that may be under reporting adverse events. See Koneswarakantha (2021) for background and README for an example.

Pharma

metalite v0.1.1: Provides A metadata structure for clinical data analysis and reporting based on Analysis Data Model (ADaM) datasets which simplifies clinical analysis and reporting tool development by defining standardized inputs, outputs, and workflow. See Zhang et al. (2022), the package Introduction and the vignettes AE Listing, AE Specification, AE Summary, and Miettinen and Nurminen Test.

Science

gcplyr v1.1.0: Implements tools to import, manipulate and analyze bacterial growth curve data as commonly output by plate readers including for reshaping common plate reader outputs into tidy formats. See README for documentation. There are several vignettes including an Introduction and Analyzing data and Dealing with noise.

locaR v0.1.2: Provides functions to conduct acoustic source localization, as well as organize and check localization data and results. Cobos et al. (2010) gives details of the algorithms. Vignettes include an Introduction, Detecting sound sources, and localize Multiple.

PooldiloutionR v1.0.0: Pool dilution is a isotope tracer technique wherein a biogeochemical pool is artificially enriched with its heavy isotopologue and the gross productive and consumptive fluxes of that pool are quantified. This package calculates gross production and consumption rates from closed-system isotopic pool dilution time series data. See Fischer and Hedin (2002) for background and the vignette for examples.

Statistics

counterfactuals v0.1.1: Implements a modular R6 interface for counterfacual explanation methods including Burghmans et al. (2022), Dandl et al. (2020), and Wexler et al. (2019). See the the Introduction and the vignettes How to extend the package and Other types of models.

semfindr v0.1.4: Provides functions for sensitivity analysis in structural equation modeling using influence measures and diagnostic plots. It supports the leave-one-out case wise sensitivity analysis of Pek and MacCallum (2011). There is an introduction and three additional vignettes: Approximate Case Influence, Selecting Cases in Rerun, and Use Case IDs.

stxplore v0.1.0: Implements statistical tools for spatio-temporal data exploration, including simple plotting functions, covariance calculations and computations similar to principal component analysis for spatio-temporal data. Look here for examples and see the vignettes Exploration using dataframes and Using stars objects.

SurrogateRsq v0.2.0: Implements the surrogate R-squared measure for categorical data analysis proposed in Liu et al. (2022). See the vignette.

Time Series

setartree v0.1.0: Implements the forecasting-specific tree-based model proposed by Godahewa et al. (2022) that is particularly suitable for global time series forecasting. Look here to get started.

StructuralDecompose v0.1.1: Provides functions, which perform very well in the presence of significant level shifts, to explain the behavior of a time series by decomposing it into trend, seasonality and residuals. See the short vignettes Decomposition and Example Walkthrough.

sufficientForecasting v0.1.0: Implements a sufficient forecasting method for a single time series using many predictors and a possibly nonlinear forecasting function. Assuming that the predictors are driven by some latent factors, the SF first conducts factor analysis and then performs sufficient dimension reduction. See Fan et al. (2017), Luo et al. (2022), and Yu et al. (2022) for background and the vignette to get started.

tseriesTARMA v0.3-2: Implements routines for nonlinear time series analysis based on Threshold Autoregressive Moving Average models and provides methods for model fitting and forecasting, tests for threshold effects and unit-root tests. See README for examples.

Utilities

currr v0..1.2: Implements the family of map() functions with frequent saving of the intermediate results. This enables stopping the evaluation and then restarting it from where you left off by reading the already evaluated work from cache. See README to get started.

dataMojo v1.0.0: Implements a grammar of data manipulation with data.table by providing a consistent a series of utility functions that help to solve the most common data manipulation challenges. See the vignette.

hexfont v0.3.1: Contains all the hex font files from the GNU Unifont Project compressed by ‘xz’. GNU Unifont is a duo spaced bitmap font that attempts to cover all the official Unicode glyphs plus several of the artificial scripts in the Under-ConScript Unicode Registry. See the vignette.

parabar v0.10.1: Provides a simple interface in the form of R6 classes for executing tasks in parallel, tracking their progress, and displaying accurate progress bars. See README for examples.

rang v0.2.0: Provides tools to resolve the dependency graph of R packages at a specific time point based on the information from various R-hub web services. The dependency graph can then be used to reconstruct the R computational environment with Rocker. See README and the FAQ.

tablexlsx v0.1.0: Provides functions to export data frames to excel workbooks. See the vignette.

xmpdf v0.1.3: Provides functions to edit XMP metadata in a variety of media file formats as well as edit bookmarks and documentation in pdf files. Functions can detect and use a variety of command-line tools to perform these operations including exiftool, ghostscript, and pdftk. See the Introduction and the FAQ vignette.

Visualization

animate v0.3.9.4: Implements a web-based graphics device to extend base R graphics functions to support frame-by-frame animation and keyframes animation. Target use cases include real-time animated visualizations, agent-based models, dynamical systems, and animated diagrams. See the Introduction and the Q&A vignette.

Durga v1.0.0: Implements a system for plotting grouped data effect sizes. compatible with base R methods for combining plots. See Khan & McLean (2023) and the vignette for examples.

ggrain v0.0.3: Extends ggplot2 to create raincloud plots. See the vignette.

PieGlyph v0.1.0: Extends ggplot2 to replace points in a scatter plot with pie-chart glyphs showing the relative proportions of different categories. There are several vignettes including PieGlyph, Multinomial Classificatio, and Time series example.

January 2023: "Top 40" New CRAN Packages

Tue, 28 Feb 2023 00:00:00 +0000

One hundred sixty-five new packages made it to CRAN in January. Here are my “Top 40” selections in thirteen categories: Actuarial Statistics, Archaeology, Computational Methods, Ecology, Genomics, Mathematics, Medicine, Machine Learning, Science, Statistics, Time Series, Utilities, Visualization.

Actuarial Statistics

actuaRE v0.1.3: Provides functions to fit random effects models using either the hierarchical credibility model alone or combined with a glm or with a Tweedie generalized linear mixed model. See Campo & Antonio (2023) for background and the vignette for an example.

Archaeology

archeoViz v0.2.2: Implements a shiny application for the visualisation, interactive exploration, and web communication of archaeological excavation data. It includes interactive 3D and 2D visualisations, th generation of cross sections and maps, basic spatial analysis methods, and excavation timeline visualisations. There is a short vignette in English and French.

shoredate v1.0.0: Provides tools for shoreline dating Stone Age sites located on the Norwegian Skagerrak coast using methods presented in Roalkvam (2023). See the vignette.

Computational Methods

em v1.0.0: Implements a generic Expectation-Maximization (EM) algorithm within a maximum likelihood framework based on Dempster, Laird, and Rubin (1977). See the vignette for some theory and examples.

ROptimus v3.0.0: Implements a general-purpose optimisation engine that supports Monte Carlo optimisation with the Metropolis criterion and acceptance ratio replica exchange Monte Carlo optimization. See the foundational papers Metropolis et al. (1953), Hastings (1970), Kirkpatrick et al. (1983) and Černý (1985) for background.

Data

CopernicusMarine v0.0.6: Provides functions to import data on the ocean’s physical and biogeochemical state from EU Copernicus Marine Service Information. See README to get started.

itol.toolkit v1.1.0: Provides helper functions to access the Interactive Tree of Life including functions to edit and annotate trees interactively. See the vignette to get started.

rdracor v0.7.2: Provides an interface to the Drama Corpora Project (DraCor) API.

usdoj v1.0.0: Provides functions to fetch data from the U.S Department of Justice API such as press releases, blog entries, and speeches. Look here for notes on data structure.

Ecology

pastclim v1.2.3: Implements methods to extract and manipulate palaeoclimate reconstructions for ecological and anthropological analyses as described in Leonardi et al. (2022). See the Overview and the vignettes available datasets and custom dataset.

PERK v0.0.9.2: Implements a shiny web application to predict and visualize concentrations of pharmaceuticals in the aqueous environment. See Jagadeesan et al. (2022) and the vignette for a walk through.

Gemomics

autoGO v0.9.1: Implements a framework to enable automated, high quality gene ontology enrichment analysis visualizations and a wrapper for differential expression analysis using the DESeq2 package described in Love et al. (2014). See the vignette.

phylter v0.9.6: Provides functions to detect and remove outliers in phylogenomics datasets that build on the Distatis approach described in Abdi et al. (2005). See the vignette.

poolHelper v1.0.0: Provides functions to simulate pooled sequencing data under a variety of conditions, and also evaluate the average absolute difference between allele frequencies computed from genotypes and those computed from pooled data. See Carvalho et al. (2022) for the details and the vignette for an introduction to the package.

Machine Learning

BT v0.3: Implements adaptive boosting trees for Poisson distributed response variables, using log-link function. See Trufin & Denuit (2021), Denuit et al. (2019), and Hainaut & Trufin (2022) for background, and the vignette for an example.

chatgpt v0.1.5: Implements a ChatGPT coding assistant for the RStudio IDE.

evreg v1.0.1: Implements a evidential neural network for the regression model recently introduced in Denoeux (2023) in which prediction uncertainty is quantified by Gaussian random fuzzy numbers as introduced in Denoeux (2023). See the vignette.

FuzzyDBScan v0.0.3: Provides an interface to the Fuzzy DBScan clustering algorithm described in Ienco and Bordogna (2018). See README for an example.

Mathematics

rgudhi v0.1.0: Implements and interface to the C++ library, GHUDI for topological data analysis (TDA) and offers state-of-the-art data structures and algorithms to construct simplicial complexes and compute persistent homology.

Medicine

injurytools v1.0.1: Provides standardized routines and utilities to simplify the data analysis of sports injuries in order to identify and describe the magnitude of sports injury problems and determine the potential risk factors. See the vignettes estimate-epi-measures, prepare-injury-data. and visualize-injury-data.

simIDM v0.0.5: Provides functions to simulate oncology trials using an illness - death model. See Meller, Beyersmann and Rufibach (2019) for background and the vignettes Getting Started and Power and Type 1 Error Correlations.

Science

gravmagsubs v1.0.1: Provides functions to compute the gravitational and magnetic anomalies generated by 3-D vertical rectangular prisms at specific observation points using the method of Plouff (1976). See the vignette.

QurvE v1.0: Implements three methods for solving high-throughput analysis of growth curves and fluorescence data: linear regression, growth model fitting, and smooth spline fits. A shiny application provides access to all features without requiring any programming knowledge. There is a User Manual and there are vignettes on Flouresence Curve Evaluation and Growth Curve Evaluation.

Statistics

BCClong v1.0.0: Implements a Bayesian consensus clustering (BCC) model for multiple longitudinal features via a generalized linear mixed model that allows simultaneous clustering of mixed-type (e.g., continuous, discrete and categorical) longitudinal features. There are vignettes for continuous and mixed type data.

clarify v0.1.2: Provides functions to perform simulation-based inference as an alternative to the delta method for obtaining valid confidence intervals and p-values for regression post-estimation quantities such as average marginal effects and predictions at representative values. The methodology is described in King et al. (2000). See the vignette for examples.

emplikAUC v0.3: Provides functions to test hypotheses and construct confidence intervals for AUC (Area Under receiver operating characteristic curve) and pAUC (partial area under ROC curve) using the method described in Zhao, Ding & Zhou .

jtdm v0.1-0: Implements The Joint Trait Distribution Model in a Bayesian framework using conjugate priors to compute joint probabilities and multivariate confidence intervals. See the vignette.

mbreaks v1.0.0: Functions provide comprehensive treatments for estimating, inferring, testing and selecting linear regression models with structural breaks. See Bai & Perron (1998) for background and the vignette examples.

sdmTMB v0.3.0: Implements spatial and spatiotemporal predictive-process GLMMs (Generalized Linear Mixed Effect Models) using TMB, INLA, and the SPDE (Stochastic Partial Differential Equation) approximation to Gaussian random fields. See Anderson et al. (2022) for background and the vignette for examples.

snapKrig v0.0.1: Provides functions for geostatistical modeling and kriging with gridded data using spatially separable covariance Kronecker covariance functions. See Koch, Lele, Lewis (2020) for descriptions of the computational methods and the vignette for examples.

Time Series

fnets v0.1.2: Implements methods for network estimation and forecasting of high-dimensional time series exhibiting strong serial and cross-sectional correlations under a factor-adjusted vector autoregressive model. See Barigozzi et al. (2022) for the methodology and Owens et al. (2023) for details of the package.

MSinference v0.0.9: Provides functions to perform multiscale analysis of a nonparametric regression or nonparametric regressions with time series errors. See Khismatullina and Vogt (2020), Khismatullina and Vogt (2022) and Khismatullina and Vogt (2023) for the theory and examples, and the vignette to get started.

Utilities

datetimeoffset v 0.2.1: Provides support for a number of datetime string standards including ISO 8601 and pdfmark and also datetimes with UTC offsets and possibly heterogeneous time zones with up to nanosecond precision. See the package Introduction.

mathml v0.5: Provides functions to translate R expressions to MathML or MathJax so that they can be rendered in rmarkdown documents and shiny applications. See the vignette

options v0.0.1: Provides a simple mechanisms for defining and interpreting package options, including helper functions for interpreting environment variables, global options, defining default values and more. See the vignettes environment variables and options.

parquetize v0.5.3: Provides functions to convertcsv, RData, rds, RSQLite, json, ndjson, SAS, SPSS and other files to the Parquet columnar storage format. See the vignette.

scenes v0.1.0: Provides functions to facilitate switching between shiny UIs depending on the information that is to be passed to the request object. See the vignettes Creating new actions and Changing scenes.

shiny.fluent v0.3.0: Provides shiny components based on the Fluent UI JavaScript library. There are several small vignettes including a Tutorial on creating a full dashboard with shiny and Fluent.

Visualization

ggplate v0.0.1: Provides functions to create simple plots of biological culture plates as well as microplates which can plot both continuous and discrete values. See README for examples.

plotRCS v0.1.3: Extends ggplot2 to draw restricted cubic spline curves from a logistic regression model or a Cox proportional hazards regression model using the method described in Harrell (2015). Look here for examples.

viscomp v1.0.0: Implements several visualization tools for exploring the behavior of the components in a network meta-analysis of multi-component interventions including heat plots of the two-by-two component combinations, leave one component combination out scatter plot, violin plots for specific component combination effects, and density plots for components effects. See the vignette for examples.

December 2022: "Top 40" New CRAN Packages

Mon, 30 Jan 2023 00:00:00 +0000

One hundred sixteen new packages stuck to CRAN in December 2022. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Ecology, Epidemiology, Genomics, Machine Learning, Mathematics, Medicine, Networks, Signal Processing, Statistics, Utilities, and Visualization.

Computational Methods

bgw v0.1.0: Implements the BGW algorithm described by Bunch, Gay and Welsch (1993) which exploits the special structure of statistical estimation problems within a trust-region-based optimization approach to produce an estimation algorithm that is much more effective than the usual practice of using optimization methods originally developed for general optimization. See the vignette.

GPArotateDF v2022.12-1: Provides functions for derivative-free gradient projection algorithms for factor rotation. See Jennrich (2004) and Bernaards and Jennrich (2005) for the theory, and the vignette for examples.

Mestim v0.2.1: Implements a flexible framework for estimating the variance-covariance matrix of parameters by computing the empirical sandwich variance. See Tsiatis et al. (2019) for background and the vignette for some theory and examples.

Data

dafishr v1.0.0: Provides functions to download, clean and analyse raw Vessel Monitoring System, VMS, data from Mexican government. See the vignette to get started.

hcidata v0.1.0: Provides a collection of datasets of human-computer interaction (HCI) experiments. Each dataset is from an HCI paper, with all fields described and the original publication linked. The datasets sources include Bergström et al. (2022), Dalsgaard et al. (2021), Larsen et al. (2019), Lilija et al. (2019), Pohl and Murray-Smith (2013), and Pohl and Mottelson (2022).

Ecology

bamm v0.4.3: Implements species distribution models as a function of biotic and movement factors. See Soberón and Osorio-Olvera (2022) for the theoretical background and the vignette for examples.

gadget3 v0.8-4: Provides a framework for creating marine ecosystem models that generatesR or C++ code which can then be optimized using the TMB package and standard tools. See Kristensen et al. (2016) and Begley & Howell (2004) for background and the vignettes Model Debugging, Model Structure, and Writing G3 Actions.

Epidemiology

finalsize v0.1: Provides functions to calculate the final size of a susceptible-infectious-recovered epidemic in a population with demographic variation in contact patterns and susceptibility to disease. See Miller (2012) for details. Additionally, there is an Introduction and vignettes on modeling uncertainty in R0 and modeling heterogeneous susceptibility.

Genomics

betaclust v1.0.0: Implements the family of novel beta mixture models developed by Majumdar et al. (2022) to appositely model the beta-valued cytosine-guanine dinucleotide (CpG) sites, to objectively identify methylation state thresholds, and to identify the differentially methylated CpG (DMC) sites using a model-based clustering approach. See the vignette.

plasma v0.9.21: Implements tools for supervised analyses of incomplete, overlapping multiomics datasets. Functions apply partial least squares in multiple steps to find models that predict survival outcomes. See the vignettes on Interpretation and Partial Least Squares.

POMS v1.0.1: Provides code to identify functional enrichments across diverse taxa in phylogenetic tree, particularly where these taxa differ in abundance across samples in a non-random pattern. See Douglas et al. (2022) for background and look here for documentation.

vivaldi v1.0.0: Provides functions to analyze minor alleles of viral genomes from sequencing data of viral genomes in Illumina vcf files. See the vignette.

Machine Learning

arf v0.1.2: Implements adversarial random forests, an iterative algorithm that recursively partition data into fully factorized leaves, where features are jointly independent. See Watson et al. (2022) for details and the vignette on Density Estimation.

correctR Provides corrected test statistics for cases when samples are not independent, such as when classification accuracy values are obtained over resamples or through k-fold cross-validation, as proposed by Nadeau and Bengio (2003) and presented in Bouckaert and Frank (2004). See the vignette.

HTT v0.1.1: Implements a novel decision tree algorithm in the hypothesis testing framework that examines the distribution difference between two child nodes over all possible binary partitions. The test statistic enables the algorithm to better detect complex structure and not only the mean difference. See the vignette

janus v1.0.0: Implements a tensorflowbased, deep-neural network recommending system with coarse-to-fine optimization. Look here for an introduction.

reservr v0.0.1: Provides functions to fit distributions and neural networks to censored and truncated data. See Bücher & Rosenstock (2022) for background and the vignettes Working with Distributions and TensorFlow Integration.

Mathematics

qspray v0.1.1: Provides functions for the symbolic calculation and evaluation of multivariate polynomials with rational coefficients. See README for an example.

Medicine

cstime v2022.11.22: Provides consistent time conversion functions for public health purposes including conversions between date, ISO week, ISO yearweek, ISO year, calendar month/year, season, and season week. There is an Introduction and there are additional vignettes Date, year, week conversion and Season week.

GNGTools v1.0.0: Implements a Go/No-Go decision-making framework based on Bayesian posterior probabilities linked to the target product profile. See the vignette.

oncomsm v0.1.2: Implements methods to fit a parametric Bayesian multi-state model to tumor response data. The model can be used to sample from the predictive distribution, to impute missing data, and to calculate probability of success for custom decision criteria during an ongoing trial. See the vignettes avoiding-bias, oncomsm, and prior-choice.

REDCapDM v0.1.0: Provides functions to to read and manage REDCap data and identify missing or extreme values. See the vignette.

sitePickR v0.0.1: Provides functions to perform a two-level process to select a representative sample of sites for a prospective study such as a randomized controlled trial where the possibility of an initially selected site may not want to participate is anticipated. See Deville & Tillé (2004) and Tillé (2011) for background on the method employed and the vignette for a demo.

Networks

metanetwork v0.7.0: Provides a toolbox to represent large trophic networks in space or time across aggregation levels. The layout algorithm uses dimension reduction on a diffusion graph kernel and trophic levels. Look here for examples.

oddnet v0.1.0: Implements a feature-based method to identify anomalies in dynamic, temporal networks. See Kandanaarachchi & Hyndman (2022) for the theory and the vignette for an example.

Signal Processing

ohun v0.1.0: Provides functions to facilitates the automatic detection of acoustic signals, including functions to diagnose and optimize the performance of detection routines. See Hossin & Sulaiman (2015) for background and the vignette for examples.

Statistics

gmvjoint v0.1.0: Implements an EM algorithm for fitting joint survival and glm models of longitudinal data. See Bernhardt (2015) for background and README for examples.

hdcate v0.1.0: Implements a two-step double-robust method to estimate the conditional average treatment effects (CATE) with potentially high-dimensional covariates as described in Fan et al. (2022). See the vignette.

jointVIP v0.1.1: Implements joint variance importance plots to assist with variable selection and parameter tuning. See by Liao et al. (2023) for background. There is a Getting Started Guide and a vignette on Options.

nieve v0.1.1: Provides utility functions and objects for extreme value analysis including probability functions with their exact derivatives and transformations exchanging the two parameterizations of peaks over threshold models. See the vignette.

RelDists v1.0.0: Provides functions for parameter estimation and linear regression models for the reliability distribution families reviewed by Almalki & Nadarajah (2014). Generalized Additive Models are used for location, scale and shape. See Rigby & Stasinopoulos (2005). There are vignettes on the [FWE]() and OQ distributions.

ridgetorus v1.0.1: Implements Principal Component Analysis (PCA) on the torus via density ridge estimation, and includes functions for evaluating, fitting, and sampling these models. See García-Portugués and Prieto-Tirado (2022) for the theory and the vignette for examples.

Utilities

dataset v0.2.0: Implements a subjective interpretation of the W3C DataSet recommendation and the datacube model, which relevant to the global Statistical Data and Metadata eXchange standards, the Dublin Core, and the DataCite standards preferred by European open science repositories to improve the findability, accessibility, interoperability and reusability of the datasets. There are several vignettes including Motivation and Datasets with FAIR Metadata.

nlmixr2rpt v0.1.0: provides functions to generate reporting workflows around nlmixr2 analyses with outputs in Word and PowerPoint including functions to specify figures, tables and report structure in a user-definable YAML file. See the vignettes Accessing Figures and Tables and Reporting nlmixr2 Fit Results.

qtwAcademic v2022.12.13: Provides Quarto website templates commonly used by academics including templates for personal websites and course/workshop websites. There is an Intraduction and there are vignettes for workshop and personal websites.

r2social v1.0: Provides JavaScript and CSS styles to allow easy incorporation of various social media elements in shiny applications, dashboards and rmarkdown documents. The elements include share buttons, connect with us buttons, and hyperlink buttons. See the vignette.

rworkflows v0.99.5: Implements functions for the continuous integration for R packages, automated testing, website building, and containerised deployment. See the vignettes depgraph, docker, repositories report, and rworlflows.

shinyDatetimePicker v1.0.0: Provides three types of datetime pickers for usage in a shiny UI. See README for examples.

Visualization

ddplot v0.0.1: Implements an API to allow users to create D3 based SVG plots using the r2d3 library. See the vignette.

swipeR v0.1.0: Provides tools to create carousels using the JavaScript library Swiper and htmlwidgets. Carousels can be displayed in the RStudio viewer pane, in Shiny applications and in rmarkdown documents. See README for examples.

November 2022: "Top 40" New CRAN Packages

Tue, 03 Jan 2023 00:00:00 +0000

One hundred sixty-seven new packages made it to CRAN in November: Here are my “Top 40” selections in fourteen categories: Climate Modeling: Computational Methods, Data, Ecology, Epidemiology, Genomics, Machine Learning, Mathematics, Networks, Pharma, Statistics, Time Series, Utilities, and Visualization.

Climate Modeling

gtfs2emis v0.1.0: Implements a bottom up model to estimate the emission levels of public transport systems based on General Transit Feed Specification data. Functions estimate several pollutants at high spatial and temporal resolutions. See Viera et al for background. There is a Getting Started Guide, and vignettes on Emission Factors and Preparing Data.

Computational Methods

jmatrix v1.1: A mainly instrumental package that allows other packages with cores written in C++ to read, write and manipulate matrices in a binary format to mitigate memory issues. See the vignette.

lazyNumbers v1.2.1: Implements lazy numbers, a new number type whose arithmetic is exact, contrary to ordinary floating-point arithmetic. The lazy numbers are implemented in C++ with the CGAL library. See README for examples.

miesmuschel v0.0.2: Provides optimization algorithms and functions that can be used to manually construct specialized optimization loops including the mixed integer evolution strategy as described in Li et al. (2013) and the multi-objective optimization algorithms NSGA-II described in Deb et al. (2002). See README for examples.

mlr3mbo v0.1.1: Implements a flexible approach to Bayesian optimization that includes both ready-to-use optimization algorithms as well as fundamental building blocks to construct custom algorithms. See Jones et al. (1998), Knowles (2006), and Ponweiser et al. (2008) for examples of ready to use optimization algorithms and the vignette for examples.

Data

googletraffic v0.1.1: Allows users to create geographically referenced traffic data from the Google Maps JavaScript API. Look here for information to get started.

IPEDS v0.1.1: Implements an interface to the US Post-Secondary Institution Statistics for 2020 which contains information on post-secondary institutions, students, faculty, demographics, financial aid, educational and recreational offerings, and completions. See the vignette.

npi v0.2.0: Provides access the US National Provider Identifier Registry API which contains administrative data linked to a specific individual or organizational healthcare providers. There is an Introduction and a vignette on Advanced Use.

rfars v0.2.0: Implements an interface to the US Fatality Analysis Reporting System. See the vignettes Crash Sequences and Rural Roads.

tidyDisasters v0.1.1: Provides a queryable data set that unites information from three complementary resources: Belgium’s Centre for Research on the Epidemiology of Disasters EMDAT, the US National Consortium for the Study of Terrorism GTD, and the US Federal Emergency Management Agency FEMA. There is a brief User Guide.

Ecology

et.nwfva v0.1.1: Implements a forest management tool developed by the Northwest German Forest Research Institute (NW-FVA) for the five main commercial tree species oak, beech, spruce, Douglas-fir and pine for northwestern Germany. See Albert et al (2022) for background and the vignettes: Beispiele, Nutzerhinweise zum Geleit, and Vergleich der Interpolationsmethoden.

mmodely v0.2.2: Provides functions to perform multivariate modeling of evolved traits, with special attention to understanding the interplay of the multi-factorial determinants of their origins in complex ecological settings. See Stephens (2007), Gruebner (2011), and Garamszegi (2011) for background, and the examples on primate vision and vocal communications.

webSDM v1.1-1: Implements the method of Poggiato et al. (2022) for working with tropic Species Distribution Models. There is an Introduction and there are vignettes on Composit variables, Differences between SDM and Trophic SDM.

Epidemiology

exDE v1.0.0: Provides tools to set up modular ordinary and delay differential equation models for mosquito-borne pathogens, focusing on malaria. See Wu et al. (2022) for a description on the methods implemented. There are ten vignettes including Basic Copmetition Aquatic Mosquito Model and SIP Human Model.

Genomics

idopNetwork v0.1.1: Implements a cartographic tool to chart spatial microbial interaction networks. See Kim et al. (2008) and Wang et al. (2011) for information on functional clustering, Chen et al. (2019) and Cao et al. (2022) for background on the model, and the vignette for examples.

tip v0.1.0: Provides functions to cluster data without specifying the number of clusters using the Table Invitation Prior (TIP) introduced by Harrison et al. (2022). There are vignettes on matrix clustering with CONSTANT and MNIW, Tensor clustering CONSTANT, iris NIW, and usarrests NIW.

scistreer v1.0.1: Enables fast maximum-likelihood phylogeny inference from noisy single-cell data using the ScisTree algorithm described in Wu (2019) making the method applicable to massive single-cell datasets (>10,000 cells). Look here for an example.

Machine Learning

tidyclust v0.1.1: Implements a common interface to specifying clustering models, in the same style as parsnip. Creates unified interface across different functions and computational engines. See README for examples.

VSOLassoBag v0.99.0: Implements an integrated Wrapped LASSO ensemble learning strategy to help select efficient, stable, and high confidential variables from omics-based data. Functions integrate and vote on variables generated from multiple LASSO models to determine the optimal candidates. See Luo et al. (2020) for details and the vignette for examples.

Mathematics

tensorFun v0.1.1: Provides basic functions to handle higher-order tensor data. See Kolda and Bader (2009) for background on tensor decompositions, Bau and Ng (2021) for information on missing values, and the vignette for a short introduction.

TransTGGM V1.0.0: Implements a transfer learning framework for tensor Gaussian graphical models, which takes full advantage of informative auxiliary domains even when non-informative auxiliary domains are present. See Ren, Zhen, and Wang for background and the vignette.

Networks

nett 1.0.0: Provides multiple methods for fitting, model selection and goodness-of-fit testing in degree-corrected stochastic blocks models. Implements the methods in Amini et al.(2013), Bickel and Sarkar (2015), Lei (2016), Wang and Bickel (2017) and more. There are vignettes Community Detection, Visualization, Explore Comm, and dcsbm testing.

Pharma

DrugSim2DR v0.1.0: Implements a tool to predict drug functional similarity for drug repurposing. See the vignette.

parttime v0.1.0: Provides classes for embedding partial missingness as a central part of datetime classes allowing for more ergonomic use of datetimes for challenging datetime computation, including calculations of overlapping date ranges, imputations, and more. See README for examples.

Statistics

GAGAs v0.5.1: Implements the Global Adaptive Generative Adjustment Algorithm for generalized liner models used for improving the computational efficiency in the high-dimensional data analysis. See Wang et al. (2022) for the theory.

kfino v1.0.0: Implements method for detecting outliers with a Kalman filter on impulsed noised outliers and prediction on cleaned data. See Cloez et al. (2022) for the details and the vignettes on outlier detection and outlier detection with parallelization.

pwrss v0.2.0: Implements power and sample size calculations for a number of one, two and three sample tests. See the vignette for examples.

spatstat.models v3.0-2: A member of the spatstat family of packages, it provides multiple functions for exploratory data analysis and nonparametric analysis of spatial data, mainly spatial point patterns. See spatstat.org for documentation.

vMF v0.0.1: Provides functions for fast sampling from von Mises-Fisher distribution using the method proposed in Wood (1994). See the vignette.

Time Series

ardl.nardl v1.2.2: Implements linear and nonlinear autoregressive distributed lag (ARDL & NARDL) models and the corresponding error correction models and includes a test for long-run and short-run asymmetry and the Pesaran, Shin & Smith (2001) bounds test for level relationships.

Utilities

cookies v0.2.0: Provides tools for working with cookies in shiny apps, in part by wrapping the js-cookie JavaScript library. See README for an example.

formatdown v0.1.1: Provides a small set of tools for formatting tasks when creating documents in rmarkdown or quarto. See the vignette.

hellorust v1.0.0: Implements tools to use Rust code in R without hacks or frameworks. Includes basic examples of importing cargo dependencies, spawning threads and passing numbers or strings from Rust to R. Look here and here for for documentation.

prismjs v1.1.0: Implements a server-side rendering in R using Prism, a lightweight, extensible syntax highlighter, built with modern web standards in mind such that no JavaScript library is required in the resulting HTML documents. Look here for examples.

rjtools v1.0.9: Provides tools to create an R Journal rmarkdown template article, that will generate HTML and PDF versions of your paper. There are vignettes rjtools-template, Check, Create article, and Format details.

Visualization

funkyheatmap v0.1.0: Provides functions for generating heatmap-like visualizations for benchmark data frames, which can be fine-tuned with annotations for columns and rows. See Saelens et al. (2019) for background and the vignettes Recreating the dynbenchmark figures and Demo with mtcars for examples.

ggbrain v0.8.0: Implements a ggplot2-consistent approach to generating 2D displays of volumetric brain imaging data from multiple NIfTI images. There is an Introduction and there are vignettes on Aesthetic refinement and Annotations and labels.

ggpcp v0.2.0: Provides a Grammar of Graphics implementation of parallel coordinate plots that incorporates categorical variables into the plots in a principled manner. Look here for examples.

ggredist v0.0.2: Provides ggplot2 extensions for political map making based on simple features and includes palettes and scales for red to blue color mapping and for discrete maps. See README for an example.

ggsector v1.6.6: Implements functions that use grid and ggplot2 to plot sectors, draw complex heat maps and interact with Seurat to plot gene expression percentages. See the vignette.

sherlock v0.5.1: Provides graphical displays and statistical tools for structured problem solving and diagnosis with functions that are especially useful for applying the process of elimination as a problem diagnosis technique. See Hartsborne (2020) for background and README for examples.

October 2022: "Top 40" New CRAN Packages

Mon, 28 Nov 2022 00:00:00 +0000

One hundred seventy-four new packages made it to CRAN in October. Here are my “Top 40” selections in sixteen categories: Astronomy, Biology, Business, Computational Methods, Data, Ecology, Finance, Genomics, Mathematics, Machine Learning, Medicine, Pharma, Statistics, Time Series, Utilities, Visualization.

Astronomy

skylight v1.1: Provides a function to calculate sky illuminance values (in lux) for both the sun and moon. The model is a verbatim translation of the code by Janiczek and DeYoung (1987). There are vignettes for Use and Advanced Use.

Biology

palaeoverse v1.0.0: Provides tools to support data preparation and exploration for palaeobiological analyses including functions for data cleaning, binning (time and space), summarisation and visualisation with the goals of improving code reproducibility and accessibility and establishing standards for the palaeobiological community. See Jones et al. for details, and the contribution guide to get involved.

pirouette v1.6.5: Implements a method to create a Bayesian posterior from a phylogeny that depicts the true evolutionary relationships. See Richèl et al. (2020) for background. There are several vignettes including a Tutorial, a demo, and a guide showing how to use the package in a scientific experiment.

Business

bupaverse v0.1.0: Facilitates loading the packages comprising the bupaverse, an integrated suite of R packages for handling and analysing business process data, developed by the Business Informatics research group at Hasselt University, Belgium. See the Getting Started Guide.

Computational Methods

fastWavelets v1.0.1: Provides an Rcpp implementation of the Maximal Overlap Discrete Wavelet Transform (MODWT) and the À Trous Discrete Wavelet Transform. See Quilty & Adamowski (2018) for background and README for examples.

gips v1.0.0: Employs the methods described in Graczyk et al. (2022) to find the permutation symmetry group under which the covariance matrix of the data is invariant. See the vignettes Optimizers, Theory, and gips.

HomomorphicEncryption v0.1.0: Implements the Brakerski-Fan-Vercauteren (2012), Brakerski-Gentry-Vaikuntanathan (2014), and Cheon-Kim-Kim-Song (2016) schema for fully homomorphic encryption. There are seven short vignettes including BFV, BGV, and CKKS.

rxode2random v2.0.9: Implements parallel random number generation. See Wang et al. (2016) and Fidler et al (2019) for background and README for an example..

Data

airnow v0.1.0: Provides functions to retrieve U.S. Government AirNow air quality data. See README to get started.

amazonadsR v0.1.0: Provides functions to collect data on digital marketing campaigns using the Windsor.ai API. See the tutorial for an example and also look at the related new packages: bingadsR, facebookadsR, googleadsR, instagramadsR, linkedinadsR, pinterestadsR, redditadsR, snapchatadsR, ticktokadsR, twitteradsR. Pablo Sanchez was on a roll in October.

congress v0.0.1: Provides functions to download and read data on United States congressional proceedings through the Congress.gov API of the Library of Congress. See README for an example.

Ecology

canaper v1.0.0: Provides functions to analyze the spatial distribution of biodiversity especially useful in the categorical analysis of neo- and paleo-endemism (CANAPE) as described in Mishler et al. (2014) and for statistical tests to determine the types of endemism that occur in a study area while accounting for the evolutionary relationships of species. There are vignettes on CANAPE, randomization, and parallel computing.

EcoEnsemble v1.0.1: Provides functions to fit and sample from the ensemble model described in Spence et al (2018). There is an Introduction and there are two additional vignettes: ExploringPriors and SyntheticData.

rTRIPLEXCWFlux v0.2.0: Encodes the carbon uptake submodule and evapotranspiration submodule of the TRIPLEX-CW-Flux model to run the simulation of carbon-water coupling. See Zhou et al. (2008) Monteith (1965) for background and the vignette for examples.

stopdetection v0.1.1: Enables stop detection in time stamped trajectory by implementing the Stay Point detection algorithm originally described in Ye (2009) that uses time and distance thresholds to characterize spatial regions as stops. See the vignette for examples.

Finance

highOrderPortfolios v0.1.0: Implements methods to select portfolios using high order moments to characterize return distributions. See Zhou & Palomar (2021) and Wang et al. (2022) for the theory and the vignette to get started.

MSTest v0.1.0: Implements hypothesis testing procedures described in Hansen (1992), Carrasco, Hu, & Ploberger (2014) and Dufour & Luger (2017) that can be used to identify the number of regimes in Markov switching models. See README for an example.

Genomics

metevalue v0.1.13: Implements the e-value method to correct p-values in omics data association studies. See Hebestreit & Klein (2022) and Akalin et.al (2012) for background and the vignette for an example.

SCpubr v1.0.4: Implements a system that provides a streamlined way of generating publication ready plots for known Single-Cell transcriptomics data. Look here for an online reference manual.

Mathematics

Boov v1.0.0: Provides functions to perform the Boolean operations union, difference and intersection on volumes. Computations are done by the C++ library CGAL. See README for some examples. Also, have a look at the package MinkowskiSum.

fitode v0.1.1: Provides methods and functions for fitting ordinary differential equations that use sensitivity equations to compute gradients of ODE trajectories with respect to underlying parameters. See the vignette for details.

manifold v0.1.1: Implements operations for Riemannian manifolds including geodesic distance, Riemannian metric, and exponential and logarithm maps, and also incorporates a random object generator on the manifolds. See Dai, Lin, and Müller (2021) for details.

Machine Learning

SoftBart v1.0.1: Implements the SoftBart model of described by Linero and Yang (2018) with the optional use of a sparsity-inducing prior to allow for variable selection. The vignette contains theory and examples.

tidyfit v0.5.1: Extends the tidy data environment with functions to fit and cross validate linear regression and classification algorithms on grouped data. There are several vignettes including Predicting Boston House Prices, Multinomial Classification, and Rolling Window Time Series Regression.

Medicine

cities v0.1.0: Provides functions to simulate clinical trials and summarize causal effects and treatment policy estimands in the presence of intercurrent events. Have a look at the demo.

RCT2 v0.0.1: Implements various statistical methods for designing and analyzing two-stage randomized controlled trials using the methods developed by Imai, Jiang, and Malani (2021) and Imai, Jiang, and Malani (2022). There are vignettes on Interference and Causal Inference.

Pharma

DTSEA v0.0.3: Implements a novel tool to identify candidate drugs against a particular disease based on the drug target set enrichment analysis. It assumes the most effective drugs are those with a closer affinity in the protein-protein interaction network to the specified disease. See Gómez-Carballa et al. (2022) and Feng et al. (2022) for disease expression profiles, Wishart et al. (2018) and Gaulton et al. (2017) for drug target information, and Kanehisa et al. (2021) for the details of KEGG database. There is a vignette.

nlmixr2lib v0.1.0: Provides tools to create model libraries for nlmixr2. Models include pharmacokinetic, pharmacodynamic, and disease models used in pharmacometrics. See the vignette Creating a model library.

Statistics

aIc v1.0: Implements set of tests for compositional pathologies including for coherence of correlations as suggested by Erb et al. (2020), compositional dominance of distance, compositional perturbation invariance as suggested by (Aitchison (1992) and singularity of the covariation matrix. See the vignette for details and examples.

ktweedie v1.0.1: Uses Reproducing Kernel Hilbert Space methods to implement Tweedie compound Poisson gamma models with high-dimensional predictors for the analyses of zero-inflated response variables. See the vignette for examples.

missoNet v1.0.0: Implements efficient procedures for fitting conditional graphical lasso models linking predictor variables to response variables or tasks, when the response data may contain missing values. See the vignette for examples.

ShalpeyOutlier v0.1.0: Provides methods to use Shapley values to detect, explain, and cell wise impute multivariate outliers. See Mayrhofer and Filzmoser (2022) for details and the vignette for examples.

SpatialfdaR v1.0.0: Provides functions to that implement finite element analysis methods to spatial functional data analysis. See Sangalli et al. (2013) and Bernardi et al. (2018) for background and the vignette for an example.

Time Series

dfms v0..1.3: Provides a user friendly and computationally efficient approach to estimate linear Gaussian dynamic factor models using Kalman filter and EM algorithm methods. See Doz et al. (2011) and Banbura & Modugno (2014) for background and the vignette for examples.

Utilities

ExclusionTable v1.0.0: Provides functions for creating tables of excluded observations by reporting the number before and after each subset() call together with the number of observations that have been excluded. See the vignette.

shiny.tailwind v0.2.2: Allows TailwindCSS to be used in Shiny apps with just-in-time compiling including custom CSS with @apply directive, and custom tailwind configurations. See README for examples.

Visualization

AlphaHull3D v1.1.0: Provides functions to compute the alpha hull of a set of points (informallly: the shape formed by these points) in 3D space. See README for some visualizations, and also have a look at the related packages MeshesTools, and PolygonSoup.

bangladesh v1.0.0: Provides sf objects, shape files, and functions to draw regional chorpleth maps for Bangladesh. See the vignette.

ggstats v0.1.0: Provides functions to create forest plots of regression model coefficients along with new statistics to compute proportions, weighted mean and cross-tabulation statistics, as well as new geometries to add alternative background color to a plot. There are vignettes on plotting coefficients and on computing cross-tabulation, custom proportions, and weighted means.

jagshelper v0.1.11: Provides tools to streamline Bayesian analyses in JAGSincluding functions for extracting output, streamlining assessment of convergence, and producing summary plots. See the vignette for examples.

roughsf v1.0.0: Provides functions to draw maps, including “sketchy”, hand-drawn-like maps using the Javascript library Roughjs. See README for examples.

September 2022: "Top 40" New CRAN Packages

Thu, 27 Oct 2022 00:00:00 +0000

Two hundred and two new packages made it to CRAN in September. Here are my “Top 40” selections in fourteen categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Pharmacology, Psychology, Science, Social Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

kimfilter v1.0.0: Provides an Rcpp implementation of the multivariate Kim filter, which combines the Kalman and Hamilton filters for state probability inference. The filter is designed for state space models and can handle missing values and exogenous data in the observation and state equations. See Kim et al. (1999) for details and the vignette for examples.

SparseChol v0.1.1: Provides a C++ implementation of sparse LDL decomposition of symmetric matrices and solvers as described in Davis (2016). See README for an example.

Data

allhomes v0.3.0: Provides tools to extract past sales data for specific suburbs and years from the Australian property website including the address and property details, date, price, block size and unimproved value of properties. See README to get started.

kgp 1.1.0: Provides access to the metadata about populations and data about samples from the 1000 Genomes Project, including the 2,504 samples sequenced for the Phase 3 release and the expanded collection of 3,202 samples with 602 additional trios. The data is described in Auton et al. (2015) and Byrska-Bishop et al. (2022), and raw data is available here. See Turner (2022) for details and look here for examples.

eHDPrep v1.2.1: Provides a tool for the preparation and enrichment of health datasets for analysis including functions to assess data quality and enable semantic enrichment of a dataset by discovering metavariables from relationships among input variables determined from user-provided ontologies. See the vignette.

Genomics

refdb v0.1.1: Implements a reference database manager offering a set of functions to import, organize, clean, filter, audit and export reference genetic data and includes functions to download sequence data from Bold Systems and NCBI GenBank. There is an Introduction and a vignette on Downloading and combining data.

RestoreNet v1.0: Implements a random-effects stochastic model that starts from an Ito-type equation describing the dynamics of cells duplication, death and differentiation at clonal level to detect clonal dominance events in gene therapy studies. See Del Core et al., (2022) for details and the vignette for the math and examples.

Machine Learning

DynForest v1.0.0: Implements a random forests model that uses multiple longitudinal predictors to make survival predictions for individual subjects. See Devaux et al.(2022) for the details and the vignette for an example.

multiview v0.4: Provides functions to fit cooperative learning models which are supervised learning models for multiple sets of features (“views”), as described in Ding et al. (2022). See the vignette for an introduction.

survex v0.1.1: Implements methods for explaining survival models. Methods include SurvSHAP(t) as described in Krzyzinski et al. (2022), SurvLIME introduced in Kovalev et al. (2020), as well as methods described in Biecek et al. (2021). See the vignettes Creating custom extensions and Package usage.

voice v0.4.14: Provides general purpose tools for voice analysis, speaker recognition and mood inference. See thevignette for an example.

Mathematics

collatz v1.0.0: Provides functions to explore the Collatz conjecture including the ability to retrieve the hailstone sequence, the stopping time, total stopping time and tree-graph. There are four vignettes including: collat, Hailstone Sequences, and Tree Graphs.

greta.dynamics v0.2.0: Implements a greta extension for analyzing transition matrices and ordinary differential equations representing dynamical systems. Have a look at the iterate-matrix and ode-solve examples.

Medicine

historicalborrow v1.0.4: Implements a hierarchical model and a mixture model to borrow historical control data from other studies to better characterize the control response a study. See Viele et al. (2013) for a discussion of the methods and the vignettes Methods and Usage.

nphRCT v0.1.0: Provides functions to perform a stratified weighted log-rank test in a randomized controlled trial which can be visualized as a difference in average score on the two treatment arms. See Magirr and Burman (2018), Magirr (2020), and Magirr (2022) for a description of the tests and the vignettes Survival tests as differences-of-means and The weighted log-rank test for examples.

Pharmacology

rPBK v0.2.0: Provides functions to fit and simulate any kind of physiologically-based kinetic model which allows for multiple compartments, links between pairs of compartments, and links between compartments and the external medium. See Charles et al. (2022) for background and the vignette for examples.

xhaz v2.0.1: Provides functions to fit relative survival regression models with or without proportional excess hazards and with the additional possibility to correct for background mortality by one or more parameters. See Touraine et al. (2020), Mba et al. (2020), and Goungounga et al. (2019) for a description of the models and the vignette for an introduction.

Psychology

rempsyc v0.0.9: Provides convenience functions for Psychology including functions to customize plots and tables following the style of the American Psychological Association which are exportable to Microsoft Word. There are nine vignettes including Test linear regression assumptions, Planned Contrasts Analyses, and Publication-ready scatter plots.

Science

Karen v1.0: Implements a stochastic framework that combines biochemical reaction networks with extended Kalman filter and Rauch-Tung-Striebel smoothing allowing biologists to investigate the dynamics of cell differentiation from high-dimensional clonal tracking data subject to measurement noise, false negative errors, and systematically unobserved cell types. See Del Core et al. (2022) for details and the vignette for an example.

LMD v1.0.0: Implements Local Mean Decomposition, an iterative and self-adaptive approach for demodulating, processing, and analyzing multi-component amplitude modulated and frequency modulated signals. See Smith (2005) for background and the vignette to get started.

oceanexplorer v0.0.2: Provides tools to explore the NOAA world ocean atlas including functions to extract NetCDF data and visualize physical and chemical parameters. A shiny app allows interactive exploration. Look here for background information and see the vignette for examples.

WormTensor v0.1.0: Implements a toolkit to detect clusters from distance matrices calculated between the cells of multiple animals (Caenorhabditis elegans) from input time-series matrices. Includes functions to generate, cluster, and visualize distance matrices, and to retrieve calculated distance matrices from figshare. See the vignette.

demcon v0.3.0: Implements an open-source toolkit developed by ISciences and the DANTE Project for exploring popular political, institutional, and constitutional datasets with the goal of reducing barriers to entry in political science research by automating common acquisition and pre-processing procedures. This package focuses on the V-Dem dataset. There are four vignettes including A Brief Review of Constitutional Datasets and Country Coding Considerations for Dataset Harmonization.

sdam v1.1.4: Provides tools for performing social dynamics and complexity analyses about the Ancient Mediterranean in the context of the SDAM project based at the Department of History and Classical Studies at Aarhus University. There are vignettes on Dates, Re-encoding people, Datasets, and Maps and Networks.

Statistics

adjustedCurves v0.9.0: Provides functions to estimate and plot confounder-adjusted survival curves using either direct adjustment, inverse probability weighting, empirical likelihood estimation, or targeted maximum likelihood estimation. See Denz et. al (2022) for details and the vignette for an introduction.

CovRegRF 1.0.1: Implements a method that uses random forests to estimate the covariance matrix of a multivariate response given a set of covariates as described in in Alakus et al. (2022). The vignette provides an example.

greta.gp v0.2.0: Provides the syntax to create and combine full rank or sparse Gaussian process kernels in greta. See Golding (2019) for background on greta and the vignette to get started.

fwb v0.1.1: Implements the fractional weighted bootstrap (aka the Bayesian bootstrap) to be used as a drop-in for functions in the boot package. The fractional weighted bootstrap involves drawing weights randomly that are applied to the data rather than resampling units from the data. See Xu et al. (2020) for the theory and README for an example.

glmmrBase v0.1.2: Provides the R6 classes Covariance, MeanFunction and Model to allow for the flexible specification of generalized linear mixed models, and also functions to produce relevant matrices, values, and analyses. See README for details.

rocbc v0.1.1: Provides functions for inferences and comparisons around the AUC, the Youden index, the sensitivity at a given specificity level, the optimal operating point of the ROC curve, and the Youden based cutoff. See Bantis et al. (2018) and Bantis et al. (2021 and the vignette for examples.

vglmer v1.0.2: Provides functions to estimate hierarchical models using mean-field variational Bayes which can accommodate models with an arbitrary number of random effects and requires no integration to estimate. See Goplerud (2022) for details and README for an example.

Time Series

bsvars v1.0.0: Implements MCMC algorithms for Bayesian estimation of Structural Vector Autoregressive (SVAR) models including a wide range of SVAR models. See Lütkepohl & Woźniak (2020), Waggoner & Zha (2003) for background.

gasmodel v0.1.0: Provides functions to estimate, forecast and simulate generalized autoregressive score (GAS) models of Creal, Koopman, and Lucas (2013) and Harvey (2013). There are two case study vignettes Bookshop Orders and Hockey Rankings and another on Probability Distributions.

kalmanfilter v2.0.0: Uses Rcpp to implement a multivariate Kalman filter for state space models that can handle missing values and exogenous data in the observation and state equations. See Kim & Nelson (1999) for details and the vignette for an example.

MultiGlarmaVarSel v1.0: Provides functions to perform variable selection in high-dimensional sparse GLARMA models. See Gomtsyan et al. (2022) for details and the vignette for examples.

VedicDateTime v0.1.1: Provides functions to facilitate conversion between the Gregorian and Vedic calendar systems. See Bokde (2021) and Ramakumar (2011) and the vignette for an overview with examples.

Utilities

bundle v0.1.0P Provides functions to serialize model objects with a consistent interface. See the vignette to get started.

r2resize v1.3: Implements an automatic resizing toolbar for containers, images and tables for markdown, rmarkdown and quarto documents. There is Welcome vignette and another on New features.

Visualization

figuRes2 v1.0.0: Provides functions and supporting documentation to streamline a variety of figure production tasks. There are vignettes on Basics, Forest plots, KM plots, and Production Workflows.

openairmaps v0.5.1: Combines openair air quality maps with leaflet to plot site maps with directional analysis figures such as polar plots, and air mass trajectories. See README for examples.

August 2022: "Top 40" New CRAN Packages

Mon, 26 Sep 2022 00:00:00 +0000

One hundred ninety-four new package made it to CRAN in August. Here are my “Top 40” picks in thirteen categories: “Computational Methods, Data, Epidemiology, Genomics, Insurance, Machine Learning, Mathematics, Medicine, Pharmaceutical Applications, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

brassica v1.0.1: Executes BASIC programs from the 1970s for historical and educational purposes. This enables famous examples of early machine learning, artificial intelligence, natural language processing, cellular automata, and so on, to be run in their original form. See BASIC.

MultistatM v1.1.0: Provides algorithms to build set partitions and commutator matrices used in the construction of multivariate d-variate Hermite polynomials. See the vignette.

Data

hubeau v0.3.1: functions to retrieve data from Hub’Eau the free and public French National APIs on water. See the vignette.

tinytiger v0.0.4: Download geographic shapes from the United States Census Bureau TIGER/Line Shapefiles. See the vignette.

Epidemiology

disbayes v1.0.0: Provides functions to estimate incidence and case fatality for a chronic disease, given partial information, using a multi-state model. See Jackson et al. (2021) for a description of the methods and the vignette for examples.

insectDiseases v1.2.1: Provides the Ecological Database of the Worlds Insect Pathogens created by David Onstad. See the vignette.

nncc v1.0.0: Provides nearest-neighbors matching and analysis of case-control data. See Cui et al. (2022) for background and the vignette for examples.

Genomics

epitopR v0.1.2: Offers a suite of tools to predict peptide MHC (major histocompatibility complex) presentation in the context of both human and mouse. Results are based on half maximal inhibitory concentration as queried through the immune epitope database API. See Vita et al. (2018) for background and the vignette for an example.

geneHapR v1.1.1: Provides functions to import genome variants data and perform gene haplotype Statistics, visualization and phenotype association. There is an Introduction, and there are vignettes on Data, and Workflow.

geniePBC v1.0.0: Linked to The American Association Research Project Genomics Evidence Neoplasia Information Exchange (GENIE) BioPharma Collaborative, this package provides an interface to the data corresponding to each release from Synapse and to prepare datasets for analysis. There are several vignettes including drug regimen sunburst and pull data synapse.

slendr v0.3.0: Implements a framework for simulating spatially explicit genomic data which leverages real cartographic information for programmatic and visual encoding of spatiotemporal population dynamics on real geographic landscapes. Population genetic models are then automatically executed using a custom built-in simulation ‘SLiM’ script as described in Haller et al. (2019). There are several vignettes including A Basic Tutorial, Programming Dispersion Dynamics, and Tree-sequence processing.

Insurance

actxps v0.2.0: Supports actuarial experience studies with functions to prepare data, create studies and perform exposure calculations as described in: Atkinson & McGarry (2016). Look here for an example.

Machine Learning

aorsf v0.0.2: Provides functions to fit, interpret, and predict with oblique random survival forests. See Jaeger et al. (2022) for methods to accelerate and interpret oblique random survival forest models. There is an Introduction and there are vignettes on Predictions and PD and ICE Curves.

calibrationband v0.2.1: Provides functions to assess the calibration of probabilistic classifiers using confidence bands for monotonic functions, and also facilitate constructing inverted goodness-of-fit tests whose rejection allows for a sought-after conclusion of a sufficiently well-calibrated model. See Dimitriadis et al. (2022) for details and README for an example.

kernelshap v0.2.0: Multidimensional version of the iterative Kernel SHAP algorithm from game theory that has become popular for interpreting Machine Learning Models. Covert & Lee (2021) for details and README for an example.

mlim v0.0.9: Uses automated machine learning techniques to fine-tune a Elastic Net, Gradient Boosting, Random Forest, Deep Learning, Extreme Gradient Boosting, or Stacked Ensemble machine learning model for imputing the missing observations of each variable. See README for examples.

tidytags v1.0.2: Facilitates the analysis of Twitter data by coordinates the simplicity of collecting tweets over time with a Twitter Archiving Google Sheet and the utility of the rtweet package for processing and preparing Twitter metadata. There is a Getting Started Guide and a vignette on using conference hashtags.

Mathematics

ambit v0.1.2: Provides tools to simulate and estimate various types of ambit processes, including trawl processes and weighted trawl processes. There is a vignette on Simulating trawl processes and another on estimating the Trawl function.

rTensor2 v0.1.1: Provides a set of tools for basic tensor operators including eigenvalue decomposition, the QR decomposition and LU decompositions, the inverse of a tensor, and the transpose of a symmetric tensor. (A tensor in this context is a multidimensional array.) See Kernfeld et al. (2015) for background.

Medicine

RIbench v1.0.1: Provides a benchmark suite of tools for indirect methods of reference interval estimation. See the vignette.

spiro v0.1.1: Provides functions to import, process, summarize and visualize raw data from metabolic carts. See Robergs, Dwyer, and Astorino (2010) for details and the vignettes Importing & Processing and Summarizing & Plotting.

Pharmaceutical Applications

gtreg v0.1.1: Provides functions that leverage the gtsummary package to creates tables suitable for regulatory agency submissions. See the vignette on adverse event counting methods.

tidyCDISC v0.1.0: Implements a drag and drop Shiny application to allow users to construct [ADaM](https://www.cdisc.org/adam-course#:~:text=The%20Analysis%20Data%20Model%20(ADaM,wide%20variety%20of%20analysis%20methods.) and CDISC compliant tables and plots for studying population and patient level data. See the vignette for more information and look here to run a simulation.

Statistics

cassowaryr v2.0.0: Computes a range of scatterplot diagnostics (scagnostics) on pairs of numerical variables in a data set including the graph and association-based scagnostics described in Wilkinson & Graham (2008) doi:10.1198/106186008X320465 and the association-based scagnostics described by Grimm (2016). See the vignette.

contsurvplot v0.1.1: Provides tools to visualize the causal effect of a continuous variable on a time-to-event outcome including survival area plots, survival contour plots, survival quantile plots and 3D surface plots. See Denz & Timmesfeld (2022) for details and the vignette for examples.

copre v0.1.0: Provides functions for Bayesian nonparametric density estimation using Martingale posterior distributions and includes the Copula Resampling (CopRe) algorithm, a Gibbs sampler for the marginal Mixture of Dirichlet Process (MDP) model, and an extension for full uncertainty quantification via a new Polya completion algorithm. See Moya & Walker (2022), Fong et al. (2021), and Escobar & West (1995) for background and README for an example .

InteractionPoweR v0.1.1: Provides functions for power analysis of regression models which tests the interaction of two independent variables on a single dependent variable. See the tutorial and README to get started.

singR v0.1.1: Implements the SING algorithm to extract joint and individual non-Gaussian components from two datasets. See Risk & Gaynanova (2021) for the theory and the vignette for examples.

SIRthresholded v1.0.0: Implements a version of the Sliced Inverse Regression method which may be used for variable selection. See the vignette.

sparseR v0.2.0: Implements ranked sparsity methods, including penalized regression methods such as the sparsity-ranked lasso, its non-convex alternatives, and elastic net, as well as the sparsity-ranked Bayesian Information Criterion. See Peterson and Cavanaugh (2022) for background and the vignette for an overview.

spmodel v0.1.0: Provides functions to fit, summarize, and predict a variety of spatial statistical models. Modeling features include anisotropy, random effects, partition factors and big data approaches. There is an Overview, a Detailed Guide and a vignette on Technical Details.

TRexSelector v0.0.1: Provides functions to perform fast variable selection in high-dimensional settings while controlling the false discovery rate at a user-defined target level. See Machkour et al. (2021) for background and the vignette for an overview.

VAJointSurv v0.1.0: Provides functions to estimate joint marker (longitudinal) and survival (time-to-event) outcomes using variational approximations which allow for correlated error terms and multiple types of survival outcomes which may be left-truncated, right-censored, and recurrent. See the vignette for some theory and examples.

Time Series

gmwmx v1.0.2: Implements the Generalized Method of Wavelet Moments with Exogenous Inputs estimator (GMWMX) presented in Cucci et al. (2022). There are vignettes on Estimating and comparing models, Removing extreme values in time series, Generating data, and Simulation.

Utilities

countdown v0.4.0: Provides a simple countdown timer for slides and HTML documents written in RMarkdown or Quarto and with Shiny. See README for an example.

D4TAlink.light v2.2.9: Provides workload management tools to facilitate structuring R&D activities to comply with FAIR principles as discussed by Jacobsen et al. (2017) and with ALCOA+ principles as proposed by the FDA. See the vignette.

tinkr v0.1.0: Provides functions to convert Markdown and RMarkdown files to XML and back to allow their editing with xml2 (XPath) instead of numerous complicated regular expressions. See the vignette.

typetracer v0.1.1: The R language includes a set of defined types, but has been described as being “absurdly dynamic” (Turcotte & Vitek (2019), and lacks tools to specify which types are expected by an expression. typetracerprovides functions to extract detailed information on the properties of parameters passed to R functions. See the vignette.

Visualization

atime v2022.9.16: Provides functions for computing and visualizing comparative asymptotic timings of different algorithms and code versions. Also includes functionality for comparing empirical timings with expected references such as linear or quadratic. There are several vignettes including Comparing git versions, Optimal segmentation examples, and Regular Expression examples.

ggsurvfit v0.1.0: Extends ggplot2 to ease the creation of publication ready survival plots. See the vignettes Gallery and Themes.

oblicubes v0.1.2: Extends the isocubes package to provide three-dimensional rendering for grid and ggplot2 graphics using cubes and cuboids drawn with an oblique projection. See the vignette.

ordr v0.1.0: Provides a tidyverse extension for ordinations and biplots. Ordination comprises several multivariate techniques including principal components analysis which rely on eigen-decompositions or singular value decompositions of pre-processed numeric matrix data. The overlay of the resulting shared coordinates on a scatterplot is called a biplot. See Roux & Rouanet (2005) for background. See the vignettes on Covariance Data and Ordination.

Beneath and Beyond the Cox Model

Tue, 06 Sep 2022 00:00:00 +0000

The Cox Proportional Hazards model has so dominated survival analysis over the past forty years that I imagine quite a few people who regularly analyze survival data might assume that the Cox model, along with the Kaplan-Meier estimator and a few standard parametric models, encompass just about everything there is to say about the subject. It would not be surprising if this were true because it is certainly the case that these tools have dominated the teaching of survival analysis. Very few introductory textbooks look beyond the Cox Model and the handful of parametric models built around Gompertz, Weibull and logistic functions. But why do Cox models work so well? What is the underlying theory? How do all the pieces of the standard survival tool kit fit together?

As it turns out, Kaplan-Meier estimators, the Cox Proportional Hazards model, Aalen-Johansen estimators, parametric models, multistate models, competing risk models, frailty models and almost every other survival analysis technique implemented in the vast array of R packages comprising the CRAN Survival Task View, are supported by an elegant mathematical theory that formulates time-to-event analyses as stochastic counting models. The theory is about thirty years old. It was initiated by Odd Aalen in his 1975 Berkeley PhD dissertation, developed over the following twenty years largely by Scandinavian statisticians and their collaborators, and set down in more or less complete form in two complementary textbooks [5 and 9] by 1993. Unfortunately, because of its dependency on measure theory, martingales, stochastic integrals and other notions from advanced mathematics, it does not appear that the counting process theory of survival analysis has filtered down in a form that is readily accessible by practitioners.

In this rest of this post, I would like to suggest a path for getting a working knowledge of this theory by introducing two very readable papers, which taken together, provide an excellent overview of the relationship of counting processes to some familiar aspects of survival analysis.

The first paper, The History of applications of martingales in survival analysis by Aalen, Andersen, Borgan, Gill, and Keiding [4] is a beautiful historical exposition of the counting process theory by master statisticians who developed a good bit of the theory themselves. Read through this paper in an hour of so and you will have an overview of the theory, see elementary explanations for some of the mathematics involved, and gain a working idea of how the major pieces of the theory fit together how they came together.

The second paper, Who needs the Cox model anyway? [7], is actually a teaching note put together by Bendix Carstensen. It is a lesson with an attitude and the R code to back it up. Carstensen demonstrates the equivalence of the Cox model to a particular Poisson regression model. Working through this note is like seeing a magic trick and then learning how it works.

The following reproduces a portion of Carstensen’s note. I provide some commentary and fill in a few elementary details in the hope that I can persuade you that it is worth the trouble to spend some time with it yourself.

Carstensen use the North Central Cancer Treatment Group lung cancer survival data set which is included in the survival package for his examples.

## # A tibble: 228 × 10
##     inst  time status   age   sex ph.ecog ph.karno pat.karno meal.cal wt.loss
##    <dbl> <dbl>  <dbl> <dbl> <dbl>   <dbl>    <dbl>     <dbl>    <dbl>   <dbl>
##  1     3   306      2    74     1       1       90       100     1175      NA
##  2     3   455      2    68     1       0       90        90     1225      15
##  3     3  1010      1    56     1       0       90        90       NA      15
##  4     5   210      2    57     1       1       90        60     1150      11
##  5     1   883      2    60     1       0      100        90       NA       0
##  6    12  1022      1    74     1       1       50        80      513       0
##  7     7   310      2    68     2       2       70        60      384      10
##  8    11   361      2    71     2       2       60        80      538       1
##  9     1   218      2    53     1       1       70        80      825      16
## 10     7   166      2    61     1       2       70        70      271      34
## # … with 218 more rows

It may not be obvious at first because there is no subject ID column, but this data frame contains one row for each of 228 subjects. The first column is an institution code. time is the time of death or censoring. status is the censoring indicator. The remaining columns are covariates. I select time, status, sex and age, drop the others from the our working data frame, and then replicate Carstensen’s preprocessing in a tidy way. The second line of mutate() adds a small number to each event time to avoid ties.

set.seed(1952)
lung <- lung %>% select(time, status, age, sex) %>% 
                  mutate(sex = factor(sex,labels=c("M","F")),
                         time = time + round(runif(nrow(lung),-3,3),2))

To get a feel for the data we fit a Kaplan-Meier Curve stratified by sex.

surv.obj <- with(lung, Surv(time, status == 2))
fit.by_sex <- survfit(surv.obj ~ sex, data = lung, conf.type = "log-log")
autoplot(fit.by_sex,
          xlab = "Survival Time (Days) ", 
          ylab = "Survival Probabilities",
         main = "Kaplan-Meier plot of lung data by sex") +  
 theme(plot.title = element_text(hjust = 0.5))

Next, following Carstensen, I fit the baseline Cox model to be used in the model comparisons below.

m0.cox <- coxph( Surv( time, status==2 ) ~ age + sex, data=lung )
summary( m0.cox )

## Call:
## coxph(formula = Surv(time, status == 2) ~ age + sex, data = lung)
## 
##   n= 228, number of events= 165 
## 
##          coef exp(coef) se(coef)     z Pr(>|z|)   
## age   0.01705   1.01720  0.00922  1.85   0.0643 . 
## sexF -0.52033   0.59433  0.16751 -3.11   0.0019 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
##      exp(coef) exp(-coef) lower .95 upper .95
## age      1.017      0.983     0.999     1.036
## sexF     0.594      1.683     0.428     0.825
## 
## Concordance= 0.603  (se = 0.025 )
## Likelihood ratio test= 14.4  on 2 df,   p=7e-04
## Wald test            = 13.8  on 2 df,   p=0.001
## Score (logrank) test = 14  on 2 df,   p=9e-04

The hazard ratios for age and sexF are given in the output column labeled exp(coef). As Carstensen points out mortality increases by 1.7% per year of age at diagnosis and that women have 40% lower mortality than men.

Carstensen next shows that this model can be exactly replicated by a particular and somewhat peculiar Poisson model. Doing this requires a shift in how time is conceived. In the Kaplan-Meier estimator and the Cox model, time is part of the response vector. In the counting process formulation, time is a covariate. Time is divided into many small intervals of length h in which an individuals “exit status” , d is recorded. d will be 1 if death occurred or 0 otherwise. The h intervals represent an individual’s risk time. The pair (d, h) are used to calculate an empirical rate for the process which corresponds to the theoretical rate:

λ(t) = lim_h→0 P[event in (t, t + h)| at risk at time t]/h (*)

The first step in formulating a Poisson model is to set up a data structure that will allow for this more nuanced treatment of time. The function Lexis from the Epi package creates an object of class Lexis, a data frame with columns that will be used to distinguish event time (death or censoring time) from the time intervals in which subjects are at risk for the event. Collectively, these intervals span period from when the first until the last recorded time.

Lung <- Epi::Lexis( exit = list( tfe=time ),
  exit.status = factor( status, labels=c("Alive","Dead") ),
  data = lung )

## NOTE: entry.status has been set to "Alive" for all.
## NOTE: entry is assumed to be 0 on the tfe timescale.

head(Lung)

##  lex.id tfe lex.dur lex.Cst lex.Xst   time status age sex
##       1   0   308.7   Alive    Dead  308.7      2  74   M
##       2   0   457.4   Alive    Dead  457.4      2  68   M
##       3   0  1008.6   Alive   Alive 1008.6      1  56   M
##       4   0   212.1   Alive    Dead  212.1      2  57   M
##       5   0   885.5   Alive    Dead  885.5      2  60   M
##       6   0  1023.7   Alive   Alive 1023.7      1  74   M

The new variables are:

lex.id - Subject ID
tfe - Time from entry at the beginning of the follow-up interval
lex.dur - Duration of the follow-up interval
lex.Cst - Entry status (Alive in our case)
lex.Xst - Exit status at the end of the follow-up interval: tfe + lex.dur (Either Alive or Dead in our case)

Next, the time variable is sorted to produce a vector of endpoints for the at risk intervals and a new time-split Lexis data frame is created using the splitMulti() function from the popEpi package.

Lung.s <- splitMulti( Lung, tfe=c(0,sort(unique(Lung$time))) )
head(Lung.s)

##  lex.id   tfe lex.dur lex.Cst lex.Xst  time status age sex
##       1  0.00    7.67   Alive   Alive 308.7      2  74   M
##       1  7.67    1.88   Alive   Alive 308.7      2  74   M
##       1  9.55    0.23   Alive   Alive 308.7      2  74   M
##       1  9.78    0.57   Alive   Alive 308.7      2  74   M
##       1 10.35    2.25   Alive   Alive 308.7      2  74   M
##       1 12.60    0.45   Alive   Alive 308.7      2  74   M

summary( Lung.s )

##        
## Transitions:
##      To
## From    Alive Dead  Records:  Events: Risk time:  Persons:
##   Alive 25941  165     26106      165      69632       228

tfe tracks the time from entry into the study. This is calendar time.

Carstensen then fits a Cox model to both the Lexis data set and the time-split Lexis data set and notes that the results match the original baseline Cox model. This is as one would expect since the three different data frames contain the same information. Nevertheless, it is a pleasant surprise that the coxph() and Surv() functions are flexible enough to assimilate the three different input data formats.

mL.cox <- coxph( Surv( tfe, tfe+lex.dur, lex.Xst=="Dead" ) ~ age + sex, eps=10^-11, iter.max=25, data=Lung )
mLs.cox <- coxph( Surv( tfe, tfe+lex.dur, lex.Xst=="Dead" ) ~ age + sex, eps=10^-11, iter.max=25, data=Lung.s )
round( cbind( ci.exp(m0.cox), ci.exp(mL.cox), ci.exp(mLs.cox) ), 6 )

##      exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5%
## age     1.0172 0.999 1.0357    1.0172 0.999 1.0357    1.0172 0.999 1.0357
## sexF    0.5943 0.428 0.8253    0.5943 0.428 0.8253    0.5943 0.428 0.8253

Now, Carstensen executes what would seem to be a very strange modeling maneuver. He turns calender time, tfe into a factor and fits a Cox model with tfe as a covariate.

mLs.pois.fc <- glm( cbind(lex.Xst=="Dead",lex.dur) ~ 0 + factor(tfe) + age + sex, family=poisreg, data=Lung.s )

An important technical point is that the time intervals in equation (*) above do not satisfy the independence assumption for a Poisson regression model. Nevertheless, the standard glm() machinery can be used to fit the model because, as Carstensen demonstrates, the likelihood function for the conditional probabilities is proportional to the partial likelihood function of the Cox model.

cbind( ci.exp(mLs.cox),ci.exp( mLs.pois.fc, subset=c("age","sex") ) )

##      exp(Est.)  2.5%  97.5% exp(Est.)  2.5%  97.5%
## age     1.0172 0.999 1.0357    1.0172 0.999 1.0357
## sexF    0.5943 0.428 0.8253    0.5943 0.428 0.8253

Carstensen concludes that this demonstrates that the Cox model is equivalent to a specific Poisson model which has one rate parameter for each time internal, and emphasizes that this is not a new result. He notes that the equivalence was demonstrated some time ago, theoretically by Theodore Holford [10], and in practice, by John Whitehead [14]. Also, in a vignette [15] for the survival package Zhong et al. state that this trick may be used to approximate a Cox model.

Carstensen then demonstrates that more practical Poisson models can be fit by using splines to decrease the number of at risk intervals. The first uses a spline basis with arbitrary knot locations and the second fits a penalized spline gam model.

splines

t.kn <- c(0,25,100,500,1000) 
mLs.pois.sp <- glm( cbind(lex.Xst=="Dead",lex.dur) ~ Ns(tfe,knots=t.kn) + age + sex, family=poisreg, data=Lung.s )

Penalized splines

mLs.pois.ps <- mgcv::gam( cbind(lex.Xst=="Dead",lex.dur) ~ s(tfe) + age + sex, family=poisreg, data=Lung.s )

Carstensen finishes up this portion of his analysis by noting the similarity of the estimates of age and sex effects from the different models.

ests <-
 rbind( ci.exp(m0.cox),
 ci.exp(mLs.cox),
 ci.exp(mLs.pois.fc,subset=c("age","sex")),
 ci.exp(mLs.pois.sp,subset=c("age","sex")),
 ci.exp(mLs.pois.ps,subset=c("age","sex")) )

cmp <- cbind( ests[c(1,3,5,7,9) ,],
 ests[c(1,3,5,7,9)+1,] )

rownames( cmp ) <-
 c("Cox","Cox-split","Poisson-factor","Poisson-spline","Poisson-penSpl")

 colnames( cmp )[c(1,4)] <- c("age","sex")
round( cmp,5 )

##                  age   2.5% 97.5%    sex   2.5%  97.5%
## Cox            1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Cox-split      1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Poisson-factor 1.017 0.9990 1.036 0.5943 0.4280 0.8253
## Poisson-spline 1.016 0.9980 1.035 0.5993 0.4316 0.8322
## Poisson-penSpl 1.016 0.9983 1.035 0.6021 0.4338 0.8358

This demonstration provides some convincing evidence that both parametric and non-parametric models are part of a single underlying theory! When you think about it, this is an astonishing idea. To further explore the counting process theory of survival models, I provide a definition of Aalen’s multiplicative intensity model and a list of references below that I hope you will find helpful.

Finally, there is much more to Carstensen’s note than I have presented. He goes on to provide a fairly complete analysis of the lung data while looking at cumulative rates, survival, practical time splitting, time varying coefficients and more ideas along the way.

Appendix: Multiplicative Intensity Model

For some direct insight into how the Cox Proportional Hazards model fits into the counting process theory have a look at Odd Aalen’s definition of the multiplicative intensity model. Aalan begins his landmark 1978 paper Nonparametric Inference For a Family of Counting Processes] [1] by defining the fundamental components of his multiplicative intensity model.

Let N = (N₁, . . . N_k) be a multivariate counting process which is a collection of univariate counting processes on the interval [0,t] each of which counts events in [0,t]. The N_i may depend on each other. Let σ(F_t) be the sigma algebra which represents the collection of all events that can be determined to have happened by time, t. Let α = α₁, . . . α be an unknown, non-negative function and let Y = (Y₁, . . . Y_k) be a process observable over [0,t].

Define Λ_i(t) = lim_h→0E(N_i(t + h) - N_i(t) | F_t )/h i = 1, … k, to be the the intensity process of the counting process N.

Then the multiplicative intensity model is defined to be: Λ_i(t) = α_i(t)Y_i(t).

This last line certainly looks like the Cox model, and it is not to difficult to confirm that this is indeed the case. You can find the gory details in Fleming and Harrington [9 p 126] or comprehensive monograph by Andersen et al. [5 p481].

References

I believe that the following references (annotated with a few comments) comprise a reasonable basis from gaining familiarity with the counting process approach to survival modeling.

Aalen (1978) Odd O. Aalen Nonparametric Inference For a Family of Counting Processes. The Annals of Statistics 1978, vol 6, no 4, 701-726 This is the ‘Ur’ paper for the multiplicative intensity process. At least the first half should be approachable with some knowledge of measure theory and conditional expectation.
Aalen & Johansen (1978) Odd O. Allen and Soren Johansen. An Empirical Transition Matrix for Non-homogenous Markov Chains Based on Censored Observations. Scand J Statistics 5: 141-150, 1978. This is the source of the Aalen-Johansen estimator. The etm package provides an implementation.
Aalen et al. (2008) Odd O. Aalen, Ørnulf Borgan and Håkon K. Gjessing. Survival and Event History Analysis; A Process Point of View Springer Verlag 2008. This is definitely the text to read first. It is comprehensive, takes a modern point of view, is well written, and introduces the difficult mathematics without all of the technical details that often slow down the process of learning some new mathematics.
Aalen et al. (2009). Odd O. Aalen, Per Kragh Andersen, Ørnulf Borgan, Richard D. Gill and Niels Keiding. History of applications of martingales in survival analysis vol 5, no 1, June 2009
Andersen et al. (1993) Per Kragh Andersen, Ørnulf Borgan, Richard D. Gill, Niels Keiding. Statistical Models Based on Counting Processes. Springer-Verlag, 1993 This text presents numerous examples along with a discussion of the theory and emphasizes parametric models.
Borgan (1997). Ørnulf Borgan Three contributions to the Encyclopedia of Biostatistics: The Nelson-Aalen, Kaplan-Meier, and Aalen-Johansen estimators - Very clear summaries.
Carstensen (2019) Bendex Carstensen. Who needs the Cox model anyway?
Cox (1972) Regression Models and Life-Tables JRSS Vol. 34, No.2, 187-200
Fleming & Harrington (1991). Thomas R. Fleming and David P. Harrington. Counting Processes & Survival Analysis, John Wiley & Sons, Inc. 1991 This text book develops all of the math needed and goes on to study non-parametric models including the Cox model.
Holford. Life table with concomitant information. Biometrics, 32:587{597, 1976.
Mohammed (2019). Introduction to Survival Analysis using R
Threneau (2022). The survival package (vignette)
Therneau et al. (1990). Martingale based residuals for survival models. Biometrika 77, 147-160. Martingale residuals appear to be the one use case where martingales openly surface in the survival calculations.
Whitehead (1980). Fitting Cox’s regression model to survival data using GLIM. Applied Statistics, 29(3):268{275, 1980.
Zhong et al. (2019). Approximating a Cox Model. This is a survival package vignette.

July 2022: "Top 40" New CRAN Packages

Mon, 29 Aug 2022 00:00:00 +0000

Ninety-four new packages stuck to CRAN in July. Here are my “Top 40” selections in thirteen categories: Climate Modeling, Computational Methods, Data, Ecology, Genomics, Machine Learning, Mathematics, Medicine, Networks, Proteomics, Statistics, Utilities, and Visualization.

Climate Modeling

climetrics v1.0-5: Provides a framework that facilitates the spatio-temporal analysis of climate dynamics through exploring and measuring different dimensions of climate change in space and time. See the vignette.

HurreconR v1.0: Implements the HURRECON model which estimates wind speed, wind direction, enhanced Fujita scale wind damage, and duration of EF0 to EF5 winds as a function of hurricane location and maximum sustained wind speed. See Boose et al. (2001) and Boose et al. (2004) for background, and the vignette for an overview.

TrenchR v0.1.0: Provides tools for translating environmental change into organism response. The biophysical modeling tools include both general models for heat flows and specific models to predict body temperatures for a variety of ectothermic taxa. See Gates (1980) for background. There are vignettes on Allometries and conversions, Estimating microclimates, and Estimating body temperatures.

Computational Methods

elastes v0.1.6: Provides functions to compute functional elastic shape means over sets of open planar curves using a novel approach where planar curves are treated as complex functions. Also, a full Procrustes mean is estimated from the corresponding smoothed Hermitian covariance surface. See Steyer et al. (2022) and Stöcker et. al. (2022) for background and the vignette for examples.

ino v0.1.0: Provides a comprehensive toolbox for comparing different initialization strategies when performing numerical optimization. There is an Introduction and a vignette with an HMM example.

Data

giedata v0.1.0: Provides functions to access the API for GIE, Europe’s Gas Infrastructure database. See README to get started.

Ecology

rgrass v0.3-3: Implements a new interface to the GRASS geographical information system that supports both starting R from within the GRASS environment, or running a free-standing R session in a temporary GRASS location. There is an Introduction and a vignette on object formats.

riverconn v0.3.22: Provides functions to calculate indices for river network connectivity. See Jumani et al. (2021) for a review of the incdices and Baldan et al. (2022) for a list of package capabilities and architecture. Have a look at the vignette for examples.

Genomics

dcifer v1.1.1: Implements Dcifer (Distance for complex infections: fast estimation of relatedness), an identity by descent based method to calculate genetic relatedness between polyclonal infections from biallelic and multiallelic data. See Gerlovina et al. (2022) for the details, and the vignette for an example.

qPCRtools v0.1.1: Provides methods to calculate the amplification efficiency of genes, a crucial step in the qPCR process. See the vignette.

scAnnotate v0.0.4: Implements a data-driven cell type annotation tool for single-cell RNA sequencing data. See Ji et al. (2022) for details and the vignette for an introduction.

Machine Learning

cito v1.0.0: Provides functions for building and training custom neural networks using torch. See the vignette.

etree v0.1.0: Provides functions to implement Energy Trees, statistical models that perform classification and regression with structured and mixed-type data. See Giubilei et al. (2022) for a description of Energy Trees and the vignettes eforest and etree for examples.

LDABiplots v0.1.2: Provides tools to extract, explore, analyze, and visualize news published on the web by digital newspapers using Latent Dirichlet and machine learning algorithms. See Ble et al. (2003) and Gabriel (1971) for background. There are English and Spanish versions of the vignette.

mildsvm v0.4.0: Implements a support vector machine classifier for weakly supervised, multiple instance data. See Kent and Yu (2022) and Muandet et al. (2012) for the theory and README for an example.

nestedcv v0.2.3: Provides functions to perform nested cross-validation as described in Stone (1977) for lasso and elastic-net regularized linear models for glmnet. See the vignette.

Mathematics

delaunay v1.1.0: Provides functions to construct and visualize 2D and 3D Delaunay triangulations. See README for some visual examples.

Medicine

activeAnalyzer v1.0.4: Provides a tool to analyse Actigraphy accelerometer data using PROactive Physical Activity in COPD (chronic obstructive pulmonary disease) instruments. See the vignette for examples.

pedbp v1.0.0: Provides data and utilities for estimating pediatric blood pressure percentiles by sex, age, and height. See Lo et al. (2013) and Martin et al. (2022) for background, and the vignette for an overview.

tipmap v 0.1.7: Implements tipping point analysis for clinical trials using Bayesian dynamic borrowing via robust meta-analytic predictive priors as described in Best et al. (2021).

Networks

ergm.multi v0.1.0: Provides a set of extensions for the ergm package to fit multilayer, multiplex, and multirelational networks as well as samples of multiple networks. See Krivitsky et al. (2020) and Krivitsky et, al. (2022) for the details and the vignette for an extended example.

Families v1.0.1: Provides tools to study kinship networks, grand parenthood, and double burden (presence of children and oldest old parents) in virtual population produced by VirtualPop. See the vignette for examples.

Proteomics

promor v0.1.0: Implements a comprehensive set of tools for label-free proteomics data analysis and machine learning modeling including differential expression analysis, predictive modeling and performance assessment. Data from MaxQuant may be used. See the vignette for an introduction.

Statistics

GPCERF v0.1.0: Provides a non-parametric Bayesian framework based on Gaussian process priors for estimating causal effects of a continuous exposure and detecting change points in the causal exposure response curves using observational data. See Ren et al. (2021). Have a look at the Introduction and the vignettes Full Gaussian Process and Nearest-neighbor Gaussian Process.

lnmCluster v0.3.1: Extends the logistic normal multinomial (LNM) clustering model proposed by Fang and Subedi (2020) to provide LNM clustering for compositional data. Details of model assumptions and interpretation can be found in the papers Tu & Subedi (2021) and Tu & Subedi (2022). See the vignette for an example.

tlars v0.0.1: Provides functions to compute the solution path of the Terminating-LARS (T-LARS) algorithm. See Machkour et al. 2022, Efron et al. (2004) and Tibshirani (1996) for the theory, and the vignette for examples.

Utilities

codebookr v0.1.5: Provides functions to create code books (i.e. data dictionaries) directly from a data frame. See README for examples.

datadictionary v0.1.0: Provides tools to creates a data dictionary from any dataset in an R environment. It includes functions to add variable labels and write to Excel.

dtrackr v0.2.4: Provides tools to track and document dplyr data pipelines. As you filter, mutate, and join your way through a data set, functions seamlessly track data flow and generate publication ready documentation of the data pipeline. See the vignettes Consort statment example, Configuration example, Basic Operation, and Joining data pipelines.

howler v0.2.0: Enables audio interactivity within shiny applications using howler.js. See the vignette.

ixplorer v0.2.2: Provides tools to create and view tickets in gitea, a self-hosted git service, using an RStudio addin. It includes helper functions to publish documentation and use git. There are several vignettes including ixplorer basics and Credential management.

nextGenShinyApps v1.5: Provides responsive tools for designing and developing Shiny dashboards and applications. The scripts and style sheets are based on jQuery and Bootstrap. See the vignette for examples.

nplyr v0.1.0: Provides functions for manipulating nested data frames in a list-column using dplyr without first having to unnest()them. See the vignette.

renderthis v0.1.0: Provides tools to render slides to different formats, including html, pdf, png, gif, pptx, and mp4, as well as tool to make a png file of the first slide of a presentation that is re-sized for sharing on social media. There is an Overview and a vignette on basic usage.

video video: 0.1.0: Enables video interactivity within shiny applications using video.js. See the vignette.

Visualization

ensModelVis v0.1.0: Provides a function to display model fits for multiple models and their ensembles. There are vignettes for Classification and Regression.

ggpackets v0.2.0: Provides tools to create groups of ggplot2 layers that can be easily migrated from one plot to another, reducing redundant code and improving the ability to format many plots that draw from the same source. See the Getting Started Guide and the vignettes Composing Templates & Functions and Misc Examples.

ggseqplot v0.7.2: Provides wrappers to render TraMinR sequence plots in ggplot2. See the vignette.

RcppColors v0.1.1: Provides C++ header files to deal with color conversion from color spaces to hexadecimal with Rcpp, and exports some color mapping functions to R. Look here for examples.

xadmix 1.0.0: Provides functions that provide a quick way of subsetting genomic admixture data and generating customizable stacked barplots. See the vignette.

June 2022: "Top 40" New CRAN Packages

Fri, 29 Jul 2022 00:00:00 +0000

One hundred eighty-nine new packages made it to CRAN in June. Here are my “Top 40” selections in eleven categories: Computational Methods, Data, Ecology, Genomics, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Utilities, and Visualizations.

Computational Methods

itp v1.2.0: Implements the interpolate, truncate, project root-finding algorithm developed by Oliveira & Takahashi (2021). The vignette provides an overview.

QR v0..1.3: Provides a function to perform QR factorization without pivoting to a real or complex matrix. It is based on LAPACK. See the vignette.

qsplines v1.0.0: Provides functions to create quaterion splines. See Barry & Goldman (1988) and Kochanek & Bartels (1984) for the details and look here for an example.

VMDecomp v1.0.1: Implements the variational mode decomposition and two-dimensional variational mode decomposition algorithm. See Dragomiretskiy & Zosso (2014) for background and the vignette for examples.

Data

cmch v0.2.0: Implements a wrapper around the Canadian Mortgage and Housing Corporation web interface and enables programmatic and reproducible access to a wide variety of housing data. See the vignette for examples.

EDIutils v1.0.1: Implements a client for the Environmental Data Initiative repository REST API and provides access to ecological data and metadata. There are five short vignettes: Evaluate & upload, Citation Metrics, Download Metrics, Search andaccess, and Tests.

globaltrends v0.0.12: Provides functions to access global search volumes from the Google Trends portal. This working paper outlines the package’s methodological foundations and potential applications. See the vignette to get started.

kaigiroku v0.5: Allows users to search and download data from the API for Japanese Diet proceedings. Look here for examples.

NasdaqDataLink v1.0.0: Provides functions to interact directly with the Nasdaq Data Link API and obtain data in a number of formats. Look here for API documentation and here for package information.

stortingscrape v0.1.1: Provides functions for retrieving data from the Norwegian Parliament, through the Norwegian Parliament API. See the vingette for an introduction.

Ecology

PointedSDMs v1.0.6: Provides tools to build integrated species distribution models and includes tools to run spatial cross-validation and plotting. See Issac et al. (2020) for and introduction to the methods. There is a Setophaga Example and an example for the Solitary Tinamou.

restoptr v1.0.1: Implements a flexible framework for ecological restoration planning that aims to identify priority areas for restoration efforts using optimization algorithms described in Justeau-Allaire et al. 2021. See the vignette to get started.

Genomics

scapGNN v0.1.1: Implements a single cell active pathway analysis tool based on the graph neural network algorithm described in Scarselli et al. (2009) and Kipf & Welling (2017). This may be used to construct a gene-cell association network, infer pathway activity scores from different single cell modalities data and more. See the vignette for an overview and examples.

SRTsim v0.99.2: Implements an independent, reproducible, and flexible Spatially Resolved Transcriptomics simulation framework that can be used to facilitate the development analytical methods and for a wide variety of SRT-specific analyses. See the vignette.

xQTLbiolinks v1.1.1: Implements tools to query, download, and visualize of molecular quantitative trait locus and gene expression data from public resources through the GTEx API. There is a Quick Start Guide and vignettes on Colocalization, Specivicity, and Visualization.

Machine Learning

agua v0.0.1: Enables users to specify h2o as an engine for several tidymodels modeling methods. See README for examples.

MagmaClustR V1.0.0: Implements two main algorithms, called Magma (Leroy et al. (2022) and MagmaClust (Leroy et al. (2020)), using a multi-task Gaussian processes (GP) model to perform predictions for supervised learning problems. See README for examples.

openai v0.1.0: Provides a wrapper for OpenAI API endpoints including engines, completions, edits, files, fine-tunes, embeddings and legacy searches, classifications, and answers endpoints. See README to get started.

sketching v0.1.0: Provides functions to construct sketches of data via random subspace embeddings. See Lee & Ng (2022) for the theory and the vignette for examples.

webmorphR v0..1.1: Provides functions to create reproducible image stimuli, specialised for face images with psychomorph or webmorph templates. See README to get started.

Mathematics

GeneralizedWendland v0.5-2: Implements the fully parameterized generalized Wendland covariance function for use in Gaussian process models, as well as multiple methods for approximating it via covariance interpolation. The available methods are linear interpolation, polynomial interpolation, and cubic spline interpolation. See Bevilacqua et al. (2022) and the vignette for examples.

jacobi v2.0.0: Evaluates Jacobi theta functions and related functions including the Weierstrass elliptic function, the Weierstrass sigma function, the Weierstrass zeta function, the Klein j-function, the Dedekind eta function, the lambda modular function, Jacobi elliptic functions, Neville theta functions, and the Eisenstein series for real and complex variables. Look here for some images.

Medicine

clinicalsignificance v1.0.0: Implements the clinical significance algorithm proposed by Jacobson et al. (1984) to determine if an intervention has a meaningful practical effect. There is a Getting Started Guide and vignettes on Cutoffs and Plots.

PlatformDesign v1.0.1: Provides functions to calculate design parameters for an optimal two-period, multi-arm platform design allowing pre-planned deferred arms to be added during the trial. See Dunnett (1955) for background and the vignette for some theory and examples.

Statistics

bayesassurance v0.1.0: Provides functions to compute Bayesian assurance under various settings characterized by different assumptions and objectives, including precision-based conditions, credible intervals, and goal functions. See Pan & Banerjee (2021) for the theory. There are vignettes for using closed form solutions, the conjugate linear model, and precision based conditions.

DSSP v0.1.1: Provides functions to draw samples from the direct sampling spatial prior model as described in White, Sun, & Speckman (2019). See the vignette for examples.

edibble v0.1.0: Implements a system to facilitate designing comparative experiments using the grammar of experimental designs. See the edibble-book for documentation.

mixgb v0.1.0: Implements a method for multiple imputation using XGBoost, bootstrapping and predictive mean matching as described in Deng and Lumley (2021). There is an Introduction and a vignette on Imputing new data with a saved imputer.

outerbase v0.1.0: Implements in new method for high-dimensional regression using outer product models. See Plumlee (2014) and Plumlee et al. (2021) for background. There is a Getting started guide, a Base walkthrough, and vignettes on Learning from data and Speeding up inference.

PFIM v5.0: Provides functions to evaluate or optimize designs for nonlinear mixed effects models using the Fisher Information matrix. See Malle & Baccar D (1997) and Retout et al. (2007) for background and the vignettes Design evaluation and optimixation (01), Design evaluation and optimixation (02), and Library of models for examples.

VirtualPop v1.0.2: Provides functions to generate lifespans and fertility histories in continuous time using individual-level state transition (multi-state) models and data. See the vignettes on Simulation of life histories, Sampling from waiting time distributions, Simulation of individual fertility careers, and Validation.

Time Series

kssa v0.0.1: Implements the known sub-sequence algorithm described in Benavides et al. (2022), which helps to automatically identify and validate the best method for missing data imputation in a time series. Look here for examples.

ts2net v0.1.0: Implements methods to transform time series into networks, a technique which may be useful for complex systems modeling, time series data mining, or time series analysis using networks. For an introduction to the topic and descriptions of the methods see Mitchell (2006), Silva & Zhao (2016), and Silva et al. (2021). See README to get started.

Utilities

cppchedkR Allows users to run Cppcheck on C/C++ files as an R command or an RStudio addin. See README. .

gtExtras v0.4.1: Provides additional functions for creating tables with gt. See README for examples.

Visualization

ggpie v0.2.2: Provides functions for creating pie, donut and rose pie plots with ggplot2. See the vignette.

ggtrace v0.2.0: Provides ggplot2 geoms that allow groups of data points to be outlined or highlighted for emphasis. See the vignettes Trace lines and Trace points.

Morphoscape v1.0.0: Implements adaptive landscape methods first described by Polly et al. (2016) for the integration, analysis and visualization of biological trait data on a phenotypic morphospace which are typically defined by shape metrics. See the vignette.

r3js v0.0.1: Provides R and JavaScript functions to allow WebGL-based 3D plotting using the three.js library. See the vignettes: Getting Started, Creating a plot from scratch, and Grouping plot elements.

rgl2gltf v1.0.0: Provides functions to work with glTF files which are used to describe 3D models. See the vignette for examples..

shapviz v0.2.0: Provides functions to visualize SHapley Additive exPlanations (SHAP), such as waterfall plots, force plots, various types of importance plots, and dependence plots. See Lundberg & Lee (2017) for background and the vignette for examples.

May 2022: "Top 40" New CRAN Packages

Tue, 28 Jun 2022 00:00:00 +0000

One hundred seventy-nine new packages made it to CRAN in May. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Ecology, Epidemiology, Finance, Machine Learning, Networks, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

graDiEnt v1.0.1: Implements the derivative-free, optim-style Stochastic Quasi-Gradient Differential Evolution optimization algorithm published in Sala, Baldanzini, and Pierini (2018) that uses population members to build stochastic gradient estimates. See README for an example.

rxode2 V2.0.7: Provides facilities for running simulations from ordinary differential equation models, such as pharmacometrics and other compartmental models, but requires both C and Fortran compilers. See the vignette for an example.

ScaleSpikeSlab v1.0: Provides a scalable Gibbs sampling implementation for high dimensional Bayesian regression with the continuous spike-and-slab prior described in Biswas, Mackey & Meng (2022). See README for an example.

Data

bluebike v0.0.3: Provides functions that facilitate importing and working with the Boston Blue Bike Data Set including functions to compute trip distances and map the locations of stations within a given radius. See the vignette for examples.

eurodata v1.4.2: Implements an interface to Eurostat’s Bulk Download Facility with fast data.table based import of data, labels, and metadata along with data search and data description and comparison functions. See README to get started.

gbifdh v0.1.2: Implements a high performance interface to the Global Biodiversity Information Facility that supports large-scale analyses using SQL or dplyr operations on complete tables. See the vignette for examples.

getwiki v0.9.0: Implements a simple wrapper for Wikipedia data to retrieve text in a tidy format that can be used for Natural Language Processing. See the vignette.

Ecology

FIESTA v3.4.1: Implements an estimation tool for analysts that work with sample-based inventory data from the U.S. Department of Agriculture, Forest Service, Forest Inventory and Analysis Program. There are nine vignettes including manuals for Module Estimates and Population Data, and Small Area Estimators and Spatial Tools.

soiltestcorr v2.1.2: Provides functions designed to assist users on the correlation analysis of crop yield and soil test values including functions to estimate crop response patterns to soil nutrient availability and critical soil test values using various approaches. See Correndo et al. (2017), Cate & Nelson (1971), Anderson & Nelson (1975), Bullock & Bullock (1994) and Melsted & Peck (1977) for background. There are seven vignettes including an Introduction and Quadratic-plateau response.

sspm v0.9.1: Implement a gam-based spatial surplus production model, aimed at modeling northern shrimp population in Atlantic Canada but potentially to any stock in any location. There is a vignette on Package and Workflow design and another that provides an example with simulated data.

Epidemiology

EpiInvert v0.1.1: Inverts a renewal equation to estimate time-varying reproduction numbers and restored incidence curves with festive days and weekly biases corrected as described in Alvarez et al. (2021) and Alvarez et al. (2022). See README for examples.

linelist v0.0.1: Provides tools to help storing and handling case line list data by adding a tagging system to classical data.frame objects to identify key epidemiological data. See the vignette for examples.

Finance

markets v1.0.3: Provides functions to estimate markets in equilibrium and disequilibrium based on full information maximum likelihood techniques given in Maddala and Nelson (1974) and implemented using the analytic derivative expressions calculated in Karapanagiotis (2020). There is an Overview providing theory and code and vignettes on Model initializion details, Market-clearing assessment, and Use cases.

portvine v1.01: Provides portfolio level risk estimates including value at Risk and Expected Shortfall following the approach described in Sommer (2022) by modeling each asset with an ARMA-GARCH model and then modeling their cross dependency via a Vine Copula in a rolling window fashion. See the vignette to get started.

usincometaxes v0.4.0: Implements a wrapper to the NBER’s TAXSIM 35 tax simulator. TAXSIM 35 to calculate federal and state income taxes. There are vignettes on Uploading Data, Input Columns, Output Columns, and Calculating Taxes.

Genomics

MixviR v3.3.5: Implements tools for exploring DNA and amino acid variation and inferring the presence of target lineages from microbial high-throughput genomic DNA samples that potentially contain mixtures of variants/lineages. MixviR was originally created to help analyze environmental SARS-CoV-2/Covid-19 samples from environmental sources such as waste water or dust, but can be applied to any microbial group. See DePristo et al. (2011) and Danecek et al. (2021) for background, and the vignette for examples.

simer v0.9.0.0: Implements a data simulator including genotype, phenotype, pedigree, selection and reproduction for animals and plants and provides data for genomic gelection, genome-wide association, and breeding studies. See Kao and Zeng (2002) and Ripley (1987) for background. Look here for extensive documentation.

Machine Learning

fastTopics v0.6-135: Implements fast, scalable optimization algorithms for fitting topic models and non-negative matrix factorization for count data. The methods exploit the special relationship between the multinomial topic model (probabilistic latent semantic indexing) and Poisson non-negative matrix factorization. See the vignettes: Relationship between NMF and topic modeling and Topic mideling vs. clustering.

metrica v1.2.3: Provides functions to evaluate prediction performance of point-forecast models accounting for different aspects of the agreement between predicted and observed values including error metrics, model efficiencies, indices of agreement, goodness of fit, concordance correlation, and error decomposition, and plots the visualized agreement. See the vignettes on Available Metrics and Model Assessment.

SparseVFC v0.1.0: Implements The sparse vector field consensus algorithm described in Ma et al. (2013) for robust vector field learning. See the vignette.

Networks

Rwclust v0.0.1: Implements the random walk clustering algorithm for weighted graphs as found in Harel and Koren (2001). See the vignette.

Science

EvoPhylo v0.1: Provides functions to support automated morphological character partitioning for phylogenetic analyses, and analyses of macroevolutionary parameter outputs. See Simões and Pierce (2021) for background. There is a vignette on Theoretical Background, and there are others on Character Partitioning, FBD parameters, and Evolutionary Rates.

sits v1.0.0: Provides an end-to-end toolkit for land use and land cover classification using big Earth observation data, based on machine learning methods applied to satellite image data cubes, as described in Simoes et al (2021). Builds regular data cubes from collections in AWS, Microsoft Planetary Computer, Brazil Data Cube, and Digital Earth Africa using the STAC protocol and the gdalcubes package. An eBook provides extensive documentation.

Statistics

biosensors.usc v1.0: Provides a framework for using distributional representations of biosensor data such as ECG, medical imaging or fMRI data in various statistical modeling tasks: regression models, hypothesis testing, cluster analysis, visualization, and descriptive analysis. See Matabuena et al. (2021) for background and the vignette for an introduction.

CopSens v0.1.0: Implements the copula-based sensitivity analysis method for observational causal inference discussed in Zheng et al. (2022). See README for examples.

GeoModels v1.0.1: Provides functions to analyze Gaussian and Non Gaussian (bivariate) spatial and spatio-temporal data and simulate random fields using likelihood methods. See Bevilacqua and Gaetan (2015) and Vallejos et al. (2020) for background, and look here for examples.

nlmixr2 v2.0.6: Fit and compare nonlinear mixed-effects models with flexible dosing information commonly seen in pharmacokinetics and pharmacodynamics using differential equations solved by compiled C code provided in the rxode2 package. See Almquist et al. (2015) and Wang et al. (2015) for background, and the vignette for examples.

stdmod v0..1.7.1: Provides functions for computing a standardized moderation effect in moderated regression and forming its confidence interval by nonparametric bootstrapping as proposed in Cheung et al. (2002). There are six vignettes including a Quick Start Guide and a vignette on Standardized Moderation.

Time Series

forceR v1.0.15: Initially written and optimized to deal with insect bite force measurements, the functions in this package can be used to clean and analyze any time series. They provide a workflow to load, plot and crop data, correct amplifier and baseline drifts, identify individual peak shapes, rescale (normalize) peak curves, and find best polynomial fits to describe and analyze force curve shapes. See the vignette for examples.

ZINARp v0.1.0: Provides functions for simulation, exploratory data analysis and Bayesian analysis of p-order integer-valued autoregressive, INAR(p), and zero-inflated p-order integer-valued autoregressive, ZINAR(p), processes, as described in Garay et al. (2020).

Utilities

async v0.2.1: Provides functions for writing sequential-looking code that pauses and resumes similarly to generator and async constructs from Python or JavaScript. Objects produced are compatible with the iterators and promises packages. See the vignette for an example.

chronicler v0.2.0: Provides tools to decorate a function so that it returns its along with a log detailing when the function was run, what were its inputs, what were the errors (if the function failed to run) and other useful information. See the vignettes: A non-mathematician’s introduction to monads, The Maybe monad, and A real world example.

jpgrid v0.2.0: Provides functions to generate Japanese grid square codes from longitude, latitude and geometries. See README for examples.

partialised v0.1.0: Provides a partialised class that extends the partialising function of purrr by making it easier to change the arguments. This is similar to the function-like object in `Julia’. See README for an example.

shinybrowser v1.0.0: Provides information about shiny app users including browser name and version, device type (mobile or desktop), operating system and version, and browser dimensions. See README for more information.

webshot2 v0.1.0: Takes screenshots of web pages, including Shiny applications and R Markdown documents using a headless Chrome or Chromium browser as the browser back-end. See README for examples.

Visualization

accrualPlotv1.0.1: Implements accrual plots for clinical trials. See the vignette.

ggbraid v0.2.2: Implements stat_braid(), that extends the functionality of geom_ribbon() to correctly fill the area between two alternating lines (or steps) with two different colors, and geom_braid(). Three vignettes, US Supreme Court, NBA Finals Game, and Average Daily Temperatures provide examples.

ggisotonic v0.1.2: Provides stat_isotonic() to add weighted univariate isotonic regression curves. See README for an example.

UpSetVP v1.0.0: Uses the ideas of variance partitioning and hierarchical partitioning as described in Lai et al. (2022) to visualize the unique, common, or individual contribution of each predictor (or matrix of predictors) towards explaining variation. Look here for examples.

April: "Top 40" New CRAN Packages

Mon, 30 May 2022 00:00:00 +0000

Computational Methods

hmer v1.0.1: Provides objects and functions for Bayes Linear emulation and history matching, including functions for automated training of emulators, diagnostic functions to ensure suitability, and a variety of methods for generating waves of points. There is a vignette Demo of the package, an Emulation and History matching Handbook, and vignettes on Examples, and Stochastic Emulation.

rminqa v0.1.1: Implements a wrapper for the C++ function bobyqa to perform derivative-free optimization algorithms in R.

rolog v0.9.4: Embeds SWI-Prolog, so that R can send deterministic and non-deterministic queries to Prolog. See README for examples.

torchopt v0..1.1: Implements optimizers for the torch deep learning that are not among the optimizers offered in torch. These include:adabelief by Zhuang et al (2020), adabound by Luo et al.(2019), adamw by Loshchilov & Hutter (2019), madgrad by Defazio and Jelassi (2021), nadam by Dozat (2019), qhadam by Ma and Yarats (2019), radam by Liu et al. (2019), swats by Shekar and Sochee (2018), and yogi by Zaheer et al.(2019)

villager v1.1.1: Provides a set of base classes with core functionality to allow users to create and run Agent Based Models. There are vignettes on Extending Agents and Extending Resources.

Data

baseballr v1.2.0: Provides numerous utilities for acquiring and analyzing baseball data from online sources such as Baseball Reference, FanGraphs, and the MLB Stats API. See the Getting Started Guide and the vignettes NCAA Scraping and Plotting Statcast data.

toRvik v1.0.2: Provides a suite of functions to quickly scrape and tidy advanced metrics, detailed player and game statistics, team and coach histories, and more from Barttorvik. See the vignette.

ustfd v0.1.0: Lets users make requests from the US Treasury Fiscal Data API endpoints. See README for an example.

valet v0.9.0: Implements a client for the recently updated Bank of Canada Valet API. See README for examples.

Ecology

OBIC v2.0.1: Provides functions to calculate the Open Boden Index method used in the Netherlands to evaluate the quality of soils of agricultural fields and evaluate the sustainability of the current agricultural practices. There are vignettes on the Open soil index, Score aggregation, and Workability.

timbeR v2.0.1: Provides functions to estimate wood volumes, for example, number of logs, diameters along the stem and heights at which certain diameters occur. See Weiskittel, A. (2021) for background and the vignette for an introduction.

Finance

AssetAllocation v1.0.0: Provides functions to implement customizable asset allocation strategies and automatically download data from Yahoo Finance. See the vignette.

multilateral v1.0.0: Implements multilateral price index calculations focused on time product dummy regression and GEKS variations, and allows for extension of the methods through automatic window splicing. See Krsinich (2016) for information on window splicing and the vignette for examples.

Machine Learning

clusterHD Provides tools for clustering high dimensional data as described in Raymaekers and Zamar (2020) and Raymaekers and Zamar (2020).

tglkmeans v0.3.4: Efficiently implements the Kmeans algorithm. See Arthur and Vassilvitskii (2007) and Ostrovsky et al. (2013) for background and the vignette to get started.

Mathematics

hilbert v0.2.1: Provides utilities for encoding and decoding coordinates to and from Hilbert curves based on the iterative encoding implementation described in Chen et al. (2006). See the vignette to get started.

gellipsoid v0.7.2: provides functions to represent degenerate and unbounded generalized geometric ellipsoids together with methods for linear and duality transformations, and for plotting. The ideas are described in Friendly, Monette & Fox (2013). See README for examples.

Medicine

crossnma v1.0.1: Provides functions for cross-design and cross-format Network Meta-Analysis Regression as described in Hamza et al. 2022. See the vignette.

exact.n v1.0.0: Allows the user to determine minimum sample sizes that achieve target size and power at a specified alternative. See Lloyd & Ripamonti (2021) for the theory and README for examples.

stoppingrules v0.1.1: Provides functions for creating, displaying, and evaluating stopping rules for safety monitoring in clinical studies including stopping rule methods described in Goldman (1987), Geller et al. (2003), Ivanova, Qaqish, & Schell (2005), and Kulldorff et al. (2011). See README for an example.

Networks

incidentally v0.9.0: Provides functions to generate random incidence matrices and bipartite graphs under different constraints or using different generative models. See the vignette.

networkscaleup v0.1-1: Provides a variety of network scale-up models to analyze aggregated relational data, including models from Laga et al. (2021) Zheng et al. (2006), Killworth et al. (1998), and Killworth et al. (1998). See the vignette.

pald v0.0.1: Implements the partitioned local depths algorithm described in Berenhaut, Moore, & Melvin (2022) which may be helpful in determining both local and global structure in data. Look here to get started.

Statistics

bmstdr v0.1.4: Provides functions to fit, validate, and compares a number of Bayesian models for spatial and space-time point referenced and areal unit data. See the vignette for theory and examples.

cubble v0.1.0: Implements a spatiotemperal data object in a relational data structure to separate the recording of time variant and invariant variables. See the vignettes: aggregation, design, cubble, import, and matching.

incubate v1.1.8: Fits parametric models to time-to-event data that show an initial incubation period, i.e., a variable phase where the hazard is zero. The delayed Weibull distribution serves as the foundational data model. Look here for an example.

pspatreg v1.0.2: Provides functions to estimate and analyze spatial and spatio-temporal semiparametric models including spatial or spatio-temporal non-parametric trends, parametric and non-parametric covariates with a possible spatial lag for the dependent variable, and temporal correlation in the noise. See Basile et al. (2014), Rodriguez-Alvarez et al. (2015), and especially Minguez et al. (2020) for background. There is an Introduction, and vignettes on Cross-sectional data and Spatial panel data.

sfdep v0.1.0: Provides and interface to spdep to integrate sf objects and the tidyverse. There are vignettes on The Basics, Conditional Permutations, and spdep and pysal.

smile v1.0.4.1: Provides functions to estimate, predict, and interpolate areal data. For estimation and prediction, areal data are assumed to be an average of an underlying continuous spatial process as in Moraga et al. (2017), Johnson et al. (2020), and Wilson and Wakefield (2020). There are vignettes on Fitting Models, Areal Interpolation, Converting to spm, Spatial Covariance, and Method.

SpatialPOP v0.1.0: Provides functions to generate a spatial population from a spatially varying regression model under the assumption that observations are collected from a uniform two-dimensional grid with unit distance between any two neighboring points. See Chao et al. (2018) for method details and the vignette for an example.

Time Series

lite v1.0.0: Performs likelihood-based inference for stationary time series extremes following the approach of Fawcett and Walshaw (2012). Marginal extreme value inferences are adjusted for cluster dependence using the methodology of Chandler and Bate (2007). See the vignette.

spooky v1.1.0: Uses the Discrete Fast Fourier Transformation to extrapolate time features beyond their boundaries. Look here for examples and references.

Utilities

chromote v0.1.0: Implements the Chrome DevTools Protocol for controlling a headless Chrome web browser. Look here for examples.

chkptstanr v0.1.1: Implements a framework to checkpoint Bayesian models fit with Stan and brms. The MCMC sampler can be stopped and then restarted where it left off. There is a vignette for brms and another for Stan.

ivs v0.1.0: Implements a new interval vector class for generic interval manipulations including locating various kinds of relationships between two interval vectors, merging overlaps within a single interval vector, splitting an interval vector on its overlapping endpoints, and applying set theoretical operations on interval vectors. The package was inspired by Allen (1983). See the vignette for an introduction.

listr v0.0.2: Pools for common operations on lists such as selecting and merging data stored in lists which can be used with pipes. See the vignette.

shinyGizmo v0.1: Provides UI components and input widgets for Shiny applications to apply non-standard operations and address performance issues. See README for examples.

shinytest2 v0.1.0: Provides automated unit testing of Shiny applications through a headless Chromium browser. There is a Getting Started Guide and seven additional vignettes including Testing in depth, Robust testing, and Monkey testing.

thaipdf v0.1.2: Provides R Markdown templates and aLaTeX preamble to create PDFs from R Markdown documents in the Thai language. See the vignette for examples.

Visualization

ggtrendline v1.0.3: Enhances ggplot2 with tools to add a trendline with a confidence interval for linear or nonlinear regression models and show the equation. See Ritz and Streibig (2008) and Greenwell and Schubert Kabban (2014) for background and look here for examples.

March: "Top 40" New CRAN Packages

Thu, 28 Apr 2022 00:00:00 +0000

Two hundred and six new packages stuck to CRAN in March. Here are my “Top 40” selections in thirteen categories: Computational Methods, Data, Finance, Game Theory, Genomics, Machine Learning, Medicine, Networks, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

RCDT v1.1.0: Provides functions to perform 2D Delaunay triangulation, constrained or unconstrained, with the help of the CDT C++ library. See the vignette.

rlemon v0.2.0: Provides access to the LEMON C++ graph library. Look here for a list of algorithms.

Data

ag5Tools v0.0.1: Offers tools for downloading and extracting data from the Copernicus Agrometeorological indicators from 1979 to present derived from reanalysis (AgERAS) dataset. See ag5Tools to get started and the vignette on extracting data.

AirMonitor v0.2.2: Provides utilities for working with hourly air quality monitoring data with a focus on small particulates (PM2.5) along with algorithms to calculate NowCast and the associated Air Quality Index (AQI) as defined by the US Environmental Projection Agency AirNow program. There is an Introduction, a Developers Style Guide, and a Data Model.

BISdata v0.1-1: Provides access to data from the Bank for International Settlements in Basel. Look here for an example.

Finance

AFR v0.1.0: Provides tools for regression, prediction and forecast analysis of macroeconomic and credit data adapted for banking sector of Kazakhstan for bank analysts and non-statisticians. There are vignettes on Data transformatiom, Diagnostic Tests, and Regression.

fixedincome v0.0.1: Implements objects that abstract interest rates, compounding factors, day count rules, forward rates and term structure of interest rates to assist with calculations of interest rates and fixed income. Look here for examples.

Game Theory

socialranking v0.1.1: Offers a set of solutions to rank players based on a transitive ranking between coalitions, including CP-Majority, ordinal Banzhaf or lexicographic excellence solution summarized Allouche et al. (2020). See the vignette.

Genomics

AquaBPsim v0.0.1: Provides tools to simulate breeding programs including functions to simulate production and reproduction systems encountered in aquaculture. See the vignette for details.

coda4microbiome v0.1.1: Provides tools for microbiome data analysis that take into account its compositional nature including functions for variable selection for both, cross-sectional and longitudinal studies, and for binary and continuous outcomes. See the vignette

MitoHEAR v0.1.0: Provides functions for the estimation and downstream statistical analysis of the mitochondrial DNA Heteroplasmy calculated from single-cell datasets. See the vignette.

ZetaSuite v1.0.0: Provides functions to score hits from two-dimensional RNAi screens analyze single cell transcriptomics to differentiate rare cells from damaged ones. See Vento-Tormo et al. (2018) for background and the vignette for examples.

Machine Learning

RGAN v0.1.1: Implements the Generative Adversarial Nets algorithm described in Goodfellow et al. (2014). See README for examples.

sentiment.ai v0.1.1: Implements Sentiment Analysis via tensorflow deep learning and gradient boosting models and also allows users to create embedding vectors for text which can be used in other analyses. See the vignette for an example.

sentopics v0.6.2: Offers a framework that joins topic modeling and sentiment analysis of textual data, and implements a fast Gibbs sampling estimation of Latent Dirichlet Allocation. See Griffiths & Steyvers (2004) and the Joint Sentiment/Topic Model of Lin, Everson & Ruger (2012). There is a vignette on Bascic Usage and another on Topical time series.

transforEmotion v0.1.0: Provides access to sentiment analysis using the Python based huggingface transformer zero-shot classification model pipelines. The default pipeline is Cross-Encoder’s DistilRoBERTa trained on the Stanford Natural Language Inference and Multi-Genre Natural Language Inference datasets. There is a vignette on setting up Python.

Medicine

adaptr v1.0.0: Simulates adaptive clinical trials using adaptive stopping, adaptive arm dropping, and adaptive randomization. There is an Overview and vignettes on Basic and Advanced examples.

EVI v0.1.1-4: Implements the epidemic volatility index, a novel early warning tool for identifying new waves in an epidemic as described in Kostoulas et al. (2021). See the vignette.

rts2 v0.3: Provides functions to support modelling case data for real-time surveillance of infectious diseases including functions to generate a computational grid over an area of interest and approximate log-Gaussian Cox Process model. See Diggle et al. (2013) and Solin and Särkkä (2020) for background and look here for examples.

Networks

BCDAG v1.0.0: Provides functions for structure learning of causal networks and estimation of joint causal effects from observational Gaussian data. See Castelletti & Mascaro (2021) and Castelletti & Mascaro (2022) for background, and the vignettes Random Data Generation, Output of learn_DAG(), and MCMC scheme for posterior inference for details.

SEset v1.0.1: Implements tools to compute and analyze the set of statistically-equivalent Gaussian, linear path models which generate the input precision or (partial) correlation matrix. See README for examples.

Science

bdc v1.1.0: Provides functions for biodiversity data cleaning organized into five themes: Merging datasets, Pre-filtering, Taxonomy, Space (Flagging low precision coordinates), and Time (flagging inconsistent data collection dates). There are vignettes on Standardization, Pre-filter, Space, Taxonomy, and Time.

CSHShydRology v1.2.1: Offers a collection of user submitted functions to aid in the analysis of hydrological data. See the vignette.

Statistics

cheem v0.2.0: Provides functions to explore local explanations of non-linear models by first calculating the tree SHapley Additive exPlanation for every observation and for calculating a projection basis, and then changing the basis with a radial tour. See Lundberg et al. (2019), Spyrison & Cook (2020) and Cook and Buja (2012) for background and the vignette to get started.

CondCopulas v0.1.2: Provides functions for the estimation of conditional copulas models, various estimators of conditional Kendall’s tau statistic as proposed in Derumigny and Fermanian 2019a, 2019b and 2020. See the vignette for examples.

multilevelmod v0.1.0: Implements bindings for hierarchical regression models for use with the parsnip package. Models include longitudinal generalized linear models as described in Liang and Zeger (1986) and mixed-effect models as described in Pinheiro and Bates (2000). See the vignette for examples.

rbmi v1.1.3: Implements reference based multiple imputation allowing for the imputation of longitudinal datasets using predefined strategies. These include conventional MI methods, conditional mean imputation methods, and bootstrapped MI methods. See the Scope section of the Statistical Specifications vignette for more information on MI methods. There is a Quick Start Guide and an additional vignette on Advanced Functionality.

remiod v1.0.0: Implements reference-based multiple imputation of ordinal and binary responses under Bayesian framework, as described in Wang and Liu (2022). See the vignette.

rlcv v1.0.0: Provides functions to estimate likelihood cross-validation bandwidth for uni- and multi-variate kernel densities which are robust with respect to fat-tailed distributions and outliers. See Wu (2019) for the theory and the vignette for an example.

sftime v0.2-0: Provides classes and methods for spatial objects that have a registered time column, in particular for irregular spatiotemporal data. See the vignette.

workboots v0.1.1: Provides functions for generating bootstrap prediction intervals from a tidymodels workflow. There is a Getting Started Guide and a vignette on estimating linear prediction intervals.

Time Series

dtts v0.1.0: Provides high-frequency time-series support via nanotime and data.table. See README for examples.

ngboostForecast v0.0.2: Implements probabilistic time series forecasting via natural gradient boosting for probabilistic prediction. See README for an example.

svines v0.1.4: Provides functions to fit and simulate from stationary vine copula models for time series. See Nagler et al. (2022) for the theory and look here for examples.

Utilities

dyn.log v0.4.0: Implements dynamic, configuration driven logging. There are vignettes on Configuration, Formats, Layouts, and Levels.

formatters v0.2.0: Provides a framework for rendering complex tables to ASCII, and a set of formatters for transforming values or sets of values into ASCII-ready display strings. See the vignette for examples.

git4r v0.1.2: Implements an interactive git user interface from the R command line that includes tools to make commits, branches, remotes, and diffs an integrated part of R coding. See the vignette.

Visualization

ggmice v0.0.1: Provides functions to enhance a mice imputation workflow with visualizations for incomplete and imputed data including functions to inspect missing data, develop imputation models, evaluate algorithmic convergence, and compare observed versus imputed data. See the Getting Started Guide and the vignette Old Firends.

langevitour v0.2: Implements an HTML widget that uses Langevin dynamics to show random walks through 2D projections of numerical data. It can be used from within R, or included in a self-contained Rmarkdown document. See the vignette for an examples.

picker v0.2.6: Provides functions to zoom, pan, and pick points from a deck.gl scatterplot and includes tooltips, labels, a grid overlay, legends, and coupled interactions across multiple plots. See README for examples.

February 2022: "Top 40" New CRAN Packages

Mon, 28 Mar 2022 00:00:00 +0000

February was a good month for new R packages on CRAN. Here are my “Top 40” selections from the two hundred packages that arrived in thirteen categories: Computational Methods, “Data”, Genomics, Linguistics, Machine Learning, Mathematics, Medicine, Networks, Science, Sports, Statistics, Utilities, Visualization.

Computational Methods

fastadi v0.1.0: Implements the adaptive-impute matrix completion algorithm described in Cho, Kim & Rohe (2016). See README for details.

invertiforms v0.1.0: Provides composable invertible transforms for sparse matrices. See README for examples.

mirai v0.1.1: Provides a simple and lightweight method for concurrent and parallel code execution, built on NNG, Nanomsg Next Gen technology. See README for examples.

Data

amerifluxr v1.0.0: Provides a programmatic interface to the AmeriFlux database and includes query, download, and data summary tools. There are vignettes on Data Import and Site Selection.

finnishgrid v0.1.0: Implements an API client for Fingrid Open Data on the electricity market and the power system. See the vignette for an introduction.

fixtuRes v0.1.3: Provides functions to generate mock data in R using YAML configurations. See the vignette.

jpstat v0.2.0: Provides tools for using the e-Stat API, a portal site for Japanese government statistics, and includes functions for automatic query generation, data collection and formatting. See README for examples.

Gemomics

fcfdr v1.0.0: Provides functions to implement the Flexible cFDR (Hutchinson et al. (2021)) and Binary cFDR (Hutchinson et al. (2021)) methodologies to leverage auxiliary data from arbitrary distributions, for example functional genomic data, with GWAS p-values to generate re-weighted p-values. There is an Introducion and there are vignettes on LDAK and TID

simphony v1.0.0: Provides a tool for simulating rhythmic data: transcriptome data using Gaussian or negative binomial distributions, and behavioral activity data using Bernoulli or Poisson distributions. See Singer et al. (2019) for details, and the vignettes on simphony’s options and evaluating rhythm detection for examples.

Linguistics

glottospace v0.0.111: Provides streamlined workflows for geolinguistic analysis, including: accessing global linguistic and cultural databases, data import, data entry, data cleaning, data exploration, mapping, visualization and export. See README to get started.

Machine Learning

aum v2022.2.7: Uses a standard template library sort to implement an efficient algorithm for computing AUM, Area Under Min(FP, FN), and directional derivatives. See Hillman & Hocking (2021) for details and the vignettes Accuracy comparison and Speed comparison.

familiar v1.0.0: Provides an unified interface for end-to-end automated machine learning and model evaluation. There is an Introduction, and five additional vignettes including Evaluation and explanation and Feature selection methods.

OptHoldoutSize v0.1.0.0: Provides tools to estimate the size of a holdout set and the associated errors when updating predictive scores. See Haidar-Wehbe et al. (2022) for details and the vignettes Comparison of algorithms, ASPRE Example, and Simulated Example.

soundClass v0.0.9.1: Implements a sound classification workflow with functions to automatically classify sound events using convolutional neural networks. See Gibb et al. (2019), Mac Aodha et al. (2018), and Stowell et al. (2019) for background and the vignette for an example.

Mathematics

gmpoly v1.1.0: Provides symbolic calculation, addition, multiplication, and evaluation of multivariate polynomials with rational coefficients. See README for examples.

gyro v0.2.0: Implements functions for three dimensional hyperbolic geometry based on the theory found in Ungar (2005). The short vignette points to resources to get you started.

sumR v0.4.6: Implements functions based on theoretical results which ensure that the summation of an infinite discrete series is within an arbitrary margin of error of its true value. See Braden (1992) for background.

Medicine

admiral v0.6.3: Implements a toolbox for programming CDISC compliant Analysis Data Model ADaM datasets in R in accordance with the Analysis Data Model Implementation Guide. There are seven vignettes including Creating ADSL and Creating a BDS Exposure ADaM.

baker v1.0.0: Provides functions to specify, fit and visualize nested partially-latent class models for inference of population disease etiology and individual diagnosis. See Wu et al. (2015), Wu et al. (201). and Wu & Chen (2020) for background and the vignette for examples.

MicroMB v0.0.12: Implements a framework based on S3 dispatch for constructing models of mosquito-borne pathogen transmission which are constructed from submodels of various components. Consistent mathematical expressions for the distribution of bites on hosts enables stochastic and deterministic models to be coherently incorporated and updated over a discrete time step. There are nine vignettes including the Ross-Macdonale mosquito model, transmission model, and Blood feeding.

musclesyneRgies v1.1.3: Provides a framework to factorize electromyography data including tools for raw data pre-processing, non negative matrix factorization, classification of factorised data and plotting of obtained outcomes. There are vignettes on analysis, plots, pro tips, and workflow.

ravetools v0.0.3: Implements signal processing tools for analyzing electrophysiology data including a fast, memory-efficient Notch-filter, Welch-periodogram, and a discrete wavelet transform algorithm for hours of high-resolution signals. See the RAVE Project and Magnotti et al. (2020) for background and README for examples.

wildmeta v0.1.0: Implements single coefficient tests and multiple-contrast hypothesis tests of meta-regression models using cluster wild bootstrapping, based on the methods examined in Joshi et al. (2021). The vignette provides examples.

Networks

leidenbase v0.1.9: Implements an R to C/C++ interface that runs the Leiden community detection algorithm to find a basic partition. It includes the required source code files from the official leidenalg distribution and functions from the R igraph package. See the vignette for an example.

netropy v0.1.0: Provides functions to conduct a statistical entropy analysis of network data as introduced by Frank & Shafie (2016). There are vignettes on Joint Entropies, Prediction Power, Uni, Bi & Trivariate Entropies, and Data Editing.

Science

IceSat2R v1.0.1: Implements an interface to to the OpenAltimetry ICESat data through the API allowing users to download and process Global Geolocated Photon Data, Land Ice Height, Sea Ice Height, Land and Vegetation Height and more. There are vignettes on IceSat-2 Atlases, Mission Orbits, and Virtual File System Orbits.

panstarrs v0.1.0: Implements an interface to the API for Pan-STARRS1, a data archive of the PS1 wide-field astronomical survey which allows access to the PS1 catalog and to the PS1 images. See the vignettes Cat and Images.

ThermalSampleR v0.1.0: Implements a range of simulations to aid researchers in determining appropriate sample sizes when performing critical thermal limits studies including a number of wrapper functions are provided for plotting and summarizing outputs from these simulations. There is both a vignette and a Shiny App.

Sports

footBayes v0.1.0: Provides functions to estimate, visualize, and predict the most well-known football models: double Poisson, bivariate Poisson, Skellam, student_t. The package allows Hamiltonian Monte Carlo (HMC) estimation through the underlying Stan environment and Maximum Likelihood estimation (MLE, for ‘static’ models only). See Dixon & Coles (1997) Karlis & Ntzoufras (2003), and Pauli & Torelli (2018) for background and the vignette for and introduction to the package.

Statistics

fido v1.0.0: Provides methods for fitting and inspecting Bayesian Multinomial Logistic Normal Models using MAP estimation and Laplace approximation as developed in Silverman et. Al. (2022). There is an Introduction and vignettes on PCR Bias, Non-linear models, Joint Modeling, and Picking priors.

geostan v0.2.1: Provides Stan-based tools for Bayesian inference with spatial data, including exploratory analysis tools, multiple spatial model specifications, spatial model diagnostics, and special methods for inference with small area survey data. See Donegan et al. (2020) for background and the vignettes on Spatial autocorrolation and Survey data for examples.

Landmarking v1.0.0: Provides functions to perform Landmark survival analyses which allow survival predictions to be updated dynamically as new measurements from an individual are recorded. There is an Introduction and a vignette on how to use the package.

PUMP v1.0.0: Provides functions to estimate power, minimum detectable effect size and sample size requirements in the context of multilevel randomized experiments with multiple outcomes. See Hunter et al. (2021) for the details. There is a Package Demo vignette and vignettes on the Sampling method, and Simulating multi-level data.

safestats v0.8.6: Provides functions to design and apply tests which are anytime valid, can be used to design hypothesis tests in the prospective/randomized control trial setting or in the observational/retrospective setting, and remain valid under both optional stopping and optional continuation. For details on the theory of safe tests, see Grunwald et al. (2019). There is a vignette on Safestats and another on Contingency tables.

Utilities

audubon v0.1.1: Provides a collection of Japanese text processing tools for filling Japanese iteration marks, Japanese character type conversions, segmentation by phrase, and text normalization which is based on rules for the Sudachi morphological analyzer and the NEologd (Neologism dictionary for MeCab). See README to get started.

rconfig v0.1.1: Allows users to manage R configuration files and override configuration statements from the command line. Look here for details.

Visualization

ggchangepoing v0.1.0: R provides tools for changepoint analysis and uses ggplot2 to visualize changepoints. See the vignette.

ggdensity v0.0.1: Provides functions for visualizing contours of 2-d kernel density estimates and implements several additional density estimators as well as more interpretable visualizations based on highest density regions instead of the traditional height of the estimated density surface. Look here for examples.

gghdr v0.1.0: Provides a framework for visualizing Highest Density Regions in ggplot2. See the vignette.

ggpattern v0.4.2: Provides geoms filled with various patterns including patterned versions of every ggplot2 geom that has a region that can be filled along with a suite of aesthetics and scales for controlling pattern appearances. There four vignettes: Developing Patterns and three more that cover gradients, polygons, and crosshatching.

January 2022: "Top 40" New CRAN Packages

Mon, 28 Feb 2022 00:00:00 +0000

Two Hundred and two new packages made it to CRAN in January. Several were in applications areas that don’t show up very often, and a few of these made it into my January “Top 40” picks. My selections are listed below in thirteen categories: Agriculture, Computational Methods, Data, Engineering, Finance, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities, and Visualization. Slightly expanding the number of categories helps to emphasize the various application areas, but makes it more difficult to classify packages which could fit in multiple categories. I hope this does not inconvenience readers.

Agriculture

ALUES v0.2.0: Provides functions for fuzzy modeling to evaluate land suitability for different crops production according to the methodology established by the Food and Agriculture Organization and the International Rice Research Institut. There is a Getting Started Guide along with seven vignettes including: Methodology and Visualizing with maps.

Computational Methods

CGNM v0.1.1: Implements the cluster Gauss-Newton method to find multiple solutions to nonlinear least squares problems. See Aoki et al. (2020) for background and the vignette to get started.

simpr v0.2.2: Implements a tidyverse friendly framework for simulation studies, design analysis, and power analysis. It enables users to generate data, fit models, and tidy up model results in a single pipeline. There is an Introduction and there are short vignettes on Optimization, Reproducing simulations, and Managing simulation errors.

Data

chessR v1.5.0: Provides functions to enable users to extract chess game data from popular chess sites, including Lichess and Chess.com and then perform analyses on game data. See the vignette.

dictionaRy v0.1.1: Implements an interface to the Free Dictionary API which allows users to retrieve dictionary definitions for English words, as well as additional information including phonetics, part of speech, origins, audio pronunciation, example usage, synonyms and antonyms which are returned in tidy format for ease of use.

flightsbr: v0.1.0: Provides functions to download flight and airport data from Brazil’s Civil Aviation Agency (ANAC) that includes detailed information on all aircraft, aerodromes, airports, and airports movements registered in ANAC, and on every international flight to and from Brazil, as well as domestic flights within the country. There is an Introduction and there are vignettes on Airport data and Flights data.

rGhanaCensus v0.1.0: Contains literacy and education data sets scrapped from the 2021 Ghana Population and Housing Census See the vignette.

Engineering

TesiproV v0.9.1: Provides functions to calculate the failure probability of civil engineering problems with Level I up to Level III Methods. See AU & BECK (2001) for background and the vignette for examples.

Finance

PDtoolkit v0.1.0: Provides functions for developing probability of default rating models including functions for imputations, binning of numeric and categorical risk factors, weights of evidence, information value calculations, and risk factor clustering as well as validation functions for testing homogeneity, heterogeneity, discriminatory and predictive power of the model. See README for examples.

ufRisk v1.0.2: Provides functions to calculate Value at Risk (VaR) and Expected Shortfall (ES) by means of various parametric and semiparametric GARCH-type models. The approaches implemented in this package are described in Feng et al. (2020) and Letmathe et al. (2021). See README to get started.

Genomics

aphylo v0.2-1: Implements a parsimonious evolutionary model to analyze and predict gene-functional annotations in phylogenetic trees as described in Vega Yon et al. (2021). See the vignette for an example.

edlibR v1.0.0: Implements an interface to the Edlib C/C++library for exact sequence alignment using Levenshtein distance. See the vignette to get started.

freqpcr v0.4.0: Implements functions for the interval estimation of the population allele frequency from qPCR analysis based on the restriction enzyme digestion (RED)-DeltaDeltaCq method as described in Osakabe et al. (2017). The vignette is in English and Japanese.

Machine Learning

longmixr v1.0.0: Adapts the consensus clustering approach from ConsensusClusterPlus for longitudinal data using flexmixflexible mixture models. See the vignette.

reclin2 v0.1.1: Provides functions to assist performing probabilistic record linkage and deduplication: generating pairs, comparing records, em-algorithm for estimating m- and u-probabilities. See Fellegi & Sunter (1969) for background. There is an Introduction and there are vignettes on Deduplication, Record lingage, and Using a cluster for parallel computing.

rego v1.3.4: Implements a machine learning algorithm for predicting and imputing time series along with a Bayesian stochastic search methodology for model selection. Written in C++, the authors claim that the package is suitable for problems with hundreds or thousands of dependent variables and problems in which the number of dependent variables is greater than the number of observations. Look here for documentation.

Mathematics

tesselation v1.0.0: Computes Delaunay and Voronoï tessellations and provides functions to plot the 2 and 3 dimensional tessellations. Delaunay tessellations are computed in C with the help of the Qhull. There is a vignette on GitHub.

weyl v0.0-1: Provides functions for working with Weyl Algebras. See the vignette.

Medicine

biodosetools v3.6.0: Implements a Shiny application to perform the various statistical tests and calculations needed by Biological Dosimetry Laboratories. There ar vignettes on Dicentrics dose estimation and dose-effect fitting and on Translocations dose estimation and dese-effect fitting.

rccola v1.0.2: Provides secure convenience functions for entering and handling API keys and pulling data directly into memory. By default, it will load from REDCap instances, but other sources are injectable via inversion of control. See README for documentation.

Science

datelife v0.6.1: Implements the functions underlying the DateLife web service to allow researchers and the general audience to obtain open scientific data on ages for their organisms of interest. Age data are extracted from dated phylogenetic trees (chronograms) that have been published and peer-reviewed in association with a scientific article in an indexed journal. There is a Getting Started Guide and a Case Study.

dynamAdes v2.0.2: Implements a model to study the population dynamics of invasive Aedes mosquitoes. See Da Re et al. (2021) for the model rationale, and Da Re et al. (2021) for the model framework. The vignette provides an example.

Statistics

autoReg v0.1.0: Provides functions to create summary tables for descriptive statistics and select explanatory variables automatically for various regression models including linear models, generalized linear models and cox-proportional hazard models. There is a Getting Started Guide and there are vignettes on Automatic Regression Modeling, Bootstrap Simulation, Statistical tests, and Survival Analysis.

conformalInference.multi v1.0.0: Provides functions to compute full conformal, split conformal and multi split conformal prediction regions when the response variable is multivariate (i.e. dimension is greater than one). For background see Lei et al. (2016), Diquigiovanni et al. (2021), and Solari & Djordjilovic (2021). Look here for examples.

gamselBayes v1.0-2: Provides functions to fit and select generalized additive models via approximate Bayesian inference according to the methodology described in He & Wand (2021). There is a vignette.

interpretCI v0.1.1: Provides functions for estimating confidence intervals for various statistics and plotting the results. There is an Introduction along with ten additional vignettes for the various statistics including CI for difference between proportions and Hypotheses test for difference between paired means.

lmls v0.1.0: Implements functions for Gaussian location-scale regression model (a multi-predictor model with explanatory variables for the mean (location) and the standard deviation (scale) of a response variable) using the algorithms described in Girolami & Calderhead (2011) and Nesterov (2009). See the vignette.

sandwichr v1.0.1: Implements the Spatial Stratified Heterogeneity (SSH) spatial interpolation algorithms described in Wang et al. (2013). See the vignette.

Utilities

gittargets v0.0.3: Provides functions to preserve historical output in targets workflows by capturing version-controlled snapshots of the data store. Each snapshot links to the underlying commit of the source code enabling users to recover contemporaneous data when rolling back to a previous commit. See the Tutorial.

httptest2 v0.1.0: For packages using httr2 this package enables testing all of the logic on the R side of the API without requiring access to the remote service. It also allows recording real API responses to use as test fixtures. There is an Introduction and there are vignettes on Modifying Recorded Requests, Writing Vignettes with API’s, and FAQs.

maybe v0.2.0: Implements a maybe type which represents the possibility of some value or nothing. This may be used instead of throwing an error or returning NULL. maybe and has the advantages of being composable and requiring the developer to explicitly acknowledge the potential absence of a value. Look here to get started.

nanonext v0.2.0: Implements an R binding for NNG (Nanomsg Next Gen), a socket library for implementing a high-performance cross-platform protocol standard for messaging and communications. It serves as a concurrency framework that can be used for building distributed applications. See README to get started.

powerjoin v0.0.1: Provides extensions of dplyr and fuzzyjoin join functions to preprocess data, apply various data checks, and deal with conflicting columns. See README for examples.

quickcheck v0.1.0: Builds on the framework provided by hedgehogto implement property based testing in R. It was inspired by QuickCheck and has been designed to seamlessly integrate with testthat. See README for an introduction.

Visualization

fisheye v0.1.0: Provides functions to transform base maps to focus on a specific location using an azimuthal logarithmic distance transformation. See README to get started.

forestplotter v0.1.1: Provides functions to create multi-column forest plots with confidence intervals that may be grouped the data. See the vignette.

geomtextpath v0.1.0: Implements a ggplot2 extension which allows text to follow curved paths. Curved text makes it easier to directly label paths or neatly annotate in polar co-ordinates. There is an Introduction and there are vignettes on Curved Text in Polar Plots and Aesthetics.

ggESDA v0.1.0: Implements an extension of ggplot2 to visualize symbolic data and also provides a function to transform classical data to symbolic data by both clustering algorithm and customized methods.

toastui v0.2.1: Implements an interface to the TOAST UI libraries for creating interactive tables and plots that can be integrated into Shiny applications. There is a vignette and a webpage with interactive examples.

tornado v0.1.1: Implements linear models, generalized linear models, survival regression models, and machine learning models using the caret package framework and draws tornado plots to visualize the range of outputs expected from a variety of inputs, or alternatively, the sensitivity of the output to the range of inputs. See the vignette.

CRAN "Golden Oldies" and "One Hit Wonders"

Wed, 16 Feb 2022 00:00:00 +0000

If you are a regular R Views reader, then you know that every month I post my Top 40 picks for new CRAN packages. In this post, I’ll borrow some additional terminology from the Top 40 AM radio of my youth, and talk about two new categories of CRAN packages: Golden Oldies and One Hit Wonders.

Golden Oldies

On Top 40 radio, Golden Oldies are tunes that continue to get a lot of air play over decades, and are generally thought to be classics of 50’s, 60’s, and 70’s pop culture. By a Golden Oldie R package I mean an R package that has been around for a long time, has gone through several version upgrades of bug fixes and enhancements, and is thought to be an indispensable package for some part of the R Community.

For example, the nlme and lme4 R packages for fitting mixed models would surely be on the Golden Oldie list for any biostatistician who works in R. nlme has been around since 1999 and has had 139 version bumps, while lme4 was introduced in 2003 and has had 118 versions so far. The download statistics indicate that lme4 has largely replaced nlme but nlme is still getting some serious play time.

d_stats_mm <- cran_stats(c("nlme","lme4"))
ggplot(d_stats_mm, aes(end, downloads, group=package, color=package)) + geom_line() +
  ggtitle("Downloads for Mixed Model Packages") + xlab("Month")

In addition to signalling usefulness, Golden Oldieness also serves as a proxy for quality. I believe that most R users would agree that packages which have been lovingly maintained over many years and still see frequent use are most likely to contain code that you can count on. This is the kind of risk mitigation metric that is driving the R Validation Hub effort.

The following is a list of ten packages that I count as Golden Oldies. They have all been around for at least 14 years and they all get considerable play.

10 Golden Oldies
pkg	date	num_pub
quantreg	1999-01-11	67
nlme	1999-11-23	139
survival	2001-06-22	94
lme4	2003-06-25	118
Hmisc	2003-07-17	70
zoo	2004-02-20	64
data.table	2006-04-15	56
ggplot2	2007-06-10	41
xts	2008-01-05	37
Rcpp	2008-11-06	91

One Hit Wonders

On Top 40 radio One Hit Wonders could also be Golden Oldies. The designation just meant that the recording artists had one big hit and then could not find their way back into the Top 40 again. (Here is a One Hit Wonder favorite of mine from 1969.)

For CRAN packages, I am using One Hit Wonder to mean a package that has been on CRAN for many years but has never had a version bump: no bug fixes, no upgrades, no faults - just the opposite of “Golden Oldies”. Finding them is not so easy though. As far as I can tell, there is no easy way to access the CRAN archive metadata. However, you can hunt for One Hit Wonders by searching through the long left tail of the publication date distribution of CRAN packages.

This code picks out the publication dates for CRAN packages.

c_db <- tools::CRAN_package_db()
cran_db <- clean_CRAN_db()
package_network <- cran_db %>% build_network(perspective = "package")

pkg <- package_network$nodes$package
pub <- package_network$nodes$published

published <- na.omit((data.frame(pkg,pub)))
clean_pub <- published %>% filter(pub > as.Date(2005-01-01, format = "%Y-%m-%d", origin = "1900-01-01"))

And, here is the histogram. The vertical lines mark the first two quantiles.

ggplot(clean_pub, aes(x=pub)) + geom_histogram(bins = 193) +
  geom_vline(aes(xintercept = median(pub)),col='grey',size=1, linetype="dotted") +
  geom_vline(aes(xintercept = quantile(pub,probs = .25, type = 1)), col='grey',size=1, linetype="dotted") + 
  xlab("CRAN Package Publication Date") + ylab("Number of packages") + ggtitle("Histogram of Publication Dates")

When I recently ran this code, the oldest current publication date of the 18,850 package on CRAN is March 15, 2006.

summary(clean_pub)

##      pkg                 pub            
##  Length:18970       Min.   :2006-03-15  
##  Class :character   1st Qu.:2018-06-19  
##  Mode  :character   Median :2020-08-07  
##                     Mean   :2019-09-22  
##                     3rd Qu.:2021-08-13  
##                     Max.   :2022-02-16

The following code extracts the packages in the first quantile and sorts them. Determining if the publication date is the original package publication date, however, requires verifying that the package has never been archived.

q1_pub <- quantile(clean_pub$pub, type = 1)[2]
q1_pkgs <- clean_pub %>% filter(pub <= q1_pub) %>% arrange(pub)
q1_pkgs[1:20,]

##              pkg        pub
## 1      coxrobust 2006-03-15
## 2  BayesValidate 2006-03-30
## 3       fuzzyFDR 2007-10-16
## 4         poilog 2008-04-29
## 5        SASPECT 2008-06-23
## 6            RM2 2008-08-13
## 7           pack 2008-09-08
## 8         expert 2008-10-02
## 9            kzs 2008-10-28
## 10           ETC 2009-01-30
## 11 CreditMetrics 2009-02-01
## 12   Reliability 2009-02-01
## 13           spe 2009-02-24
## 14          mcsm 2009-04-28
## 15    SEMModComp 2009-05-05
## 16   bootStepAIC 2009-06-04
## 17      HybridMC 2009-06-08
## 18    PearsonICA 2009-06-29
## 19    crantastic 2009-08-08
## 20         km.ci 2009-08-30

Eyeballing the package documentation on CRAN revealed that there are nine One Hit Wonders among the first twenty oldest packages in the left tail. Here are their download stats.

But what can you say about these One Hit Wonders? Except for fuzzyFDR, the first nine don’t seem to be getting much action. The small download numbers, however, do not themselves signal a lack quality or irrelevance. There may indeed be some treasures among One Hit Wonders that survive on CRAN because unlike the music business, as long as a package plays and does not break any other package, the CRAN D.J.s keep it in the playlist.

So, are you the kind of person who enjoys sorting through old records: 33s, 45s, and maybe even 78s. Would you visit a shop like Rough Trade, the AI Record Shop or Stranded Records on a trip to New York City? If so, maybe you would enjoy looking through CRAN for esoteric packages like BayesValidate which supports an interesting paper on validating Bayesian models. Wouldn’t it be special to find a One Hit Wonder out there on CRAN in mint condition, needing no bug fixes or enhancements, that does something really sweet?

December 2021: "Top 40" New CRAN Packages

Mon, 24 Jan 2022 00:00:00 +0000

One hundred thirty-four new packages made it to CRAN last December. Here are my “Top 40” picks in eight categories: Data, Genomics, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Data

aurin v0.5.1: Implements an API for AURIN, Australia’s largest resource for accessing clean, integrated, spatially enabled and research-ready data on issues surrounding health and well being, socio-economic metrics, transportation, and land-use. See README for examples.

fastRhockey v0.1.0: Implements a utility to scrape and load play-by-play data and statistics from the Premier Hockey Federation (PHF), formerly known as the National Women’s Hockey League (NWHL), and access the National Hockey League’s stats API. See README.

pedalfast.data v1.0.0: Provides data files and documentation for PEDiatric vALidation oF vAriableS in TBI (PEDALFAST) used in Bennett et al. (2016). There is a vignette describing the PEDALFAST Data and another on Functional Status Scale.

rasterbc v1.0.1: Provides access to a large data set hosted at the Federated Research Data Repository relevant to forest ecology in British Columbia, Canada. The collection includes: elevation, biogeoclimatic zone, wildfire, and cutblocks forest attributes from Hansen et al. (2013) and Beaudoin et al. (2017). See the vignette.

spectator v0.1.0: Provides an interface to the Spectator Earth API, mainly for obtaining the acquisition plans and satellite overpasses for Sentinel-1, Sentinel-2 and Landsat-8 satellites. See the vignette.

Genomics

CAMML v0.1.1: Provides functions to create multi-label cell-types for single-cell RNA-sequencing data based on weighted VAM scoring of cell-type specific gene sets. See Schiebout (2022). There is a Quick Start guide, a Figure vignette and an extended example with melanoma data.

hacksig v0.1.1: Provides a collection of cancer transcriptomics gene signatures as well as a simple, tidy interface to compute single sample enrichment scores either with the original procedure or with three alternatives: the combined z-score of Lee et al. (2008) the single sample GSEA of Barbie et al. (2009) and the singscore of Foroutan et al. (2018). See README for examples.

mispitools v0.1.5: Provides functions for computing likelihood ratios thresholds and error rates in DNA kinship testing. See Marsico et al. (2021) for background and README for examples.

paleobuddy v1.0.0: Provides functions to simulate species diversification, fossil records, and phylogenies along with environmental data from Morlon et al. (2016). See the vignette for an overview.

toolStability v0.1.1: Provides a collection of functions for describing the stability of a trait in terms of genotype and environment and also includes a data set from Casadebaig et al. (1966). Computed indices are from Döring & Reckling (2018), Eberhart & Russell (1966), Finlay & Wilkinson GN (1963), Hanson WD (1970), Lin & Binnsn(1988), Pinthus (1973) and others. Th vignette provides an overview of the the theory and examples.

twosigma v1.0.2: Implements the TWO-Component Single Cell Model-Based Association Method for gene-level differential expression analysis and DE-based gene set testing of single-cell RNA-sequencing datasets. See Van Buren et al. (2020) and Van Buren et al. (2021) for the theory and README for examples.

Machine Learning

brulee v0.0.1: Provides high-level modeling functions to define and train models using the torch R package. Models include linear, logistic, and multinomial regression as well as multilayer perceptrons. See README for an example.

imageseg v0.4.0: Implements a general-purpose workflow for image segmentation using TensorFlow models based on the U-Net architecture by Ronneberger et al. (2015) and provides pre-trained models for assessing canopy density and understory vegetation density from vegetation photos. See the vignette.

TrueSkillThroughTime v0.1.0: Provides methods to model the entire history of game activities using a single Bayesian network allowing the information to propagate correctly throughout the system. The core ideas implemented in this project were developed by Dangauthier et al. (2007). Look here for examples.

Medicine

aba v0.0.9: Offers a tool to fit clinical prediction models and plan clinical trials using biomarker data across multiple analysis factors (groups, outcomes, predictors). There is a Package Overview and an Introduction to aba models.

expirest v0.1.2: Provides functions to estimate the release limit and the associated shelf life for chemically derived medicines in accordance with the recommendations of The Australian Regulatory Guidelines for Prescription Medicines guidance on Stability testing for prescription medicines and the International Council for Harmonisation’s guidance Q1E Evaluation of stability data. See README for examples.

fca v0.1.0: Provides functions to perform various floating catchment area methods to calculate a spatial accessibility index for demand point data. See Bauer & Groneberg (2016) for background and the vignette for an example.

grpseq v1.0: Provides functions to help with the design of group sequential trials, including non-binding futility analysis at multiple time points. See Gallo, Mao, & Shih (2014) for background and the vignette for an example.

lemna v0.9.0: Implements the model equations and default parameters for the toxicokinetic-toxicodynamic model of the Lemna (duckweed) aquatic plant. Lemna is a standard test macrophyte used in ecotox effect studies. See Schmitt et al. (2013) for background. There is an introduction and a vignette on verification.

Science

AeroSampleR v0.1.12: Provides functions to estimate ideal efficiencies of aerosol sampling through sample lines. See Hogue et al. (2014) for background and README for examples.

crestr v1.0.0: Implements the CREST climate reconstruction method to reconstruct past climates using biological proxies. See Chevalier (2021) for background. There is a Get started guide and vignettes on Theory, Formatting, and Calibration.

renz v0.1.1: Provides utilities to analyze Michaelis-Menten Equation models of enzyme kinetics. See Aledo (2021) for background along with the vignettes: Enzyme Kinetic Parameters, M-M and the Lambert W function, Linearized M-M Equations, Fitting the M-M Model, and Integrated M-M Equation .

Statistics

brmsmargins v0.1.1: Provides functions to calculate Bayesian marginal effects and average marginal effects for models fit using the brms package including fixed effects, mixed effects, and location scale models. See Pavlou et al. (2015) for background and the vignettes on marginal effects for Fixed Effects, Location Scale, and Mixed models.

changepoints v1.0.0: Implements a series of offline and/or online change-point detection algorithms for univariate mean, univariate polynomials, univariate and multivariate nonparametric settings, high-dimensional covariances, high-dimensional networks with and without missing values, high-dimensional linear regression models, high-dimensional vector autoregressive models, high-dimensional self exciting point processes, dependent dynamic nonparametric random dot product graphs, and univariate mean against adversarial attacks. See README for references for all of these methods and the vignettes VAR and Univariate means for examples.

cmtest v0.1-1: Provides functions to perform conditional moments test, as proposed by Newey (1985) and Tauchen (1985), useful to detect specification violations for models estimated by maximum likelihood. See the vignette.

gplsim v0.9.1: Provides functions that employ penalized splines to estimate generalized partially linear single index models which extend the generalized linear models to include nonlinear effect for some predictors. See Yu et al. (2017) and Yu & Ruppert (2002) for the details and README for an example.

gremes v0.1.0: Provides tools for estimation of the tail dependence parameters in graphical models parameterized by family of Huesler-Reiss distributions. See Asenova et al. (2021) for background. There is an Introduction and fifteen small vignettes described here.

kfa v0.1.0: Provides functions to explore possible factor structures for a set of variables and helps identify plausible and replicable structures via k-fold cross validation. See Flora & Flake (2017) and MacCallum et al.(1996) for background and README for examples.

lrstat v0.1.1: Provides functions to perform power and sample size calculation for non-proportional hazards model using the Fleming-Harrington family of weighted log-rank tests. See Tsiatis (1982) for background. There are six vignettes including Sample Size Calculation with Fixed Follow-up and Power Calculation Using Max-Combo Tests.

nphPower v1.0.0: Provides tools to perform combination tests and sample size calculation for fixed designs with survival endpoints using combination tests under either proportional hazards or non-proportional hazards. See Brendel et al. (2014) and Cheng & He (2021) for background on projection and maximum weighted logrank tests, Schoenfeld (1981) and and Freedman (1982) on sample size calculation under the proportional hazards assumption and Lakatos (1988) for calculation under non-proportional hazards, and the [README]() for an example.

optedr v1.0.0: Provides functions to calculate D-optimal, Ds-optimal, A-optimal, and I-optimal designs for non-linear models, via an implementation of the cocktail algorithm described in Yu (2011). See README for examples.

quid v0.0.1: Provides functions to test whether equality and order constraints hold for all individuals simultaneously by comparing Bayesian mixed models through Bayes factors. See Haaf & Rouder (2017) and Haaf, Klaassen & Rouder (2019), and Rouder & Haaf (2021) for background and the Quick Start guide and Manual for examples.

Utilities

abbreviate v0.1: Provides functions to abbreviate strings to at least minlength characters, such that they remain unique (if they were) and are recognizable. See README for examples.

fledge v0.1.0: Offers functions to streamline the process of updating changelogs (NEWS.md) and versioning R packages developed in git repositories. There is Quick Start guide, a demo, and a short vignette on internals.

qlcal v0.0.1: Provides QuantLib bindings using Rcpp via an evolved version of the initial header-only Quantuccia project offering an subset of QuantLib. See README for examples.

jagstargets Builds on targets and JAGS to implement a pipeline toolkit tailored to Bayesian statistics making it easy to set up scalable JAGS pipelines that automatically parallelize the computation and skip expensive steps when the results are already up to date. There is an Introduction and a vignette that uses simulation to validate a Bayesian model.

RCLables v0.1.0: Provides functions to assist manipulation of matrix row and column labels for all types of matrix mathematics where row and column labels are to be respected. See the vignette.

trampoline v0.1.1: Implements a trampoline algorithm based on on the Python trampoline module that enables users to write theoretically infinite recursive functions. See the vignette and README for examples.

Visualization

recolorize v0.1.0: Offers automatic, semi-automatic, and manual functions for generating color maps from images. The idea is to simplify the colors of an image according to a metric that is meaningful to the user, using deterministic methods whenever possible. There is an Introduction and a vignette for each of the six steps involved in the process: Acquisition & Preparation, Loading & Processing, Clustering, Refinement, Tweaks & Edits, and Export & Visualization.

scImmuneGraph v1.1.3: Provides functions to compute statistics and visualize multiple distributions including diversity, composition of clonotypes, abundance and length of CDR3, the abundance of V and J genes, and the abundance of V-J gene pairs; all of which are basic for single-cell immune group analysis. See the vignette.

November 2021: "Top 40" New CRAN Packages

Tue, 21 Dec 2021 00:00:00 +0000

Two hundred eleven new packages made it to CRAN in November. Here are my “Top 40” picks in thirteen categories: Computational Methods, Data, Ecology, Finance, Genomics, Humanities, Machine Learning, Medicine, Networks, Statistics, Time Series, Utilities, and Visualization. It was gratifying to see multiple packages developed for applications in the Computational Humanities. R is helping to extend the reach of data literacy.

Computational Methods

bigQF v1.6: Implements a computationally efficient leading-eigenvalue approximation to tail probabilities and quantiles of large quadratic forms (e.g. in the sequence kernel association test) and also provides stochastic singular value decomposition for dense or sparse matrices. See Lumley et al. (2018) for background and the vignettes: Checking pQF vs SKAT, Matrix-free computations, SKAT, weights, and projections.

LMMsolver v1.0.0: Implements an efficient system to solve sparse, mixed model equations. See the vignette.

nimbleAPT v1.0.4: Provides functions for adaptive parallel tempering with nimble models. See Lacki & Miasojedow (2016) and the vignette to get started.

Data

binancer v1.2.0: Implements an R client to the Binance Public Rest API for data collection on cryptocurrencies, portfolio management and trading. See README for an example.

japanstat v0.1.0: Provides tools for using the API of e-Stat, a portal site for Japanese government statistics which include functions for automatic query generation, data collection and formatting. See README for examples.

openeo v1.1.0: Provides to access data and processing for openEO compliant back-ends. See the vignette to get started.

Ecology

mFD v1.0.1: Provides functions to compute functional traits-based distances between pairs of species for species gathered in assemblages. Chao et al. (2018) and Mouillot et al. (2013) along with the other references listed for background. There are five vignettes including General Workflow and Compute and Interpret Quality of Functional Spaces

popbayes v1.0: Implements a Bayesian framework to infer the trends of animal populations over time from series of counts by accounting for count precision, smoothing the population rate of increase over time, and accounting for the maximum demographic potential of species. There is a Get started vignette and another on Getting species information.

Finance

MultiATSM v0.0.1: Provides functions to model the affine term structure of interest rates based on the single-country unspanned macroeconomic risk framework of Joslin et al. (2014) and the multi country extensions of Jotikasthira et al.(2015), and Candelon and Moura (2021). See the vignette for an example.

Genomics

gwasrapidd v0.99.12: Provides easy access to the NHGRI-EBI GWAS Catalog via the REST API. There is a Getting Started Guide, a vignette on Variants associated with BMI, and an FAQ.

phylosamp v0.1.6: Implements tools to estimate the probability of true transmission between two cases given phylogenetic linkage and the expected number of true transmission links in a sample. The methods are described in Wohl et al. (2021). There are vignettes on estimating FDR from sample size, Sample size from FDR, Sensitivity and specificity, and Examples.

Rtropical v1.2.1: Provides functions to process phylogenetic trees with tropical support vector machines and principal component analysis defined with tropical geometry. See Tang et al. (2020) for details about tropical SVMs, Page et al. (2020) and Yoshida et al. (2019) for tropical PCA, and Ardila & Develin (2007) for some background on tropical mathematics. The vignette will get you started.

Humanities

litRiddle v0.4.1: Provides a dataset, functions to explore the quality of literary novels, the data of a reader survey about fiction in Dutch, a description of the novels the readers rated, and the results of stylistic measurements of the novels. There is a vignette.

kairos v1.0.0: Provides a toolkit for absolute dating and analysis of chronological patterns, including functions for chronological modeling and dating of archaeological assemblages from count data. There is a Manual, and a Bibliography.

Machine Learning

cuda.ml v0.3.1: Implements an R interface for RAPIDS cuML, a suite of GPU-accelerated machine learning libraries powered by CUDA. Look here for an example.

innsight v0.1.1: Implements methods to analyze the behavior and individual predictions of modern neural networks including Connection Weights as described by Olden et al. (2004), Layer-wise Relevance Propagation as described by Bach et al. (2015), Deep Learning Important Features as described by Shrikumar et al. (2017), and gradient-based methods like SmoothGrad described by Smilkov et al. (2017), and Gradient x Input described by Baehrens et al. (2009). There is an Introduction and a vignette on Custom Model Definition.

topicmodels.etm v0.1.0: Provides functions to find topics in texts which are semantically embedded using techniques like word2vec or GloVe. See Dieng et al. (2019) for the details, and look here for an example.

Medicine

clustra v0.1.5: Provides functions to cluster medical trajectories of unequally spaced and unequal length time series aligned by an intervention time. See the vignette for examples.

eSIR v0.4.2: Implements the extended state-space SIR models developed by Song Lab at UM school of Public Health which include capabilities to model time-varying transmission, time-dependent quarantine, and time-dependent antibody-immunization. See Wang et al. (2020) for background, and README for examples.

QHScrnomo v2.2.0: Provides functions for fitting and predicting competing risk models, creating nomograms, k-fold cross validation, calculating the discrimination metric, and drawing calibration curves. See the vignette for a short tutorial.

rnmamod v0.1.0: Implements a comprehensive suite of functions to perform and visualize pairwise and network meta-analyses. The package covers core Bayesian one-stage models implemented in a systematic review with multiple interventions, including fixed-effect and random-effects network meta-analysis. There are vignettes on Performing a network meta-analysis and Describing the network.

whomds v1.0.1: Provides functions for calculating and presenting the results from the WHO Model Disability Survey (MDA). See Andrich (2014) for background on Rasch measurement. There are English and Spanish versions of six vignettes including Background on disability measurement and Best practices with Rasch Analysis.

Networks

diffudist v1.0.0: Enables the evaluation of diffusion distances for complex single-layer networks by providing functions to evaluate the Laplacians, stochastic matrices, and the corresponding diffusion distance matrices. See De Domenico (2017) and Bertagnolli & De Domenico (2021) for the details and the vignette for some theory and examples.

NetFACS V0.3.0: Provides functions to analyze and visualize facial communication data, based on network theory and resampling methods and primarily targeted at datasets of facial expressions coded with the Facial Action Coding System. See Farine (2017) and Carsey & Harden (2014) for background, and the vignette for an introduction.

Statistics

adass v1.0.0: Implements the adaptive smoothing spline estimator for the function-on-function linear regression model described in Centofanti et al. (2020). Look here for an example.

OneSampleMR v0.1.0: Provides functions for one sample Mendelian randomization and instrumental variable analyses including implementations of the Sanderson & Windmeijer (2016) conditional F-statistic, the multiplicative structural mean model Hernán and Robins (2006), and two-stage predictor substitution and two-stage residual inclusion estimators explained by Terza et al. (2008). There are short vignettes on the Multiplicative structural mean model and the Coditional F-Statistic.

PhaseTypeR v1.0.1: Implements functions to model continuous and discrete phase-type distributions, both univariate and multivariate. See Navarro (2019) for background. There is an Introduction and vignettes on Population Genetics and The Site Frequency Spectrum.

stan4bart v0.0-2: Fits semiparametric linear and multilevel models with non-parametric additive Bayesian additive regression trees BART and Stan. Multilevel models can be expressed using lme4 syntax. Look here to get started.

zoid v1.0.0: Fits Dirichlet regression and zero-and-one inflated Dirichlet regression (also called trinomial mixture models) with Bayesian methods implemented in Stan. There are vignettes on Fitting Models, Simulating data, Priors, and Prior sensitivity for overdispersion.

Time Series

surveil v0.1.0: Fits time series models for routine disease surveillance tasks and returns probability distributions for a variety of quantities of interest, including measures of health inequality, period and cumulative percent change, and age-standardized rates. See the vignette for an introduction.

VisitorCounts v1.0.0: Provides functions for modeling and forecasting of park visitor counts using social media data and (partial) on-site visitor counts. See Wood et al. (2013) for background and the vignette for examples.

Utilities

dateutils v0.1.5: Provides utilities for mixed frequency data, in particular, to aggregate and normalize tabular mixed frequency data, index dates to end of period, and seasonally adjust tabular data. There is an Introduction.

datefixR v0.1.2: Provides functions to fix messy dates such as those entered via text boxes. Standardizes “/” or “-”, whitespace separation, month abbreviations, and more. See the vignette.

lambdar v1.1.0: Provides functions for serving containers that can execute R code on the AWS Lambda serverless compute service. There are five vignettes including api-gateway-invocations and lambda-runtime-in-container.

mtb v0.1.1: Provides functions to summarize time related data, generate axis transformation from data, and assist Markdown and Shiny file editing. There are vignettes on working with axes, color, Markdown, and time related data.

rocker v0.2.1: Provides an R6 class interface for handling database connections using the DBI package as backend which allows handling of connections to PostgreSQL, MariaDB, SQLite and other databases. There are seven short vignettes including DBI package objects and functions in R6 rocker class and Database transactions with R6 rocker class.

shinySelect v1.0.0: Provides a customizable, select control widget for Shiny to enable using HTML in the items and KaTeX to type mathematics. Look here for examples.

Visualization

gggrid v0.1-1: Extendsggplot2 to make it easy to add raw grid output, such as customised annotations, to a plot. See the vignette for examples.

NHSRplotthedots v0.1.0: Provides tools for drawing statistical process control charts with functions to draw XmR charts, use change points and apply rules with summary indicators for when rules are breached. There is an Introduction and vignettes Deviations from Excel defaults and Number of points required.

shinymodels v0.1.0: Allows users to launch a shiny application for tidymodels results. For classification or regression models, the app can be used to determine if there is lack of fit or poorly predicted points. Look here to get started.

October 2021: "Top 40" New CRAN Packages

Mon, 29 Nov 2021 00:00:00 +0000

One hundred forty-one new packages made it to CRAN in October. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Networks, Science, Social Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

gslnls v1.0.2: Implements an R interface to nonlinear least-squares optimization with the GNU Scientific Library (GSL). See M. Galassi et al. (2009).

rsvddpd v1.0.0: Implements a robust method for computing the singular value decomposition using power density divergence. See Roy et. al (2021) for background and the vignette for an example.

Data

blsR v0.2.1: Implements v2 of the Bureau of Labor Statistics API for requests of survey information and time series data. See README to get started.

insiderTrades v0.0.1: Provides functions to download insider trading transactions and insider holdings from a public NoSQL SEC database using keyword criteria and generate a relational dataframe. See the vignettes insiderTrades and pullAndScrape.

KingCountyHouses v0.1.0: Contains data on houses in and around Seattle WA are included. Basic characteristics are given along with sale prices. See README.

meteospain v0.0.3: Provides access to different Spanish meteorological stations data services and APIs including AEMET, SMC, MG, and Meteoclimatic. There are seven short vignettes including API limits and loops and Compatibility between services.

owidR v1.1.0: Provides function to import, search, download, and visualize data from the Our World in Data website. See the vignette.

QBMS v0.6: Enables users to query the Breeding Management System database. There is a short Tutorial.

Genomics

digitalDLSorteR v0.1.1: Implements tools for the deconvolution of bulk RNA-Seq data using context-specific deconvolution models based on Deep Neural Networks using scRNA-Seq data as input. See Torroja & Sanchez-Cabo (2019) for details. There are five vignettes including Keras/TensorFlow installation and configuration and Building new deconvolution models.

Platypus v3.2.3: Implements an open-source software platform for investigating B-cell receptor and T-cell receptor repertoires from scSeq experiments which also incorporates transcriptional information involving unsupervised clustering, gene expression and gene ontology. See Yermanos et al. (2021) and the vignette to get started.

quincunx v0.1.4: Provides programmatic access to the PGS Catalog through the REST API. See README for the cheatsheet.

SNPfilter v0.1.0: Provides functions to interactively and reproducibly visualize and filter SNP (single-nucleotide polymorphism) datasets, including functions for visualizing various quality and missing data metrics for a SNP dataset, and then filtering the dataset based on user specified cutoffs. See the vignette.

Machine Learning

conText v1.0.0: Implements a framework to estimate context-specific word and short document embeddings using the a la carte embeddings approach developed by Khodak et al. (2018) and evaluate hypotheses about covariate effects on embeddings using the regression framework developed by Rodriguez et al. (2021). There is a Quick Start Guide.

pixelclassifier v1.0.0: Implements a Support Vector Machine to classify the pixels of an image file by its color. See the vignette.

predtools v0.0.2: Provides functions for evaluating predictive models, including plotting calibration curves and model-based Receiver Operating Characteristic (mROC) curves based on Sadatsafavi et al. (2021). There are three vignettes: Calibration Plot, Intercept Adjustment, and Model-based ROC.

text2sdg v0.1.1: Provides functions to identify Sustainable Development Goals in text using scientifically developed query systems, opening up the opportunity to monitor any type of text-based data, such as scientific output or corporate publications. There is a vignette.

Twitmo v0.1.1: Provides functions to collect, pre-process and analyze the contents of tweets using LDA and STM models including functions to generate tweet and hashtag maps and built-in support for LDAvis. See README for examples.

Medicine

ctrialsgov v0.2.5: Provides tools to access and query the U.S. National Library of Medicine’s Clinical Trials database including functions for searching the data using range queries, categorical filtering, and by searching for full-text keywords. There is an Introduction and a vignette on Text Analysis.

epidemia v1.0.0: Implements functions to specify and fit Bayesian statistical models for epidemics. Infections are propagated over time using self-exciting point processes. Multiple regions can be modeled simultaneously with multilevel models. See Bhatt et al. (2021) for background on the models and the package website for examples and extensive documentation.

Networks

dissCqN v0.1.0: Provides functions to calculate multiple or pairwise dissimilarity for orders q = 0-N for a set of species assemblages or interaction networks. See Chao et al. 2008 for background and the vignette for an example.

modnets v0.9.0: Implements methods for modeling moderator variables in cross-sectional, temporal, and multi-level networks including model selection techniques and a variety of plotting functions. See Swanson (2020) for details and the README for an example.

Science

cmpsR v0.1.0: Implements the Congruent Matching Profile Segments (CMPS) method Chen et al. (2019) to provide an objective comparison of striated tool marks for fired bullet correlation. See the vignette.

pvcurveanalysis v1.0.0: Provides functions to analyze and display pressure volume curves which enable the derivation of the turgor loss point, osmotic potential and apoplastic fraction. See Bartlett et al. (2012) for background and the vignette for examples.

rcbayes v0.1.0: Provides functions to estimate Rogers-Castro migration age schedules using Stan. See Rogers and Castro (1978) for background and the vignettes on Model Convergence and Rogers Castro Migration Models.

QCAcluster v0.1.0: Provides functions to allow Qualitative Comparative Analysis researchers to supplement the analysis of pooled data with a disaggregated perspective focusing on selected partitions of the data. The pooled data can be partitioned along the dimensions of the clustered data (individual cross sections or time series) to perform partition-specific truth table minimizations. There are vignettes that discuss the Aggregation, Diversity, Miimization, and Weight of partitions.

Statistics

bayesPO v0.3.1: Provides functions to create Bayesian point process models of presence only data. See Moreira (2020) for background, and the vignette for examples.

hierbase v0.1.2: Implements the algorithms for hierarchical testing of variable importance described in Meinshausen (2008) which control for family-wise error rate. See the vignette for an example.

pedmod v0.1.0: Provides functions to estimate mixed probit models, commonly called liability threshold models, for pedigree data such as that studied in Pawitan et al. (2004). See Genz & Bretz (2012) for background. There is an introduction to pedigree models and a vignette on positioning families.

slasso v1.0.0: Implements the smooth LASSO estimator for the function-on-function linear regression model described in Centofanti et al. (2020). See README for examples.

StanMoMo v1.0.0: Implements Bayesian mortality models including those developed in Lee & Carter (1992) and Cairns, Blake & Dowd (2006). There is a vignette introducing the Lee & Carter model and another on Bayesian Model Averaging.

Time Series

esemifar v1.0.1: Implements algorithms that provide non-parametric estimates of trend and its derivatives in equidistant time series with long-memory errors. See Letmathe et al. (2021) for a description of the smoothing methods employed, and README for an example.

GDPuc v0.5.1: Provides a function to convert GDP time series from one unit to another. All common GDP units are included, i.e. current and constant local currency units, US$ via market exchange rates and international dollars via purchasing power parities. See README for examples.

Utilities

framecleaner v0.2.0: Provides a friendly interface for modifying data frames with a sequence of piped commands built upon the tidyverse. See the vignette for examples.

presentr v0.1.1: Implements wrapper functions using packages openxlsx, flextable, and officer to create highly formatted MS office friendly output of data frames. Vignettes include: exportToExcel, flextableAndPowerpoint, and formattedFlextable.

shinyGovstyle v0.0.7: Implements a collection of shiny application styling tools that are the based on the GOV.UK Design System. See README for an introduction and examples.

validata v0.1.0: Provides functions for validating the structure and properties of data frames helping users to answer essential questions about a data set such as: What are the unique or missing values? and What columns form a primary key?. See the vignette for an introduction.

Visualization

easylabel v0.2.4: Implements interactive labeling of scatter plots, volcano plots and Manhattan plots using a shiny or plotly interface. Users can hover over points to see where specific points are located and click points on/off to easily label them. See the vignette for examples.

ggokabeito v0.1.0: Provides discrete scales for the colorblind-friendly Okabe-Ito palette, including color, fill, and edge_colour. See README for examples.

metaconfoundr v0.1.0: Implements an approach for evaluating bias in meta-analysis studies based on the causal inference framework. See the vignette for examples.

scatterPlotMatrix v0.1.0: Provides functions which make use of the packages htmlwidgets package and d3.js to create scatter plot matrices. See the vignette.

An R Community Public Library

Thu, 04 Nov 2021 00:00:00 +0000

If you haven’t recently visited bookdown.org, RStudio’s free site for publishing books written with the bookdown R package, you many be amazed at what is available. Currently, there are over one hundred fifty titles listed under the Books tab. These are written in a panoply of languages including Bulgarian, Chinese, English, French, German, Hindi, Italian, Japanese, Korean, Lithuanian (maybe), Norwegian, Portuguese, Russian, Slovenian (I think) and Vietnamese. The breadth of topics is extraordinary! Most books are concerned with R programming or statistical analyses, but you can find the odd volume of Russian poetry or treatise on Capital. Even with the aid of the bubble chart of tags there are hours of browsing here.

The sophistication of the content ranges from student notes to texts that have been published in hardcover. The latter include:

Happy Reading!! And, please let us know if you would like review one of these books.

September 2021: "Top 40" New CRAN Packages

Thu, 28 Oct 2021 00:00:00 +0000

Two hundred twenty new packages stuck to CRAN in September. Here are my “Top 40” picks in fourteen categories: Art, Computational Methods, Data, Econometrics, Finance and Insurance, Genomics, Machine Learning, Medicine, Networks and Graphs, Science, Statistics, Time Series, Utilities, and Visualization.

Art

rfishdraw v0.1.0: Automatically generates fish drawings using the fishdraw JavaScript library. See the vignette.

Computational Mehods

abmR v1.0.4: Supplies tools for running agent-based models (ABM) as discussed in Gochanour et al. (2021) including two movement functions based on the Ornstein-Uhlenbeck model (1930). See the vignette to get started.

SemiEstimate v1.1.3: Implements an improved method of two-step Newton-Raphson iteration based on implicit profiling. See the vignette for examples.

SimEngine v1.0.0: Implements functions for structuring, maintaining, running, and debugging statistical simulations on both local and cluster-based computing environments. Emphasis is placed on documentation and scalability. There is an Introduction, an example Power Calculation, and a vignette Comparing SE Estimators.

Data

asylum v1.0.1: Provides data on asylum and resettlement from the UK Home Office. See README to get started.

hockeyR v0.1.1: Provides functions to scrape hockey play-by-play data from NHL.com and Hockey-Reference.com including standings, player stats, and jersey number history. There is a Getting Started Guide and a vignette on Scraping.

yowie v0.1.0: Provides longitudinal wages data sets and several demographic variables from the National Longitudinal Survey of Youth from 1979 to 2018 including: the wages data from the cohort whose highest grade completed is up to high school; the wages data of the high school dropouts and; the demographic data of the cohort in the survey year 1979. See the vignette for details.

Econometrics

COINr v0.5.5: Implements a development environment for composite indicators and scoreboards including utilities for construction (indicator selection, denomination, imputation, data treatment, normalization, weighting and aggregation) and analysis (multivariate analysis, correlation plotting. Look here for an online book, and see the vignette for an extended overview.

spflow v0.1.0: Provides functions to estimate spatial econometric models of origin-destination flows which may exhibit spatial autocorrelation in both the dependent variable and the explanatory variables. See LeSage and Pace (2008) for information on the model, Dargel (2021) for information on the estimation procedures, and the vignette for examples.

Finance and Insurance

DOSPortfolio v0.1.0: Implements dynamic optimal shrinkage estimators for the weights of the global minimum variance portfolio reconstructed at given reallocation points as derived in Bodnar et al. (2021). See the Introduction.

SPLICE v1.0.0: Extends the individual claim simulator SynthETIC to simulate the evolution of case estimates of incurred losses through the lifetime of an insurance claim. See Taylor & Wang (2021) for background and the vignette to get started.

Genomics

MAnorm2 v1.2.0: Implements a method for normalizing ChIP-seq signals across individual samples or groups of samples and a system of statistical models for calling differential ChIP-seq signals between two or more biological conditions. Refer to Tu et al. (2021) and Chen et al. (2021) for the statistical details. The vignette provides several examples.

RevGadgets v1.0.0: Provides functions to process and visualize the output of complex phylogenetic analyses from the RevBayes phylogenetic graphical modeling software. Look here for a tutorial.

Machine Learning

morphemepiece v1.0.1: Provides functions to tokenize text into morphemes ether by table lookup or a modified wordpiece tokenization algorithm. There is a vignette on testing the fall-through algorithm and another on Generating a Vocabulary.

NIMAA v0.1.0: Implements a pipeline for nominal data mining, which can effectively find special relationships between data. See Jafari et al. (2021) for a description of the method and the vignette for an introduction.

Medicine

gapclosing v1.0.2: Provides functions to estimate the disparities across categories (e.g. Black and white) that persists if a treatment variable (e.g. college) is equalized. See Lundberg (2021) for the methodology and the vignette for an overview with examples.

iDOVE v1.3: Implements a nonparametric maximum likelihood method for assessing potentially time-varying vaccine efficacy against SARS-CoV-2 infection under staggered enrollment and time-varying community transmission, allowing crossover of placebo volunteers to the vaccine arm. See Lin et al. (2021) for background and the vignette for the details.

powerly v1.5.2: Implements the sample size computation method for network models proposed by Constantin et al. (2021) which takes the form of a three-step recursive algorithm to find an optimal sample size given a model specification and a performance measure of interest. See README for an example.

ppseq v0.1.1: Provides functions to design clinical trials using sequential predictive probability monitoring. See the vignette for details.

Networks and Graphs

multigrapher v0.1.0: Implements methods and models for analyzing multigraphs as introduced by Shafie (2015) including methods to study local and global properties and goodness of fit tests. See Shafle (2016). The vignette provides an introduction.

nevada v0.1.0: Implements a statistical framework for network-valued data analysis which leverages the complexity of the space of distributions on graphs. See Lovato et al. (2020) and Lovato et al. (2021) for the statistical background, and the vignette for an example.

robber v0.2.2: Implementation of a variety of methods to compute the robustness of ecological interaction networks with binary interactions as described in Chabert-Liddell et al. (2021). There is a vignette on Topological Analysis.

Science

argoFloats v1.0.3: Supports the analysis of oceanographic data recorded by Argo autonomous drifting profiling floats. See Kelley et al. (2021) for more on the scientific context and applications. There is an Introduction and a vignette on Quality Control and Adjusted Data.

biosurvey v0.1.1: Provides tools that allow users to plan systems of sampling sites with the goals of increasing the efficiency of biodiversity monitoring. See Arita et al. (2011) and Soberón & Cavner (2015) for background. There are vignettes on Preparing Data, Selecting Sample Sites, Using Preselected Points, and Testing Efficacy of Selected Sites.

HostSwitch v0.1.0: Provides functions to aims to investigate the dynamics of the host switch in the population of an organism that interacts with current and potential hosts over generations. The underlying model is based on Araujo et al. (2015). See the vignette for an example.

Statistics

cvam v0.9.2: Extends R’s implementation of categorical variables (factors) to handle coarsened observations and implements log-linear models for coarsened categorical data, including latent-class models. There is a vignette describing Coarsened Factors and another on Fitting Log-linear Models.

JustifyAlpha v0.1.1: Provides functions to justify alpha levels for statistical hypothesis tests by avoiding Lindley’s paradox, or by minimizing or balancing error rates. See Maier & Lakens (2021) for the theory and the vignette for an introduction.

sphunif v1.0.1: Implements functions to test for uniformity on circles and hyperspheres. See García-Portugués et al. (2020) for background and the vignette for examples.

spatialRF v1.1.3: Implements methods to automatically generate and select spatial predictors for spatial regression with Random Forest. See Dray et al. (2006) and RFsp Hengl et al. (2017). Look here for documentation and examples.

vdra v1.0.0: Implements three protocols for performing secure linear, logistic, and Cox regression on vertically partitioned data across several data partners in such a way that data is not shared among data partners. See Slavkovic et. al. (2007) for background. There is an Introduction and vignettes on Using PopMedNet, Communications and Files, and Workflow.

Time Series

GlarmaVarSel v1.0: Implements functions for variable selection in high-dimensional sparse GLARMA models. See Gomtsyan et al. (2020) for the theory and the vignette for an example.

mrf v0.5.1: Implements a method to forecast univariate times series using a feature extraction algorithm based on the Haar wavelet transform. See the vignette.

Utilities

colorblindcheck v1.0.0: Provides functions to compare color palettes with simulations of color vision deficiencies - deuteranopia, protanopia, and tritanopia. It includes a calculation of distances between colors and creates summaries of differences between color palettes and simulations of color vision deficiencies. See the post by G. Aisch for background and the vignette for details.

filearray v0.1.1: Implements file-backed arrays for out-of-memory computation using gigabyte-level multi-threaded read/write via OpenMP. The vignette compares performance with native R.

httr2 v0.1.1: Implements tools for creating and modifying HTTP requests, executing them, and then processing the results. httr2 is a modern re-imagining of httr that uses a pipe-based interface and solves more of the problems that API wrapping packages face. See the vignettes httr2 and Wrapping APIs.

quadtree v0.1.2: Provides region quadtrees for working with spatial data, which allow for variable-sized cells. There are vignettes on code, creation, lcp, and usage.

tabxplor v1.0.2: Implements tools to create, manipulate, and color highlight cross-tables, and export them to Excel or HTML with formats and colors. See the vignette.

Visualization

bittermelon v0.1.3: Provides functions for creating and modifying bitmaps with special emphasis on bitmap fonts and their glyphs including native read/write support for the hex and yaff bitmap font formats. Look here for documentation.

d3po v0.3.2: Implements an Apache licensed alternative to Highcharter which provides functions for interactive visualization for Markdown and Shiny. See the vignette for examples.

ggHoriPlot v1.0.0: implements geom_horizon() and other functions for building horizon plots with ggplot2. There is an Introduction and a vignette with examples.

plotbb v0.0.6: Provides a proof of concept for implementing a grammar for base R plots.See the vignette.

semptools v0.2.9.3: Implements functions for customizing structural equation plots that can be chained using a pipe operator. There is a Quick Start Guide, and vignettes on Nodes, Matrix Layout, CFA, and SEM.

August 2021: "Top 40" New CRAN Packages

Mon, 27 Sep 2021 00:00:00 +0000

One hundred sixty new packages covering a wide array of topics made it to CRAN in August. I thought I would emphasize the breadth of topics by expanding the number of categories organizing my “Top 40” selections beyond core categories that appear month after month. Here are my picks in fourteen categories: Archaeology, Computational Methods, Data, Education, Finance, Forestry, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization. Based on informal impressions formed over the last several months, I believe a new category combining applications in forestry, animal populations, climate change could become a regular core category.

Archaeology

DIGSS v1.0.2: Provides a simulation tool to estimate the rate of success that surveys including user-specific characteristics have in identifying archaeological sites given specific parameters of survey area, survey methods, and site properties. See Kintigh (1988) for background and the vignette for examples.

Computational Methods

simlandr v0.1.1: Provides a set of tools for constructing potential landscapes for dynamical systems using Monte-Carlo simulation which is especially suitable for formal psychological models. There are vignettes on Dynamic Models and Simulations, Constructing Potential Landscapes, and Calculating the Lowest Elivation Path.

Data

metaboData v0.6.2: Provides access to remotely stored data sets from a variety of biological sample matrices analyzed using mass spectrometry metabolomic analytical techniques. See the vignette.

metadat v1.0-0: Contains a collection of data sets useful for teaching meta analysis. See README for more information.

nflreadr v1.1.0: Provides functions for downloading data from the GitHub repository for the nflverse project. There is a brief Introduction and several short vignettes that serve as the data dictionary for the various files Draft Picks, Rankings, etc.

OCSdata v1.0.2: Provides functions to access and download data from the Open Case Studies repositories on GitHub. See the vignette to get started.

rATTAINS v0.1.2: Implements an interface to United States Environmental Protection Agency (EPA) ATTAINS database used to track information provided by states about water quality assessments conducted under federal Clean Water Act requirements. There is a vignette.

taylor v0.2.1: Provides access to a curated data set of Taylor Swift songs, including lyrics and audio characteristics. Data comes Genius and the Spotify API. See README for examples,

Education

karel v0.1.0: Provides an R implementation of Karel the robot, a programming language for teaching introductory concepts about general programming in an interactive and fun way, by writing programs to make Karel achieve tasks in the world she lives in. There are several vignettes including one on Control Structures and another on Algorithmic Decomposition.

roger v0.99-0: Implements tools for grading the coding style and documentation of R scripts. This is the R component of Roger the Omni Grader, an automated grading system for computer programming projects based on Unix shell scripts. Look here for more information.

Finance

dispositionEffect v1.0.0: Implements four different methodologies to evaluate the presence of the disposition effect and other irrational investor behaviors based on investor transactions and financial market data. There is a Getting Started Guide, and vignettes on Analysis, Disposition Effects in Parallel, and Time Series Disposition Effects.

HDShOP v0.1.1: Provides functions to construct shrinkage estimators of high-dimensional mean-variance portfolios and performs high-dimensional tests on optimality of a given portfolio. See Bodnar et al. (2018), Bodnar et al. (2019), and Bodnar et al. (2020) for background.

tcsinvest v0.1.1: Implements an interface to the Tinkoff Investments API which enables analysts and traders can interact with account and market data from within R. Clients for both REST and Streaming protocols have been implemented. There is a vignette.

Forestry

APAtree v1.0.1: Provides functions to map the area potentially available (APA) using the approach from Gspaltl et al. (2012) and also aggregation functions to calculate stand characteristics based on APA-maps and the neighborhood diversity index as described in Glatthorn (2021). See the vignette for examples.

efdm v0.1.0: Implements the European Forestry Dynamics Model (EFDM), a large-scale forest model that simulates the development of a forest and estimates volume of wood harvested for any given forested area. See Packalen et al. (2015) for background and the vignette for examples.

Genomics

molnet v0.1.0: Implements a network analysis pipeline that enables integrative analysis of multi-omics data including metabolomics. It allows for comparative conclusions between two different conditions, such as tumor subgroups, healthy vs. disease, or generally control vs. perturbed. The case study presented in the vignette uses data published by Krug (2020).

simtrait v1.0.21P Provides functions to simulate complex traits given a SNP genotype matrix and model parameters with an emphasis on avoiding common biases due to the use of estimated allele frequencies. Traits can follow three models: random coefficients, fixed effect sizes, and multivariate normal. GWAS method benchmarking functions as described in Yao and Ochoa (2019) are also provided. See the vignette.

statgenIBD v1.0.1: Provides functions to calculate biparental, three and four-way crosses Identity by Descent (IBD) probabilities using Hidden Markov Models and inheritance vectors following Lander & Green (1987) and Huang (2011). See the vignette for examples.

Machine Learning

text2map v0.1.0: Provides functions for computational text analysis for the social sciences including functions for working with word embeddings, text networks, and document-term matrices. For background on the methods used see Stoltz and Taylor (2019), Taylor and Stoltz (2020), Taylor and Stoltz (2020), and Stoltz and Taylor (2021). There is a Quick Start Guide and a vignette on Concept Class Analysis.

NPRED v1.0.5: Uses partial informational correlation (PIC) to identify the meaningful predictors from a large set of potential predictors. Details can be found in Sharma & Mehrotra, (2014), Sharma et al.(2016), and Mehrotra & Sharma (2006). See the vignette for examples.

stabiliser v0.1.0: Implements an approach to variable selection through stability selection and the use of an objective threshold based on permuted data. See Lima et al (2021) and Meinshausen & Buhlmann (2010) for details and the vignette for an example.

Medicine

dreamer v3.0.0: Fits longitudinal dose-response models utilizing a Bayesian model averaging approach as outlined in Gould (2019) for both continuous and binary responses. See the vignette.

smartDesign v0.72: Implements the SMART trial design, as described by He et al. (2021) which includes multiple stages of randomization where participants are randomized to an initial treatment in the first stage and then subsequently re-randomized between treatments in the following stage. There is a Dynamic Treatment Tutorial and a Sequential Design Tutorial.

Science

bootf2 v0.4.1: Provides functions to compare dissolution profiles with confidence intervals of the similarity factor f2 and also functions to simulate dissolution profiles. There are multiple vignettes including and Introduction a Simulation Example.

track2KBA v1.0.1: Provides functions to prepare and analyze animal tracking data in order to identify areas of potential interest for population level conservation. See Lascelles et al. (2016) for background on the methodology employed and the vignette for examples and workflow.

Statistics

chyper v0.3.1: Provides functions to work with the conditional hypergeometric distribution. See the vignette.

sprtt v0.1.0: Provides functions to perform sequential t-tests including those of Wald (1947), Rushton (1950), Rushton (1952), and Hajnal (1961). There is an Introduction to the package, a Use Case, and a vignette on the Sequential t-test.

SurvMetrics v0.3.5: Implements popular evaluation metrics commonly used in survival prediction including Concordance Index, Brier Score, Integrated Brier Score, Integrated Square Error, Integrated Absolute Error and Mean Absolute Error. For detailed information, see Ishwaran et al. (2008) and Moradian et al. (2017). The vignette offers examples.

Time Series

DCSmooth v1.0.2: Implements nonparametric smoothing techniques for data on a lattice or functional time series which allow for modeling a dependency structure of the error terms of the nonparametric regression model. See Beran & Feng (2002), Mueller & Wang (1994), Feng & Schaefer (2021), and Schaefer & Feng (2021) for the background and the vignette for examples.

STFTS v0.1.0: Implements statistical hypothesis tests of functional time series including a functional stationarity test, a functional trend stationarity test and a functional unit root test.

WASP v1.4.1: Implements wavelet-based variance transformation methods for system modeling and prediction. For details see Jiang et al. (2020), Jiang et al. (2020), and Jiag et al. (2021) There is a vignette with examples.

Utilities

ExpImage v0.2.0: Provides an image editing tool for researchers which includes functions for segmentation and for obtaining biometric measurements. There are several vignettes including: Contagem de bovinos, Contagem de objetos, and Como editar imagens.

meltr v1.0.0: Provides functions to read non-rectangular data, such as ragged forms of csv (comma-separated values), tsv (tab-separated values), and fwf (fixed-width format) files. See README to get started.

plumbertableau v0.1.0: Implements tools for building plumber APIs that can be used in Tableau workbooks. There is a package Introduction and vignettes on Writing Extensions, Using Extensions in Tableau, and Publishing Extensions to RStudio Connect.

string2path v0.0.2: Provides functions to extract glyph information from a font file, translate the outline curves to flattened paths or tessellated polygons, and return the results as a data.frame. See README for an example.

trackdown v1.0.0: Uses Googel Drive to implement tools for collaborative writing and editing of R Markdown and Sweave documents. There are some Tech Notes and vignettes on Features and Workflow.

Visualization

aRtsy v0.1.1: Provides algorithms for creating artwork in the ggplot2 language that incorporate some form of randomness. See README for examples and package use.

ggcleveland v0.1.0: Provides functions to produce ggplot2 versions of the visualization tools described in William Cleveland’s book Visualizing Data. The vignette contains several examples.

ggtikz v0.0.1: Provides tools to annotate ggplot2 plots with TikZ code using absolute data or relative coordinates. See the vignette.

tidycharts v0.1.2: Provides functions to generate charts compliant with the International Business Communication Standards (IBCS) including unified bar widths, colors, chart sizes, etc. There is a Getting Started guide and vignettes on EDA, Customization, and Joining Charts.

A Guide to Binge Watching R / Medicine 2021

Thu, 09 Sep 2021 00:00:00 +0000

R / Medicine is a big deal. This year, the conference grew by 13% with 665 people from over 60 countries signing up for the virtual event which was held last month. 34% percent of the registrants were from outside of the United States and 17% identified as physicians.

The conference is now an established international event where experts report on the advanced use of the R language, Machine Learning, and statistical analysis, and discuss the successes and challenges associated with bringing these technologies to day-to-day medical practice.

Almost all of the talks, including keynotes, regular talks, lightning talks, pre-conference workshops and poster sessions are available online. Find the links on the R / Medicine site or look through the playlist on the R Consortium Youtube Channel. Note that the posters can be viewed by going to the conference spatial.chat site. (If you and a friend visit at the same time you should be able to “walk around” the posters and chat about what you see.)

To kick off an evening of binge watching the conference I would begin with the keynotes.

The Keynotes

Dr. Karandeep Singh sets the hook for his talk, Bringing Machine Learning Models to the Bedside at Scale, two minutes into the video when he asks:

Who are the twenty sickest patients in the hospital right now who are not in the ICU?

This straightforward question immediately gets to the promise and the problems of introducing large scale machine learning algorithms into the hospital, and indicates how medical practice interacts with big money questions about allocating resources. Both physicians and administrators would like to identify high risk patients and treat them proactively while being able to confidently spend less on unnecessary test for low risk patients. About (5:10) into the talk, Karandeep begins discussing the challenges associated with introducing machine learning models.

In the remainder of the talk he describes the technical infrastructure and then the governance or “social infrastructure” needed for success.

If you enjoy a good detective story, and take pride in your ability to interpret a well-done statistical plot you are certainly going to want to watch Ziad Obermeyer’s keynote Dissecting Algorithmic Bias. About two minutes into the video Professor Obermeyer sets the stage with the warning:

The single greatest threat to all of the gains that we can make in using algorithms in medicine is letting them go wrong in increasingly well known ways.

and the observation that due to the focus of the US health care management on “high risk care management” an estimated 150 to 200 million Americans are sorted by algorithms every year. He goes on to work through a case study that illustrates how an algorithm built with good intentions had the effect of scaling up racial bias.

A second case study features an algorithm that “fights against” racial bias. Along the way, Ziad weaves two common themes into his presentation:

So many of the ways that algorithms can go wrong come from training algorithms with the wrong target variables, often “convenient and tempting proxies”.
The necessity of follow-up work to fix underlying problems.

In the remainder of this post, I have organized the talks into six categories that you may find helpful for setting your viewing program: Clinical Practice, Clinical Trials, Medical Data, R in Production, R Tools, and Short Courses. The majority of the talks have a machine learning angle. There is quite a bit of Shiny and several R packages, not all of them on CRAN, are featured. I have provided links when I could find them. I don’t want to spoil anyone’s fun in searching through the videos for “Easter Eggs”, but the Reproducible Research with R short course contains the first preview on the Quarto Publishing system in a talk from anyone at RStudio. (Note that the video needs some editing. Start watching at 9 minutes.)

Clinical Practice

Building an Interpretable ML Model API for Interpretation of CNVs in Patients with Rare Diseases - Francisco Requena
Subgroup Identification and Precision Medicine with the personalized R Package - Jared Huling
R and Shiny Dashboards to Facilitate Quality Improvement in Anesthesiology and Periopeartive Care - Robert Lobato
tidytof: Predicting Patient Outcomes from Single-cell Data using Tidy Data Principles - Timothy Keyes
Assessing ML Model Performance in DIverse Populations and Across Time - Victor Castro, Roy Perlis

Clinical Trials

Designing Early Phase Clinical Trials with ppseq - Emily Zabor
Collaborative, Reproducible Exploration of Clinical Trial Data - Michael Kane
Graphical Displays in R for Clinical Trials - Steven Schwager
ctrialsgov: Access, Visualization, and Discovery of the ClinicalTrials.gov Database - Taylor Arnold

Medical Data

Scaling Up and Deploying Shiny and Text Mining for National Health Decisions - Andreas Soteriade, Chris Beeley
Mapping African Health Data with afrimapr Package, Training & Community - Andy South
You R What You Measure: Digital Biomarkers for Insights in Personalized Health - Irene van den Broek
Shiny and REDCap for a Global Research Consortium - Judith Lewis, Stephany Duda
Diving into Registry Data: Using R for Large Norwegian Health Registries - Julia Romanowska
ReviewR: A Shiny App for Reviewing Clinical Records - Laura Wiley, David Mayer
DOPE: An R package for Processing and Classifying Drug Names - Layla Bouzoubaa
medicaldata for Teaching #Rstats - Peter Higgins
Stem Cell Transplant Outcomes Reporting using R/Shiny - Richard Hanna, Stephan Kadauke

R in Production

Second Server to the Right and Straight On ‘til Production: Deploying a GxP Shiny Application - Marcus Adams
Target Markdown and stantargets for Bayesian model validation pipelines - Will Landau
GENETEX: A Genomics Report Text Mining R Package to Capture Real-world Clinico-genomic Data - David Miller, Sophia Shalhout

R Tools

Generalized Additive Models for Longitudinal Biomedical Data - Ariel Mundo
Multistate Data Using the survival Package - Beth Atkinson
Bayesian Random-Effects Meta-analysis using bayesmeta - Christian Rover
An arsenal of R Functions for Statistical Summaries - Ethan Heinzen, Beth Atkinson, Jason Sinnwell
R Markdown and officedown to Automate Clinical Trial Reporting - Damian Rodziewicz
Creating and Styling PPTX Slides with rmarkdown - Emil Hvitfeldt
runway: an R Package to Visualize Prediction Model Performance - Jie Cao, Karandeep Singh
clinspacy: An R package for Clinical Natural Language Processing - Jie Cao, Karandeep Singh
Data Visualization for Machine Learning Practitioners - Julie Silge
Animated Data Visualizations with gganimate for Science Communication during the Pandemic - Kristen Panthagani
Incorporating Risk-of-Bias Assessments into Evidence Syntheses with robvis - Luke McGuinness, Randall Boyes, Alex Fowler
‘gpmodels’: A Grammar of Prediction Models - Sean Meyer, Karandeep Singh
CONSORT Diagrams in R with ggconsort - Travis Gerke

Short Courses

Secure Medical Data Collection: Best Practices with Excel, and Leveling Up to REDCap and CollaboratoR - Peter Higgins, Will Beasley, Kenneth MacLean, Amanda Miller
Introduction to R for Medical Data - Ted Laderas, Daniel Chen, Mara Alexeev
An Introductory R Guide for Targeted Maximum Likelihood Estimation in Medical Research - Ehsan Karim, Hanna Frank
Mapping Spatial Health Data - Marynia Kolak, Susan Paykin
From SAS to R - Joe Krsszun
Reproducible Research with R - Alison Hill, Stephan Kaduke, Paul Villanueva

July 2021: "Top 40" New CRAN Packages

Thu, 26 Aug 2021 00:00:00 +0000

One hundred eighty-three new packages stuck to CRAN in July. Here are my “Top 40” picks in eleven categories: Data, Ecology, Finance, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization. Although I don’t have any formal specification for these categories, I do my best to main my subjective sense of consistency from month to month. Nevertheless, watching the monthly ebb and flow of the number of packages that fit into the various categories is interesting. This month, developers seemed to be focused on utilities. I classified forty-five packages as utilities this month.

Data

beans v0.1.0: Contains data on 13,611 beans from Koklu and Ozkan (2020). The beans have been quantified using 16 morphologic image features and labeled with one of 6 values.

geckor v0.1.1: Provides functions to collect current and historical cryptocurrency market data using the public CoinGecko API. There is a brief overview and vignettes on Current and Historical market data.

ISLR2 v1.0: Contains the data sets used in the book An Introduction to Statistical Learning with Applications in R, Second Edition.

sageR v0.3.0: Provides the data sets used in the book Statistiques pour l’économie et la gestion, Théorie et applications en entreprise. Look here for descriptions of the data with code and plots.

Ecology

DA v1.2.0: Provides functions for Discriminant Analysis (DA) for evolutionary inference, especially population genetic structure and community structure inference as described in Qin et al. (2020). See the Introduction.

Ostats v0.1.0: Provides functions to calculate O-statistics, or overlap statistics, which measure the degree of community-level trait overlap. See Read et al. (2018) for background. There is an Introduction and a vignette on Multivariate Ostats.

Finance

ffp v0.1.0: Implements numerical entropy-pooling for scenario analysis as described in Meucci (2010) See the vignette.

Genomics

utr.annotation v1.0.4: Implements a fast tool for annotating potential deleterious variants in the untranslated regions for both human and mouse species. See Liu & Dougherty (2021) for background and the vignette for and introduction and examples.

OmicNavigator v1.4.3: Provides a for interactive exploration of the results from ‘omics’ experiments to facilitate novel discoveries from high-throughput biology. There is a User’s Guide and a vignette on the package’s API.

Machine Learning

BOSO v1.0.3: Implements the BiLevel Optimization Selector Operator feature selection algorithm for linear regression as described in Valcarcel et al. (2020). See the vignette for examples.

FeatureTerminatoR v1.0.0: Implements a feature selection engine that removes features with minimal predictive power. See Boughaci (2018) for background and the vignette for and example.

hhcartr v1.0.0: Implements the HHCART-G algorithm described in Wickramarachchi et al. (2019). See the vignette.

recometrics v0.1.3: Implements evaluation metrics for implicit-feedback recommender systems based on low-rank matrix factorization models, given the fitted model matrices and data. See the vignette for examples.

tidylda v0.0.1: Implements an algorithm for Latent Dirichlet Allocation (LDA) as described in Blei et at. (2003) using tidyverse principles. See README for an overview.

Medicine

dcurves v0.2.0: Implements a decision curve analysis method for evaluating and comparing prediction models that incorporates clinical consequences and requires only the data set on which the models are tested. See Vickers (2006), Vickers (2008), and Pfeiffer (2020) for background, and the vignette for examples.

KHQ v0.2.0: Provides methods to calculate scores for each dimension of the The King’s Health Questionnaire (KHQ) ; converts KHQ item scores to KHQ5D scores; and also calculates the utility index of the KHQ5D. See the vignette.

Science

AvInertia v0.0.1: Provides functions to compute the center of gravity and moment of inertia tensor of any flying bird. See the vignette for some insight into bird design.

bayesnec v1.0.1: Provides functions to fit dose concentration response curves to toxicity data, and derive No-Effect-Concentration (NEC), No-Significant-Effect-Concentration (NSEC), and Effect-Concentration (of specified percentage ‘x’, ECx) thresholds from non-linear models fitted using Bayesian MCMC fitting methods via brms and Stan. There are vignettes on Single model usage, Multi model usage, Model details, Priors, and Comparing posterior predictions.

Statistics

itdr v1.0: Provides functions to estimate the sufficient dimension reduction subspaces, i.e., central mean subspace or central subspace in regression, using Fourier transformation proposed by Zhu & Zeng (2006), the convolution transformation proposed by Zeng & Zhu (2010), and an iterative Hessian transformation methods proposed by Cook & Li (2002). See the vignette for an example.

mlfit v0.5.2: Extends the [Iterative Proportional Fitting] (IPF) algorithm which operates on count data to nested structures with constraints. See Müller & Axhausen (2011) for background and look here for and example.

optimall v0.1.0: Provides functions for the survey sampling design process with specific tools for multi-wave and multi-phase designs. Users can perform optimum allocation according to Neyman (1934) or Wright (2012). There is a User’s Guide, there are vignettes Multiwave Object and Splitting Strata with Optimall Shiny.

posterior v1.0.1: Provides tools for both users and package developers for fitting and working with Bayesian models; including tools to: efficiently convert between different formats of draws from distributions, provide consistent methods for common operations, provide summaries in convenient formats, and implement state of the art posterior inference diagnostics. See Vehtari et al. (2021) for background. There is an Introduction and a vignette on the Random Variable Datatype.

PropensitySub v0.2.0: Provides functions to estimate treatment effects in strata via inverse probability weighting or propensity score matching when subjects have missing strata labels. See the vignette for examples.

ReplicationSuccess v1.0.0: Provides utilities for the design and analysis of replication studies featuring both traditional methods based on statistical significance and more recent methods such as the skeptical p-value Held L. (2020), the harmonic mean chi-squared test Held, L. (2020), and intrinsic credibility Held, L. (2019). See the vignette for examples.

Time Series

kinematics V1.0.0: Provides functions to analyze time series representing two-dimensional movements. It accepts a data frame with a time (t), horizontal (x) and vertical (y) coordinate as columns, and returns several dynamical properties such as speed, acceleration or curvature. See the vignette for examples.

profoc v0.8.3: Provides methods to combine probabilistic forecasts using CRPS learning algorithms proposed in Berrisch & Ziel (2021) including multiple online learning algorithms such as Bernstein online aggregation as described in Wintenberger (2014).

tscopula v0.2.1: Provides functions for the analysis of time series using copula models. See McNeil (2021) and Bladt & McNeil (2020) for background. There are vignettes on Bitcoin Analysis, Models with Margins, Basic Time Series Copula Processes, and Copula Processes with V-Transforms.

Utility

codemeta v0.1.0: Provides core utilities to generate metadata with a minimum of dependencies as specified by the Codemata Project. See README for examples.

ctf v0.1.0: Provides functions to read and write data in Column Text Format (CTF), a new tabular data format that is a simple column store. See the vignette.

fedmatch v2.0.2: Provides tools for matching two un-linked data sets using exact matches, fuzzy matches, or multi-variable matches. There is an Introduction and vignettes on Fuzzy Matching, Multivar Matching, Tier Matching, and Using clean_strings.

fusen v0.2.4: Implements a method to use R Markdown to build an R package. Users start by including documentation, functions, examples and tests in the same file. Then inflating the R Markdown template copies the relevant chunks and sections in the appropriate files required for package development. There is an Introduction and a vignette on maintaining packages.

katex v1.10: Provides functions to convert LaTeX math expressions to HTML and MATHML for use in markdown documents or package manual pages in a way which eliminates the need for embedding the MathJax library into your web pages. See the vignette.

pacs v0.3.3: Provides utilities for CRAN package maintainers and R packages developers, including tools for validating packages and exploring the complexity of a specific package. See README for a list of features and examples.

RcppFarmHash v0.0.2 Implements an interface to the The Google FarmHash family of hash functions is used by the Google BigQuery data warehouse. Look here to get started.

shinyauthr v10.0: Provides in-app user authentication for Shiny applications, allowing developers to secure publicly hosted apps and build dynamic user interfaces from user information. See README to get started.

tantram.pipe v1.0.0: Allows users to build tables with customizable rows by specifying the type of data to use for each row, as well as how to handle missing data and the types of comparison tests to run on the table columns. See the vignette.

Visualization

chameleon v0.2-0: Provides functions to assign distinct colors to arbitrary multi-dimensional data, considering its structure. See the vignette for examples.

deeptime v0.1.0: Extends plotting packages such as ggplot2 and lattice to facilitate the plotting of data over long time intervals, including, but not limited to, geological, evolutionary, and ecological data. Look here for examples.

figpatch v0.1.0.1: Provides functions to arrange external figures with patchwork alongside ggplot2 plots. See README for examples.

spiralize v1.0.2: Provides functions to visualize data along an Archimedean spiral which has the advantages of being able to visualize data with very a long axis with high resolution and reveal periodic patterns in time series. Look here for links to the five vignettes.

R / Medicine 2021

Thu, 12 Aug 2021 00:00:00 +0000

R/Medicine 2021, the premier conference for the use of R in clinical applications is less than two weeks away! This conference reflects the increasing importance of data science, computational statistics and machine learning to clinical applications, and emphasizes the effectiveness of the R language as a vehicle for making data driven medicine accessible to clinicians with diverse backgrounds. The two conference keynote talks Bringing Machine Learning Models to the Bedside by Karandeep Singh and Dissecting Algorithmic Bias by Ziad Obermeyer directly address important technical and ethical issues confronting modern data driven medicine.

R/Medicine will offer six short courses spread out over the two days of August 24th and August 25th. These courses are included in the registration price. The Tuesday short courses will be:

Secure Medical Data Collection: Best Practices with Excel, and Leveling Up to REDCap and collaboratoR taught by Peter Higgins, Will Beasley, Kenneth McLean
Intro to R for Medical Data taught by Ted Laderas, Daniel Chen, Mara Alexeev
An Introductory R Guide for Targeted Maximum Likelihood Estimation in Medical Research taught by Ehsan Karim

The Wednesday short courses will be:

Mapping Spatial Health Data taught by Marynia Kolak
Reproducible Research with R taught by Alison Hill and Stephan Kadauke
From SAS to R taught by Joe Korszun

The conference will proper will run from 11:00 to 19:30 EDT on Thursday, August 26th, and from 10:50 to 18:00 EDT on Friday, August 27th. The full schedule is here. Priced at only $50 full fare, $25 for academics and $10 for students and trainees, this is an affordable, important conference that you will not want to miss.

You can register here.

To get an idea of the international scope of the conference, and a feel for what the virtual conference experience might be like, have a look at the R Journal article written by the organizing team about last year’s conference: R Medicine 2020: The Power of Going Virtual.

And finally, to get an idea of the various ways that R is contributing to getting everyday work done in clinical practice: the following are the packages from the Medicine category that made it to my lists of “Top 40” new CRAN packages in posts over the past twelve months.

“Top 40” Picks for new CRAN Packages for Medicine

afdx v1.1.1: Provides functions to estimate diagnosis performance (Sensitivity, Specificity, Positive predictive value, Negative predicted value) of a diagnostic test when there is no golden standard by estimating the attributable fraction using either a logitexponential model or a latent class model.

aldvmm v0.8.4: Fits health state utility adjusted limited dependent variable mixture models, i.e. finite mixtures of normal distributions with an accumulation of density mass at the limits, and a gap between 100% quality of life and the next smaller utility value. See Alava and Wailoo (2015) for background and the vignette for an example.

babsim.hospital v11.5.14: Implements a discrete-event simulation model for a hospital resource planning. Motivated by the challenges faced by health care institutions in the current COVID-19 pandemic, it can be used by health departments to forecast demand for intensive care beds, ventilators, and staff resources. See Ucar, Smeets & Azcorra (2019), Lawton & McCooe (2019) and the website for background, and the vignette to get started.

beats v0.1.1: Provides functions to import data from UFI devices and process electrocardiogram (ECG) data. It also includes a Shiny app for finding and exporting heart beats. See README to get started.

bhmbasket v0.9.1: Provides functions to evaluate basket trial designs with binary endpoints using Bayesian hierarchical models and Bayesian decision rules. See Berry et al. (2013), Neuenschwander et al. (2016) and Fisch et al. (2015) for background and the vignette for an example.

bp v1.0.1: Provides functions to aid in the analysis of blood pressure data of all forms by providing both descriptive and visualization tools for researchers. There is a vignette.

card v0.1.0: Provides tools to help assess the autonomic regulation of cardiovascular physiology with respect to electrocardiography, circadian rhythms, and the clinical risk of autonomic dysfunction on cardiovascular health through the perspective of epidemiology and causality. For background on the analysis of circadian rhythms through cosinor analysis see Cornelissen (2014) and Refinetti et al. (2014). There are two vignettes: circadian and cosinor.

causalCmprisk v1.0.0: Provides functions to estimate average treatment effects of two static treatment regimes on time-to-event outcomes with competing events. The method uses propensity scores weighting for emulation of baseline randomization. See the vignette.

ccoptimalmatch v0.1.0: Uses sub-sampling to create pseudo-observations of controls to optimally match cases with controls. See Mamoiris (2021) for the theory and the vignette for examples.

CHOIRBM v0.0.2: Provides functions for visualizing body map data collected with the Collaborative Health Outcomes Information Registry CHOIR). See the vignette.

clinDataReview v1.1.0: Provides functions to create interactive tables, listings and figures and associated reports for exploratory analysis in a clinical trial setting. There are vignettes on Prerocessing, Visualization, and Creating Reports.

clinUtils v0.0.4: Provides utility functions to facilitate importing, exploring, and reporting clinical data along with datasets in SDTM and ADaM format. See the Introduction.

cmprskcoxmsm v0.2.0: Provides functions to estimate treatment effect a under marginal structure model for the cause-specific hazard of competing risk events. Functions also estimate the risk of the potential outcomes, risk difference and risk ratio. See Hernan et al. (2001) for the theory and the vignette for examples.

coder v0.13.5: Provides functions to classify individuals or items based on external code data identified by regular expressions. A typical use case considers patients with medically coded data, such as codes from the International Classification of Diseases. There is an overview and vignettes on class codes, interpreting regular expressions, and example data.

covid19us v0.1.2: Implements wrapper around the COVID Tracking Project API providing data on cases of COVID-19 in the US.

covidcast v0.4.2: Provides an interface to Delphi’s COVIDcast Epidata including tools for data access, maps and time series plotting, and basic signal processing, and a collection of numerous indicators relevant to the COVID-19 pandemic in the United States. There is a Getting Started Guide, and vignettes on Computing Signal Correlations, Combining Data Sources, Manipulating Multiple Signals, and Plotting and Mapping Signals.

dataQuieR v1.0.4: Provides functions to assess data quality issues in studies. See the TMF Guideline and the DFG Project for background, and the vignette for examples.

epigraphdb v0.2.1: Provides access to the EpiGraphDB platform. There is an overview, vignettes on the API, Platform Functionality, Meta Functions and three case studies on SNP protein associations, Drug Targets and Causal Evidence

EpiNow2 v1.2.1: Provides functions to estimate the time-varying reproduction number, rate of spread, and doubling time using a range of open-source tools Abbott et al. (2020) for background, Gostic et al. (2020) for current best practices, and README for examples.

escalation v0.1.2: Implements methods for working with dose-finding clinical trials and includes a common interface to various dose-finding methodologies such as the continual reassessment method (CRM) by O’Quigley et al. (1990), the Bayesian optimal interval design (BOIN) by Liu & Yuan (2015), and the 3+3 described by Korn et al. (1994). There are vignettes on Working with dose-paths, Working with dose selectors, and Simulating dose-escalation trials.

eventglm v1.0.2 Implements methods for doing event history regression for marginal estimands, including cumulative incidence the restricted mean survival, as described in the methodology reviewed in Andersen & Perme (2010). See the vignette for examples.

eventTrack v1.0.0: Implements the hybrid framework for event prediction in clinical trials as described in Fang & Zheng (2011). See the vignette for an example.

goldilocks v0.3.0: Implements the Goldilocks adaptive trial design for a time to event outcome using a piecewise exponential model and conjugate Gamma prior distributions as described in Broglio et al. (2014). See the vignette for an example.

healthyR v0.1.1: Implements hospital data analysis workflow tools including modeling tools, and tools to review common administrative hospital data such as average length of stay, readmission rates, average net pay amounts by service lines, and more. See the vignette.

hlaR v0.1.0: Implements a tool for the eplet analysis of donor and recipient HLA (human leukocyte antigen) mismatches. There are vignettes on Imputation and Eplet Mismatch and a Shiny App as well.

inTextSummaryTable v3.0.1: Provides functions to create tables of summary statistics or counts for clinical data for TLFs. These tables can be exported as in-text table for a Clinical Study Report in MS Word format or a presentation MS PowerPoint format, or as interactive table. There is an Introduction and six more vignettes including Aesthetics and Visualization.

IPDfromKM v0.1.10: Implements a method to reconstruct individual patient data from Kaplan-Meier (KM) survival curves, visualize and assess the accuracy of the reconstruction, and perform secondary analysis on the reconstructed data. The package also implements iterative KM estimation algorithm proposed in Guyot (2012).

metaSurvival v0.1.0: Provides a function to assess information from a summary survival curve and test the between-strata heterogeneity. See the GitHub repo for an example.

nCov2019 v0.4.4: Implements an interface to disease.sh - Open Disease Data API to access real time and historical data of COVID-19 cases, vaccine and therapeutics data. There is a vignette.

NHSDataDictionaRy v1.2.1: Provides a common set of simplified web scraping tools for working with the NHS Data Dictionary.This package was commissioned by the NHS-R community to provide this consistency of lookups. See the vignette to get started.

NMADiagT v0.1.2: Implements the hierarchical summary receiver operating characteristic model developed by Ma et al. (2018) and the hierarchical model developed by Lian et al. (2019) for performing meta-analysis. It is able to simultaneously compare one to five diagnostic tests within a missing data framework.

packDAMipd v0.1.2: Provides functions to construct both time-homogenous and time-dependent Markov models for cost-effectiveness analyses, perform decision analyses, and conduct deterministic and probabilistic sensitivity analyses. There are vignettes on deterministic and probabilistic sensitivity analyses, simple “sick-sicker” models, age-dependent “sick-sicker” models, and cycle dependent models.

patientProfilesVis v2.0.1: Provides functions to create patient specific profile visualizations for exploration, diagnostic or monitoring purposes during a clinical trial which display the evolution of parameters such as laboratory measurements, ECG data, vital signs, adverse events and more. There is template for creating patient profiles from CDISC SDTM datasets, and an Introduction to the package.

psrwe v1.2: Provides tools to incorporate real-world evidence (RWE) into regulatory and health care decision making and includes functions which implement the PS-integrated RWE analysis methods proposed in Wang et al. (2019), Wang et al. (2020), and Chen et al. (2020). There is a vignette on propensity score integration.

QDiabetes v1.0-2: Calculates the risk of developing type 2 diabetes using risk prediction algorithms derived by ClinRisk. Look here for information and examples.

raveio v0.0.3: implements an interface to the RAVE (R analysis and visualization of human intracranial electroencephalography data) project which aims at analyzing brain recordings from patients with electrodes placed on the cortical surface or inserted into the brain. See Mafnotti et al. (2020) for background.

reconstructKM v0.3.0: Provides functions for reconstructing individual-level data (time, status, arm) from Kaplan-MEIER curves published in academic journals. See Sun et al. (2018) for background and the vignette for the reconstruction procedure.

RevieweR v2.3.6: Implements a portable Shiny tool to explore patient-level electronic health record data and perform chart review in a single integrated framework. This tool supports the OMOP common data model as well as the MIMIC-III data model, and chart review through a REDCap API. See the RevieweR Website for more information. There are several vignettes including Local, Docker, BigQuery and Shiny Server deployment and performing a Chart Review.

RHRT v1.0.1: Provides methods to scan RR interval data for Premature Ventricular Complexes and parameterise and plot the resulting Heart Rate Turbulence. See Schmidt et al. (1999) and Blesius et al. (2020) and the vignette for examples.

SAMBA v0.9.0: Implements several methods, as proposed in Beesley & Mukherjee (2020) for obtaining bias-corrected point estimates along with valid standard errors using electronic health records data with misclassifird EHR-derived disease status. See the vignette for details.

SteppedPower v0.1.0: Provides tools for power and sample size calculations and design diagnostics for longitudinal mixed models with a focus on stepped wedge designs using methods introduced in Hussey and Hughes (2007) and extensions discussed in Li et al. (2020). See the vignette to get started.

tboot v0.2.0: Provides functions to simulate clinical trial data with realistic correlation structures and assumed efficacy levels by using a tilted bootstrap resampling approach. There is a tutorial on The Tilted Bootstrap and another on Bayesian Marginal Reconstruction.

Tplyr v0.1.3: Implement a tool to simplify table creation and the data manipulation necessary to create clinical reports. There is a Getting Started Guide, and vignettes on Layers, Options, and Tables.

visR v0.2.0: Provides functions to generate clinical graphs and tables with sensible defaults based on graphical principles as described in: Vandemeulebroecke et al. (2018), Vandemeulebroecke et al. (2019), and Morris et al. (2019). Vignettes include Survival Analysis using CDISC ADaM standard, Creating Consort Flow Diagram, Styling Survival Plots and Survival Analysis.

June 2021: "Top 40" New CRAN Packages

Mon, 26 Jul 2021 00:00:00 +0000

One hundred ninety-seven new packages made it to CRAN in June. Here are my selections for the “Top 40” in ten categories: Computational Methods, Data, Finance, Genomics, Machine Learning, Medicine, Statistics, Time Series, Utilities, and Visualization. The Medicine category includes multiple packages for medical reporting and table building. Note that eight new packages were removed from CRAN by the time I began my research for this post on July 16th, so they were not considered for the “Top 40”.

Computational Methods

disordR v0.0-2: Provides tools for manipulating values of associative maps which are stored in arbitrary order. When associating keys with values one needs both parts to be in 1-1 correspondence. See the vignette for the theory and examples.

ICvectorfields v0.0.2: Provides functions for converting time series of spatial abundance or density data in raster format to vector fields of population movement using the digital image correlation technique. See the vignette.

rim v0.4.1: Provides an interface to the computer algebra system Maxima which includes running Maxima commands from within R, generating output in LaTeX and MathML and R Markdown. Look here for examples.

Data

PlanetNICFI v1.0.3: Provides functions to download and process Planet NICFI satellite imagery from Norway’s International Climate and Forest Initiative utilizing the Planet Mosaics API. See the vignette.

rgovcan v1.0.3: Provides access to data and other resources available through the Canadian Open Government portal. See README for examples.

wcde v0.0.1: Implements an interface to the Wittgenstein Human Capital Data Explorer. See the vignette for an overview.

Finance

etrm v1.0.1: Provides functions to perform core tasks within Energy Trading and Risk Management including calculating maximum smoothness forward price curves for electricity and natural gas contracts with flow delivery, as presented in Benth et al. (2007) and portfolio insurance trading strategies for price risk management in the forward market as described in Black (1976). There are vignettes describing the Maximum Smoothness Forward Curve and Portfolio Insurance Trading Strategies.

mshap v0.1.0: Provides functions to compute mSHAP values on two-part models as proposed by Matthews & Hartman (2021) using the TreeSHAP algorithm described in Lundberg et al. (2020). See the vignettes mSHAP and mshap plots.

Genomics

harmony v0.1.0: Implements the harmony algorithm for single cell integration described in Korsunsky et al. (2019). See the vignette.

Machine Learning

AutoScore v0.2.0: Implements an interpretable machine learning framework to automate the development of a clinical scoring model for predefined outcomes. See Xie et al. 2020 for the details, and the Guide Book to get started.

daiR v0.9.0: Implements an interface for the Google Cloud Services Document AI API with additional tools for output file parsing and text reconstruction. See the package website for more information and examples. There are six vignettes including a User Guide, Basic processing, and Extracting Tables.

luz v0.1.0: Implements a high level interface to torch providing utilities to reduce the the amount of code needed for common tasks, abstract away torch details and make the same code work on both CPUs and GPUs. See Howard et al. (2020) and Falcon et al. (2019) for background and the vignettes Get started with Luz, Custom Loops and Accelerator API.

mcboost v0.3.0: Implements Multi-Calibration Boosting and Multi-Accuracy Boosting to for the multi-calibrate the predictions of machine learning models. See the vignettes Basics and Extensions and Health Survey Example.

RFpredInterval v1.0.2: Implements various prediction interval methods with random forests and boosted forests. See Alakus et al. (2021) and Roy and Larocque (2020) for the mathematical background.

Medicine

clinUtils v0.0.4: Provides utility functions to facilitate importing, exploring, and reporting clinical data along with datasets in SDTM and ADaM format. See the Introduction.

Statistics

admix v0.3.2: Implements several methods to estimate the unknown quantities related to two-component admixture models, where the two components can belong to any distribution. See Bordes & Vandekerkhove (2010), Patra & Sen (2016), and Milhaud et al. (2021) for background. There are vignettes on Clusterine, Estimation, and Hypothesis Testing.

ahMLE v1.18: Implements methods for fitting additive hazards model which include the maximum likelihood method as well as Aalen’s method for estimating the additive hazards model. See Chengyuan Lu(2021) for details and the vignette for an example.

bayesrules v0.0.1: Provides datasets and functions used for analysis and visualizations in the online Bayes Rules book. There is a vignette on Conjugate Families and another on Model Evaluation.

dbglm v1.0.0: Provides a function to fit generalized linear models on moderately large datasets, by taking an initial sample, fitting in memory, then evaluating the score function for the full data in the database. See Lumley (2019) for the details.

fasano.franceschini.test v1.0.1: Implements the the 2-D Kolmogorov-Smirnov (KS) two-sample test as defined in Fasano & Franceschini (1987). See the vignette for examples.

flatness v0.1.4: Provides S3 classes, plotting functions, indices and tests to analyze the flatness of histograms including functions for flatness tests introduced in Jolliffe & Primo (2008), flatness indices described in Wilks (2019), and the procedure for multiple hypothesis described in Benjamini & Hochberg (1995). See the vignette for examples.

outlierensembles v0.1.0: Provides ensemble functions for detecting outliers and anomalies including a new method based on Item Response Theory described in Kandanaarachchi (2021) and methods described in Schubert et al. (2012), Chiang et al. (2017), and Aggarwal and Sathe (2015). See the vignette for examples.

susieR v0.11.42: Implements methods for variable selection in linear regression based on the Sum of Single Effects model, as described in Wang et al (2020). The Iterative Bayesian Stepwise Selection algorithm allows fitting models to large data sets with thousands of samples and hundreds of thousands of variables. There are ten short vignettes including Trend Fitting and a minimal example.

Time Series

psdr v1.0.1: Provides functions to generate and compare power spectral density plots given time series data and to compare the dominant frequencies of multiple groups of time series. Look here and here for the mathematical background. For examples look here or see this vignette. There is also an Introduction.

proteus v1.0.0 Implements a Sequence-to-Sequence Variational Model for time-feature analysis based on a wide range of different distributions. Look here for an overview with examples.

Utilities

archive v1.0.2: Implements bindings to libarchive, the multi-format archive and compression library which offer connections and direct extraction for many archive formats including tar, ZIP, 7-zip, RAR, CAB and compression formats including gzip, bzip2, compress, lzma and xz. See README for examples.

pasadr v1.0: Implements the anomaly detection method described in Aoudi et al. (2018).

recogito v0.1.1: Provides htmlwidgets bindings to the recogito and annotorious libraries to annotate text and areas of interest in images. See README to get started.

rextendr v0.2.0: Provides functions to compile and load Rust code from R along with helper functions to create R packages that use Rust code. There is a vignette on Using Rust code in R packages and another on Setting up Rust.

shinymeta v0.2.0.1: Provides tools for capturing logic in a Shiny app and exposing it as code that can be run outside of Shiny (e.g., from an R console). It also provides tools for bundling both the code and results to the end user. See README for examples

Visualization

dynplot v1.1.1: Provides functions to visualize a single-cell trajectory as a graph or dendrogram as a dimensionality reduction or heatmap of the expression data, or a comparison between two trajectories as a pairwise scatterplot or dimensionality reduction projection. See Saelens et al. (2019) for background and the vignette for examples.

gridpattern 0.2.1: Provides grid grabs to fill in a user-defined plot area with various patterns, including geometric and image-based patterns, and support for custom user-defined patterns. There is a vignette.

netplot v0.1-1: Implements a graph visualization engine that puts an emphasis on aesthetics while providing default parameters that yield out-of-the-box-nice visualizations. There is a vignette with base plot examples, and another showing graph drawing with netplot.

NGLVieweR v1.3.1: Implements an htmlwidgets interface to NGL.js enabling users to visualize and interact with protein databank PDB and structural files in R and Shiny applications. See the vignette for examples.

precisePlacement v0.1.0: Provides a selection of tools that make it easier to place elements onto a base R plot exactly where you want them. See the vignette for an overview.

May 2021: "Top 40" New CRAN Packages

Thu, 24 Jun 2021 00:00:00 +0000

Two hundred five packages made it to CRAN in May, but seven were removed before this post went to print. Here are my “Top40” picks in ten categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

madgrad v0.1.0: Implements MADGRAD, a Momentumized, Adaptive Dual Averaged Gradient method for stochastic optimization. See Defazio & Jelassi (2021) for details and README to get started.

TriDimRegression v1.0.0.0: Provides functions to fit 2D and 3D transformations using Stan which return posterior distributed for fitted parameters. There are vignettes on Transformation Matrices, Eye Gaze Mapping, and Comparing Faces. See README to get started.

Data

AtmChile v0.1.0: Provides access to air quality and meteorological information from the Chile’s National Air Quality System (S.I.N.C.A.). See READMW to get started.

basemaps v0.0.1: Provides a lightweight interface to access spatial basemaps from open sources such as OpenStreetMap, Mapbox and others. See README to get started.

causaldata v0.1.1: Contains the data sets to run the example problems in the online causal inference textbooks The Effect and Causal Inference: What If and more.

exoplanets v0.2.1: Provides access to NASA’s Exoplanet Archive. See the vignette to get started.

frenchdata v0.1.1: Provides access to Kenneth’s French finance data library. See the vignette for basic usage.

tradepolicy v0.5.0: Provides access to the data sets from Yotov et al. (2016) along with an online book containing commentary and the code to recreate the original analysis.

Genomics

artemis v1.0.7: Provides a modeling framework for the design and analysis of experiments collecting environmental DNA. There is an Introduction and also vignettes on Modeling eDNA and qPCR Data and Simulating eDNA Data.

MAGEE v1.0.0: Provides functions to perform variant set-based main effect tests, gene-environment interaction tests, and joint tests for association, as proposed in Wang et al. (2020). See the vignette for details.

MultIS v0.5.1: Implements a bioinformatical approach to detect the multiple integration of viral vectors within the same clone. See the vignette for how to use the package.

TopDom v0.10.0: Provides functions to identify topological domains in genomes from Hi-C sequence data as described in Shin et al. (2016). See README to get started.

Machine Learning

cjbart v0.1.0: Implements a tool for analyzing conjoint experiments using Bayesian Additive Regression Trees (BART), a machine learning method developed by Chipman & McCulloch (2010). See the vignette for examples.

fastText v1.0.1: Implements an interface to Facebook’s fastText Library. See Bojanowski et al. (2017) for a description of the algorithm. There is a Benchmark vignette and an Introduction.

Medicine

eventTrack v1.0.0: Implements the hybrid framework for event prediction in clinical trials as described in Fang & Zheng (2011). See the vignette for an example.

Science

CopernicusDEM v1.0.1: Provides an interface to the Copernicus DEM Digital Elevation Model of the European Space Agency with 90 and 30 meters resolution using the AWS CLI command line tool. See the vignette for an example.

nimbleCarbon v0.1.2: Provides functions and a custom probability distribution for Bayesian analyses of radiocarbon dates within the nimble modeling framework, including a suite of functions for prior and posterior predictive checks for demographic inference as described in Crema & Shoda (2021). See the Introduction.

Statistics

bayesmodels v0.1.0: Implements a framework to bring a number of Bayesian models into the tidymodels ecosystem. See the vignette for an overview.

div v0.3.1: Provides functions to facilitate the analysis of teams in a corporate setting, assess the diversity per grade and job, search for bias and also provides methods to simulate the effects of bias. See De Brouwer (2021) and De Brouwer (2020) for background. Look here to get started.

HotellingEllipse v0.1.1: Provides functions to compute the semi-axes lengths and coordinate points of Hotelling ellipse. See Bro & Smilde (2014) and Brenton (2016) for background. Look here and at the vignette for examples.

makemyprior v1.0.0: Provides tools to construct and visualize joint priors for variance parameters. Vignettes provide examples for Latin Square, i.i.d. models, neonatal mortality, and wheat breeding.

Rage v1.0.0: Provides functions for calculating life history metrics using matrix population models (MPMs) as described in Jones et al. (2021). There is a Getting Started Guide and vignettes on Vital Rates, Life History Traits, Deriving Age, and Ternary Plots.

unusualprofile v0.1.0: Provides functions to calculate Mahalanobis distance for every row of a set of outcome variables. There is an Introduction and a vignette on the calculations.

Time Series

gsignal v0.3-2: Implements the Ovtave signal package which provides a variety of signal processing tools, such as signal generation and measurement, correlation and convolution, filtering, filter design, filter analysis and conversion, power spectrum analysis, system identification, decimation and sample rate change, and windowing. See the vignette for an introduction.

legion v0.1.0 Provides functions for implementing multivariate state space models such as Vector Exponential Smoothing and Vector Error-Trend-Seasonal models, for time series analysis and forecasting as described in de Silva et al. (2010) There is a Function Overview and vignettes on Vector ES and Vector ETS.

Utilities

parsermd v0.1.2: Implements formal grammar and parser for R Markdown documents using the Boost Spirit X3 library. It also includes a collection of high level functions for working with the resulting abstract syntax tree. There is a Getting Started Guide and a vignette on Rmd Templates.

riskmetric v0.1.0: Provides facilities for assessing R packages against a number of metrics to help quantify their robustness. Look here for background on the package and here for background on the R Consortium, R Validation Hub project. There is a Quick Start Guide and a vignette on Extending riskmetric.

shinyvalidate v0.1.0: Provides functions to improve the user experience of Shiny apps by providing feedback when required inputs are missing, or input values are not valid. See README to get started.

ttt v1.0: Provides tools to create structured, formatted HTML tables. See the vignette.

Visualization

fitbitViz v1.0.1: Implements a connection to the Fitbit Web API to provide ggplot2, Leaflet and Rayshader visualizations. See the vignette for examples.

ggbreak v0.0.3: Implements scale functions for setting axis breaks for ggplot2. See the vignette.

ggpp v0.4.0: Provides extensions to ggplot2 to add inserts to plots using both native and npc data coordinates. See the vignette for examples.

ggseg v1.6.3: Implements a ggplot geom for plotting brain atlases using simple features. The largest component of the package is the data for two built-in atlases. See Mowinckel & Vidal-Piñero (2020) for background. There is an Introduction along with vignettes on external data, Freesurfer files, using atlases.

ichimoku v0.2.0: Implements Ichimoku Kinko Hyo, also commonly known as cloud charts, including static and interactive visualizations with tools for creating, backtesting and developing quantitative ichimoku strategies. There is a Reference and a vignette on Strategies.

liminal v0.1.2: Provides functions for composing interactive visualizations and creating linked interactive graphics for exploratory high-dimensional data analysis. See Lee et al. (2020) for background. There is a vignette on Exploring Non-linear Embeddings and another on the the geometry of Parameter Space.

mipplot v0.3.1: Provides generic functions to produce area, bar, box, and line plots following Integrated Assessment Modeling Consortium (IAMC) submission format in order to visualize climate migration scenarios. See the vignette for first steps.

qqboxplot v0.1.0: Implements Q-Q boxplots as an extension to ggplot2. There is a vignette on Basic Usage and another that provides Examples.

Summer Conferences!

Thu, 17 Jun 2021 00:00:00 +0000

Summer is here, but it is not too late sign up for some summer conferences. The following short list promises interesting speakers, a wide range of topics and plenty of R content.

June (21 - 23) - The PSI 2021 conference is online and the Registration Portal is still open. Keynote speakers Alan Smith and Ian Bott from the Financial Times, and Janet Wittes, President of the WCG Statistics Collaborative, head the program.

June (21 - 24) - The Nonclinical Biostatistics Conference 2021 is virtual. Wendy Martinez of the Bureau of Labor statistics will present A Conversation About Data Ethics, Nassim Taleb of Black Swan fame will deliver the keynote on Statistical Consequences of Fat Tails, and Di Cook and RStudio’s Carson Sievert will both talk in the Statistical Computational & Visualization Session. Registration is open.

July (1 to 2) - R for HTA Annual Workshop This online workshop from the Health Technology Assessment Consortium will be focused on R for trial and model-based cost-effectiveness analysis. Registration is open until June 30.

July (5 - 9) - useR!2021 looks like it is going to be a blockbuster of a conference. The keynote talks alone would be worth the price of admission. This exceptional lineup comprises a remarkably diverse, international group of long-time contributors, new faces, R developers, statisticians, journalists, and educators representing the global R community and speaking on a wide range of topics. I am very pleased to be presenting A little bit about RStudio on July 9 at UTC 9PM. Registration closes on June 25.

July (28 - 30) - Juliacon 2021 will be online and everywhere and Free! Long time R contributor Jan Vitek, Xiaoye Li of the Lawrence Livermore National Laboratory, and Soumith Chintala of Facebook AI Research will be the keynote speakers. It is free but you need to Register.

Aug (8 - 12) - The JSM will be online. A keyword search using R and Shiny will turn up quite a few interesting talks. I am particularly looking forward to the talk Simulating Clinical Trials Data with Synthetic.Cdisc.Data and Respectables] by Gabe Becker and Adrian Waddell and the session on Tools to Enable the Use of R by the Biopharmaceutical Industry in a Regulatory Setting which contains five talks from members of the R Consortium’s R Validation Hub working group.

August (24 - 27) - R/Medicine 2021 is online and on track to repeat the last year’s international success. Registration is open. The deadline for submitting Abstracts is June 25. Workshops being planned include:

R/Med 101: Intro to R for Clinicians and Healthcare Professionals
R Markdown for Reproducible Research (R³)
SAS 2 R: Getting off the Island!
From Excel to R+REDcap
Spatial Analysis of Healthcare Data

Sept (6 to 9) - RSS 2021 International Conference The Royal Statistical Society conference hopes to be in person in Manchester, UK. The keynote speakers will be Tom Chivers and David Chivers, Melinda Mills, Jonty Rougier, Eric Tchetgen Tchetgen, Bin Yu, and Hadley Wickham. Submissions for poster presentations are currently open with a deadline of July 1. Registration is open with an early booking discount available until June 4.

Functional PCA with R

Thu, 10 Jun 2021 00:00:00 +0000

In two previous posts, Introduction to Functional Data Analysis with R and Basic FDA Descriptive Statistics with R, I began looking into FDA from a beginners perspective. In this post, I would like to continue where I left off and investigate Functional Principal Components Analysis (FPCA), the analog of ordinary Principal Components Analysis in multivariate statistics. I’ll begin with the math, and then show how to compute FPCs with R.

As I have discussed previously, although the theoretical foundations of FDA depend on some pretty advanced mathematics, it is not necessary to master this math to do basic analyses. The R functions in the various packages insulate the user from most of the underlying theory. Nevertheless, attaining a deep understanding of what the R functions are doing, or looking into any of the background references requires some level of comfort with the notation and fundamental mathematical ideas.

I will define some of the basic concepts and then provide a high level roadmap of the mathematical argument required to develop FPCA from first principals. It is my hope that if you are a total newcomer to Functional Data Analysis you will find this roadmap useful in apprehending the big picture. This synopsis closely follows the presentation by Kokoszka and Reimherr (Reference 1. below).

We are working in $\mathscr{H}$, a separable Hilbert space of square integrable random functions where each random function, $X(\omega,t)$, where $\omega \in \Omega$ the underlying space of probabilistic outcomes, and $t \in [0,1]$. (After the definitions below, I will suppress the independent variables and in most equations assume $EX = 0$.)

Definitions

A Hilbert Space $\mathscr{H}$ is an infinite dimensional vector space with an inner product denoted by $<.,.>$. In our case, the vectors are functions.
$\mathscr{H}$ is separable if there exists an orthonormal basis. That is, there is an orthogonal collection of functions $(e_i)$ in $\mathscr{H}$ such that $<e_i,e_j>\; = 0$ if $i = j$ and 0 otherwise, and every function in $\mathscr{H}$ can be represented as a linear combination of these functions.
The inner product of two functions $X$ and $Y$ in $\mathscr{H}$ is defined as $<X,Y>\; = \int X(\omega,t) Y(\omega,t)dt$.
The norm of $X$ is defined in terms of the inner product: $\parallel X(\omega) \parallel ^2\; = \int X(\omega, t)^2 dt < \infty$.
$X$ is said to be square integrable if $E\parallel X(\omega) \parallel ^2 < \infty$.
The covariance operator $C(y): \mathscr{H} \Rightarrow \mathscr{H}$ for any square integrable function $X$ is given by: $C(y) = E[<X - EX,y>(X - EX)]$

The Road to Functional Principal Components

As we have seen, the fundamental idea of Functional Data Analysis is to represent a function $X$ by a linear combination of basis elements. In the previous posts we showed how to accomplish this using a basis constructed from more or less arbitrarily selected B-spline vectors. But, is there a an empirical, some would say natural basis that can be estimated from the data? The answer is yes, and that is what FPCA is all about.

A good way to start is to look at the distance between a vector $X$ and its projection down into the space spanned by some finite, p-dimensional basis $(u_k)$ which is expressed in the following equation,

$D = E\parallel X - \sum_{k=1}^{p}<X, u_k>u_k\parallel^2$ $(*)$

This expands out to:

$= E [< (X - \sum_{k=1}^{p}<X, u_k>u_k, X - \sum_{k=1}^{p}<X, u_k>u_k)>]$

and with a little algebra this:

$= E\parallel X \parallel^2 - \sum_{k=1}^{p}E<X, u_k>^2$

It should be clear that we would want to find a basis that makes $D$ as small as possible, and that minimizing $D$ is equivalent to maximizing the term to be subtracted in the line above.

A little algebra shows that, $E<X, u_k>^2 \;=\; <C(u_k),u_k>$ where $C(u_k)$ is the covariance operator defined above.

Now, we are almost at our destination. There is a theorem (e.g. Theorem 11.4.1 in reference 1.) that says for any fixed number of basis elements p, the distance D above is minimized if $u_j = v_j$ where the $v_j$ are the eigenfunctions of $C(y)$ with respect to the unit norm. From this it follows that $E<X, v_j>^2 \;=\; <C(v_,),v_j>\; =\; <\lambda_j, v_j>\; =\; \lambda_j$.

Going back to equation $(*)$, we can expand $X$ in terms of the basis $(v_j)$ so $D = 0$ and we have what is called the Karhunen–Loève expansion: $X = \mu + \sum_{j=1}^{\infty}\xi_jv_j$

where:

$\mu = EX$
The deterministic basis functions $(v_j)$ are called the functional principal components
The $(v_j)$ have unit norm and are unique up to their signs. (You can work with $v_j$ or $-v_j$.)
The eigenvalues are such that: $\lambda_1 > \lambda_2 > . . . \lambda_p$
The random variables $\xi_j =\; <X - \mu,v_j>$ are called the scores.
$E\xi_j = 0$, $E\xi_j^2 = \lambda_j$ and $E|\xi_i\xi_j| = 0,\; if\; i\;\neq\;j$.

And finally, with one more line:
$\sum_{j=1}^{\infty}\lambda_j \: = \: \sum_{j=1}^{\infty}E[<X,v_j>^2] = E\sum_{j=1}^{\infty}<X,v_j>^2 \; = \; E\parallel X \parallel^2 \; < \infty$

we arrive at our destination, the variance decomposition: $E\parallel X - \mu \parallel^2 \;= \;\sum_{j=1}^{\infty}\lambda_j$

Let’s Calculate

Now that we have enough math to set the context, let’s calculate. We will use the same simulated Brownian motion data that we used in the previous posts, and also construct the same B-spline basis that we used before and save it in the fda object W.obj. I won’t repeat the code here.

The following plot shows 120 simulated curves, each having 1000 points scattered over the interval [0, 100]. Each curve has unique observation times over that interval.

For first attempt at calculating functional principal components we’ll use the pca.fd() function from the fda package. So set up wise, we are picking up our calculations exactly where we left off in the previous post. As before, the basis representations of these curves are packed into the fda object W.obj. The function pca.fd() takes W.obj as input. It needs the non-orthogonal B-spline basis to seed its computations and the estimate the covariance matrix and the orthogonal eigenvector basis $v_j$. The nharm = 5 parameter requests computing 5 eigenvalues.

The method of calculation roughly follows the theory outlined above. It starts with a basis representation of the functions, computes the covariance matrix, and calculates the eigenfunctions.

fun_pca <- pca.fd(W.obj, nharm = 5)

The object produced by pca.fd() is fairly complicated. For example, the list fun_pca$harmonics does not contain the eignevectors themselves, but rather coefficients that enable the eigenvectors to be computed from the original basis. However, because there is a special plot method for plot.pca.fd() it is easy to plot the eigenvectors.

plot(fun_pca$harmonics, lwd = 3)

## [1] "done"

It is also to obtain the eivenvalues $\lambda_j$,

fun_pca$values

##  [1] 37.232207  3.724524  1.703604  0.763120  0.547976  0.389431  0.196101
##  [8]  0.163289  0.144052  0.116587  0.089307  0.057999  0.054246  0.050683
## [15]  0.042738  0.035107  0.031283  0.024905  0.019079  0.016428  0.011657
## [22]  0.007392  0.002664

and, the proportion of the variance explained by each eigenvalue.

fun_pca$varprop

## [1] 0.81965 0.08199 0.03750 0.01680 0.01206

A Different Approach

So far in this short series of FDA posts, I have been mostly using the fda package to calculate. In 2003 when it was released, it was ground breaking work. It is still the package that you are most likely to find when doing internet searches, and is the foundation for many subsequent R packages. However, as the CRAN Task View on Functional Data Analysis indicates, new work in FDA has resulted in several new R packages. The more recent fdapace takes a different approach to calculating principal components. The package takes its name from the (PACE) Principal Components by Conditional Expectation algorithm described in the paper by Yao, Müller and Wang (Reference 4. below). The package vignette is exemplary. It describes the methods of calculation, develops clear examples and provides a list of references to guide your reading about PACE and FDA in general.

A very notable feature of the PACE algorithm is that it is designed specifically to work with sparse data. The vignette describes the two different methods of calculation that package functions employ for sparse and non-sparse data. In this post, In this post we are not working with sparse data, but hope to do so in the future. See the vignette for examples of FPCA with sparse data.

library(fdapace)

The fdapace package requires data for the functions (curves) and associated times be organized in lists. We begin by using the fdapace::CheckData() function to check the data set up in the tibble df. (See previous post on descriptive statistics for the details on the data construction.)

CheckData(df$Curve,df$Time)

No error message is generated, so we move on th having the FPCA() function calculate the FPCA outputs including:

W_fpca <- FPCA(df$Curve,df$Time)

the eigenvalues:

W_fpca$lambda

## [1] 37.5386  3.2098  1.0210  0.3533

the cumulative percentage of variance explained by the eigenvalue

W_fpca$cumFVE

## [1] 0.8875 0.9634 0.9875 0.9959

and the scores:

head(W_fpca$xiEst)

##         [,1]    [,2]     [,3]     [,4]
## [1,]  0.8187  1.1971 -1.77755  0.20087
## [2,] -8.7396  2.4165 -0.33768 -0.10565
## [3,] -2.7517 -0.4879 -0.06747 -0.13953
## [4,]  3.9218  0.9419  0.39098 -0.25449
## [5,] -1.4400  0.7691  2.11549 -0.59947
## [6,]  7.3952  0.5114 -2.16391  0.05199

All of these are in fairly good agreement with what we computed above. I am, however, a little surprised by the discrepancy in the value of the second eigenvalue. The default plot method for FPCA() produces a plot indicating the density of the data, a plot of the mean of the functions reconstructed from the eigenfunction expansion, a scree plot of the eigenvalues and a plot of the first three eigenfunctions.

plot(W_fpca)

Finally, it has probably already occurred to you that if you know the eigenvalues and scores, the Karhunen–Loève expansion can be used to simulate random functions. It can be shown that for the Wiener process:

$v_j(t) = \sqrt2 sin(( j - \frac{1}{2})\pi t)$ and $\lambda_j = \frac{1}{(j - \frac{1}{2})^2\pi^2}$

This gives us:

$W(t) = \sum_{j=1}^{\infty} \frac {\sqrt2}{(j - \frac{1}{2})\pi)} N_j sin(( j - \frac{1}{2})\pi t$

where the $N_j \:are\; iid \;N(0,1)$.

The fdapace function fdapace::Wiener() uses this information to simulate an alternative, smoothed version of the Brownian motion, Wiener process.

set.seed(123)
w <- Wiener(n = 1, pts = seq(0,1, length = 100))
t <- 1:100
df_w <- tibble(t, as.vector(w))
ggplot(df_w, aes(x = t, y = w)) + geom_line()

In future posts, I hope to continue exploring the fdapace package, including its ability to work with sparse data.

References

I found the following references particularly helpful.

Kokoszka, P. and Reimherr, M. (2017). Introduction to Functional Data Analysis. CRC.
Hsing, T and Eubank, R. (2015). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators Wiley
Wang, J., Chiou, J. and Müller, H. (2015). Review of Functional Data Analysis
Yao, F., Müller, H, Wang, J. (2012). Functional Data Analysis for Sparse Longitudinal Data JASA J100, I 470

R for Public Health

Wed, 02 Jun 2021 00:00:00 +0000

The COVID19 pandemic has raised the profile of public health workers at all levels from the nurses and doctors working on the front lines at our hospitals, to high level state and federal public health officials. I think its a good bet that eighteen months ago few of us had any clear idea about how the public health care system works, or thought much about the people charged with the awesome responsibility to keep us safe. We are all a little bit wiser now. It strikes me as obvious that we will have a continuing need to improve our public health systems and that this need will create opportunities for data scientists to make significant contributions, in research, logistics, data management, reporting, public communication and many more areas. The recent post by my colleague Jesse Mostipak describes how R and Shiny made a big difference in the nitty gritty work required to roll out vaccine distribution in West Virginia. This four minute video about how the West Virginia Army National Guard built a COVID vaccine inventory management system is inspiring.

The following are some R resources that you may find helpful if you are seeking to increase your R skills with a eye toward public health applications. Some may be useful to public health professionals seeking to learn R, others may be interesting to R users who want to investigate data science in a public health context.

Courses

The Johns Hopkins Graduate Institute of Epidemiology and Biostatistics has a slew of of summer courses (many of them online), including Introduction to R For Public Health Researchers.
If you are feeling optimistic about traveling and can make your to Estonia this summer you might consider Statistical Practice in Epidemiology using R.
If you hurry, you may be able to get into the Coursera course offered by Imperial College London Statistical Analysis with R for Public Health Specialization.
There are several Coursera Courses for R in a public health context.
Frank Harrell’s free online course Biostatistics for Biomedical Research, available on YouTube: BBRcourse, is an excellent introduction to the basic statistical concepts underlying all medical applications.

Books

If you can learn on your own with the help of a good book, here are some ideas.

The Epidemiologist R Handbook: R for applied epidemiology and public health is a delightful, brief, modern introduction to R that covers the basics of the Tidyverse, R Markdown, Shiny and data.table.
Modern Statistics for Modern Biology: This book is focused more on genomics than public health applications, but it is probably the best introductory statistical text available. The modern statistics part in the title means statistics as a data driven, computational based science. Every topic is illustrated with well-crafted R code and visualizations. The book is great read.
Population Health Data Science with R: An introduction to R by authors who see population health as a systems framework for studying and improving the health of populations through collective action and learning.
Epidemiology with R: This is a new, reasonably priced book that covers the basics. It emphasizes reproducibility with R Markdown. Code examples are written in a base R style that matches the extensively used Epi package.

R Packages

If working through a book on you own seems too much of a stretch, then digest an R package or two. Here are a few examples of packages on public health themes with sufficient documentation to make interesting self-learning projects.

EpiModel: An R Package for Mathematical Modeling of Infectious Disease over Networks
PHEindicators: Common Public Health Statistics and their Confidence Intervals
SimInf: An R Package for Data-Driven Stochastic Disease Simulations
SPARSEMODr: Construct spatial, stochastic disease models that show how parameter values fluctuate in response to public health interventions

Shiny

Finally, if you find yourself in a situation similar to the West Virginia Army National Guard team featured in the video, you may just want to teach yourself Shiny. The RStudio Shiny Tutorial, along with Hadley Wickham’s book Mastering Shiny, is a very good place to start. If you need more structure, you might check out the Udemy Courses, or work through the online workshops from Duke University or the University of Manchester. And by all means, immerse yourself in the examples, posts and podcasts of the Shiny Developer Series.

April 2021: "Top 40" New CRAN Packages

Tue, 25 May 2021 00:00:00 +0000

One hundred seventy-nine new packages made it to CRAN in April. Here are my “Top 40” picks in twelve categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Networks, Operations Research, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

abess v0.1.0: Provides a toolkit for solving the best subset selection problem in linear regression, logistic regression, Poisson regression, Cox proportional hazard model, multiple-response Gaussian, and multinomial regression. It implements and generalizes algorithms described in Zhu et al. (2020) that exploit a novel sequencing-and-splicing technique to guarantee exact support recovery and globally optimal solution in polynomial times. There is an Introduction.

eat v0.1.0: Provides functions to determine production frontiers and technical efficiency measures through non-parametric techniques based upon regression trees. See Esteve et al. (2020) for details. There is an Introduction.

Data

childdevdata v1.1.0: Bundles publicly available data sets with individual milestone data for children aged 0-5 years, with the aim of supporting the construction, evaluation, validation and interpretation of methodologies that aggregate milestone data into informative measures of child development. See README.

datagovindia v0.0.3: Allows users to search the open data platform of the government of India to communicate with the more than 80,000 available APIs. See the vignette.

lehdr v0.2.4: Provides functions to query the LODES FTP server to obtain longitudinal Employer-Household Dynamics data and optionally aggregate Census block-level data. See the vignette.

rbioapi v0.7.0: Provides a consistent R interface to the Biologic Web Services API and fully supports miEAA, PANTHER, Reactome, String, and UniProt. See this vignette to get started.

tidywikidatar v0.2.0: Provides functions to query Wilidata, get tidy data frames in response, and cache data in a local SQLite database. See README.

Genomics

protti v0.1.1: Provides functions and workflows for proteomics quality control and data analysis of both limited proteolysis-coupled mass spectrometry and regular bottom-up proteomics experiments. See Feng et. al. (2014) for background. There are vignettes for various workflows: Dose Response, Single Treatment Dose Response, Input Preparation, and Quality Control.

Rediscover v0.1.0: Implements an optimized method for identifying mutually exclusive genomic events based on the Poisson-Binomial distribution that takes into account that some samples are more mutated than others. See Canisius et al. (2016). The vignette provides an introduction.

Machine Learning

geocmeans v0.1.1: Provides functions to apply spatial fuzzy unsupervised classification, visualize and interpret results, as well as indices for estimating the spatial consistency and classification quality. See Cai et al. (2007), Zaho et al. (2013), and Gelb & Appaericio (2021) for background. There is an Introduction and an additional vignette.

Rforestry v0.9.0.4: Provides fast implementations of Honest Random Forests, Gradient Boosting, and Linear Random Forests, with an emphasis on inference and interpretability. See Kunzel et al. (2019). See README to get started.

Mathematics

elasdics v0.1.2: Provides functions to align curves and to compute mean curves based on the elastic distance defined in the square-root-velocity framework. For information on the framework see Srivastava and Klassen (2016), For more theoretical details see Steyer et al. (2021)

jordan v1.0-1: Provides functions to manipulate Jordan Algebras, commutative but non-associative algebraic structures that satisfy the Jordan Identify: (xy)x² = x(yx²). See McCrimmon (204).

Medicine

ccoptimalmatch v0.1.0: Uses sub-sampling to create pseudo-observations of controls to optimally match cases with controls. See Mamoiris (2021) for the theory and the vignette for examples.

nCov2019 v0.4.4: Implements an interface to disease.sh - Open Disease Data API to access real time and historical data of COVID-19 cases, vaccine and therapeutics data. There is a vignette.

hlaR v0.1.0: Implements a tool for the eplet analysis of donor and recipient HLA (human leukocyte antigen) mismatches. There are vignettes on Imputation and Eplet Mismatch and a Shiny App as well.

Networks

greed v0.5.1: Provides an ensemble of algorithms to enable clustering of networks and data matrices with different type of generative models. Model selection and clustering is performed in combination by optimizing the Integrated Classification Likelihood. The optimization is performed with a combination of greedy local search and a genetic algorithm. See Côme et al. (2021) for background and the vignettes on Gaussian Mixture Models and Clustering.

Operations Research

critpath v0.1.2: Provides functions to compute critical paths, schedules, PERT charts and Gantt charts. There is a vignette on CPM and PERT and another on the LESS Method.

himach v0.1.2: Provides functions to compute the best routes between airports for supersonic aircraft flying subsonic over land. There is an Introduction to Supersonic Routing and a vignette on Advanced Supersonic Routing.

Statistics

convdistr v1.5.3: Provides functions to compute convolutions of probability distributions via a method that creates a new random number function for individual random samples from the random generator function of each distribution. There is an Introduction and a vignette on Sample Size.

gamlss.lasso v1.0-0: Provides an interface for extra high-dimensional smooth functions for Generalized Additive Models for Location Scale and Shape (GAMLSS) including lasso, ridge, elastic net and least angle regression. The gamlss website provides considerable information.

GGMnonreg v1.0.0: Provides functions to estimate non-regularized Gaussian graphical models, Ising models, and mixed graphical models. See Williams et al. (2019), Williams & Rast (2019), and Williams (2020) for details. README contains examples.

relevance v1.1: Implements the concepts of relevance and significance measures introduced in Stahel (2021) to augment inference with p-values. See the vignette for examples.

sasfunclust v1.0.0: Implements the sparse and smooth functional clustering method described in Centofanti et al. (2021) that aims to classify a sample of curves into homogeneous groups while jointly detecting the most informative portions of domain. See README to get started.

survMS v0.0.1: Provides functions to simulate data from the Accelerated Hazard, Accelerated Failure Time, and Cox survival models. See Bender et al. (2004) for the methods used to implement the Cox model, and the vignette and GitHub for an introduction and examples.

TestGardener v0.1.4: Provides functions to develop, evaluate, and score multiple choice examinations, psychological scales, questionnaires, and similar types of data involving sequences of choices among one or more sets of answers. See Ramsay et al. (2020) and Ramsay et al. (2019) for the methodology and the vignettes Symptom Distress Analysis and SweSAT Quantitative Analysis.

wpa v1.5.0: Provides opinionated functions to enable easier and faster analysis of Workplace Analytics data. See the vignette for an introduction.

Time Series

garchmodels v0.1.1: Implements a framework for using GARCH models with the tidymodels ecosystem. It includes both univariate and multivariate methods from the rugarch and rmgarch packages. There is a Getting Started Guide and a vignette on tuning univariate GARCH models.

tensorTS v0.1.1: Provides functions for estimating, simulating and predicting factor and autoregressive models for matrix and tensor valued time series. See Chen et al. (2020), Chen et al. (2020), and Han et al. (2020) for the math.

Utilities

diffmatchpatch v0.1.0: Implements a wrapper for Google’s diff-match-patch library. It provides basic tools for computing diffs, finding fuzzy matches, and constructing / applying patches to strings. See README for examples.

erify v0.2.0: Provides several validator functions to check if arguments passed by users have valid types, lengths, etc., and if not, to generate informative and good-formatted error messages in a consistent style. See the vignette to get started.

juicr v0.1: Provides a GUI interface for automating data extraction from multiple images containing scatter and bar plots, semi-automated tools to tinker with extraction attempts, and a fully-loaded point-and-click manual extractor with image zoom, calibrator, and classifier. See the vignette for examples, and the Youtube channel for a course on meta analysis.

mailmerge v 0.2.1: Allows users to mail merge using markdown documents and gmail, parse markdown documents as the body of email, use the yaml header to specify the subject line of the email, preview the email in the RStudio viewer pane, and send (draft) email using gmailr. See the vignette for examples.

m61r v0.0.2: Provides dplyr and tidyr like data manipulation functions using only base R and no dependencies. See the vignette for examples.

Visualization

flametree v0.1.2: Implements a generative art system for producing tree-like images using an L-system to create the structures. See README to get started.

leafdown v1.0.0: Provides drill down functionality for leaflet choropleths in shiny apps. There is an Introduction and a Showcase example.

mapping v1.2: Provides coordinates, linking and mapping functions for mapping workflows of different geographical statistical units. Geographical coordinates automatically link with the input data to generate maps. See the vignette to get started.

materialmodifier v1.0.0: Provides functions to apply image processing effects to modify the perceived material properties such as gloss, smoothness, and blemishes. Look here for documentation and practical tips of the package is available at

svplots v0.1.0: Implements two versions of sample variance plots illustrating the squared deviations from sample variance as described in Wijesuriya (2020). See the vignette.

vivid v0.1.0: Provides a suite of plots for displaying variable importance and two-way variable interaction. Plots include partial dependence plots laid out in “pairs plot”” or zenplots style. There is an Introduction and a Quick Start Guide.

Introduction to Functional Data Analysis with R

Tue, 04 May 2021 00:00:00 +0000

Suppose you have data that looks something like this.

This plot might depict 80 measurements for a participant in a clinical trial where each data point represents the change in the level of some protein level. Or it could represent any series of longitudinal data where the measurements are taken at irregular intervals. The curve looks like a time series with obvious correlations among the points, but there are not enough measurements to model the data with the usual time series methods. In a scenario like this, you might find Functional Data Analysis (FDA) to be a viable alternative to the usual multi-level, mixed model approach.

This post is meant to be a “gentle” introduction to doing FDA with R for someone who is totally new to the subject. I’ll show some “first steps” code, but most of the post will be about providing background and motivation for looking into FDA. I will also point out some of the available resources that a newcommer to FDA should find helpful.

FDA is a branch of statistics that deals with data that can be conceptualized as a function of an underlying, continuous variable. The data in FDA are smooth curves (or surfaces) in time or space. To fix a mental model of this idea, first consider an ordinary time series. For example, you might think of the daily closing prices of your favorite stock. The data that make up a time series are the individual points which are considered to be random draws from an underlying stochastic process.

Now, go up a level of abstraction, and consider a space where the whole time series, or rather an imaginary continuous curve that runs through all of your data points is the basic item of analysis. In this conceptual model, the curve comprises an infinite number of points, not just the few you observed. Moreover, unlike in basic time series analysis, the observed points do not need to be equally spaced, and the various curves that make up your data set do not need to be sampled at the same time points.

Mathematically, the curves are modeled as functions that live in an infinite dimensional vector space, what the mathematicians call a Hilbert Space. One way to think of this is that you are dealing with the ultimate large p small n problem. Each curve has infinitely many points, not just the 3 or 30 or 3,000 you happen to have.

The theory of Hilbert Spaces is part of the area of mathematical analysis called Functional Analysis, a subject usually introduced as part of a second or third course in mathematical analysis, or perhaps in a course on Quantum Mechanics.

It would be a heavy lift to expect someone new to Functional Data Analysis to start with the mathematics. Fortunately, this is not really necessary. The practical applications of FDA and the necessary supporting software have been sufficiently developed so that anyone familiar with the basics of ordinary vector spaces should have sufficient background to get started. The salient points to remember are:

Hilbert space is an infinite dimensional linear vector space
The vectors in Hilbert space are functions
The inner product of two functions in the Hilbert space is defined as the integral of two functions, but it behaves very much like the familiar dot product.

Moreover, for the last twenty years or so mathematical statisticians have been writing R packages to put FDA within the reach of anyone with motivation and minimal R skills. The CRAN Task View on Functional Data Analysis categorizes and provides brief explanations for forty packages that collectively cover most of the established work on FDA. The following graph built with functions from the cranly package shows part of the network for two core FDA packages.

The fda package emphasized in the network plot above is the logical place for an R user to begin investigating FDA. With thirty-two reverse depends, thirty-eight reverse imports and thirteen reverse suggest, fda is at the root of Functional Data Analysis software for R. Moreover, in a very real sense, it is at the root of modern FDA itself. fda was written to explicate the theory developed in the 2005 book by Ramsay and Silverman$^{1}$. Kokoszka and Reimnerr state that the first edition of this book published in 1997: “is largely credited with solidifying FDA as an official subbranch of statistics” (p xiv)$^{2}$. The refund package is used extensively throughout the book by Kokoszka and Reimnerr.

First Steps

The synthetic data in the figure above were generated by a Wiener, Brownian Motion process, which for the purposes of this post, is just a convenient way to generate a variety of reasonable looking curves. We suppose that the data points shown represent noisy observations generated by a smooth curve f(t). We estimate this curve with the model: $y_{i} = f(t_{i}) + \epsilon_{i}$ where the $\epsilon_{i}$ are normally distributed with mean 0 and variance $\sigma^{2}$.

Notice that the measurement times are randomly selected within the 100 day window and not uniformly spaced.

set.seed(999)
n_obs <- 80
time_span <- 100
time <- sort(runif(n_obs,0,time_span))
Wiener <- cumsum(rnorm(n_obs)) / sqrt(n_obs)
y_obs <- Wiener + rnorm(n_obs,0,.05)

Remember that the task ahead is to represent the entire curve of infinitely many points and not just the handful of observed values. Here is where the linear algebra comes in. The curve is treated as a vector in an infinite dimensional vector space, and what we want is something that will serve as a basis for this curve projected down into the subspace where the measurements live. The standard way to do this for non-periodic data is to construct a B-spline basis. (B-splines or basis splines are splines designed to have properties that make them suitable for representing vectors.) The code that follows is mostly borrowed from Jiguo Cao’s Youtube Video Course$^{3}$ which I very highly recommend for anyone just starting with FDA. In his first five videos, Cao explains B-splines and the placement of knots in great detail and derives the formula used in the code to calculates the number of basis elements from the number of knots and the order of the splines.

Note that we are placing the knots at times equally spaced over the 100 day time span.

times_basis = seq(0,time_span,1)
knots    = c(seq(0,time_span,5)) #Location of knots
n_knots   = length(knots) #Number of knots
n_order   = 4 # order of basis functions: cubic bspline: order = 3 + 1
n_basis   = length(knots) + n_order - 2;
basis = create.bspline.basis(c(min(times_basis),max(times_basis)),n_basis,n_order,knots)
n_basis

## [1] 23

and there are 23 basis vectors.

Next, we use the function eval.basis() to evaluate the basis functions at the times where our data curve was observed The matrix PHI contains the values of the 23 basis functions $\phi_j(t)$ evaluated at 80 points.

PHI = eval.basis(time, basis) 
dim(PHI)

## [1] 80 23

We plot the basis functions and locations of the knots.

matplot(time,PHI,type='l',lwd=1,lty=1, xlab='time',ylab='basis',cex.lab=1,cex.axis=1)
for (i in 1:n_knots)
{
  abline(v=knots[i], lty=2, lwd=1)
}

The plot shows that for interior points, four basis functions contribute to computing the value of any point. The endpoints, however, are computed from a single basis function.

Estimating the Basis Coefficients

As in ordinary regression, we express the function in terms of the coefficients $c_j$ and basis functions $\phi_j$ using the formula: $f(t) = \sum c_j \phi_j(t)$. Later we will see how to use built-in fda functions to estimate the coefficients, but now we follow Cao’s lead and calculate everything from first principles.

The following code uses matrix least squares equation $\hat{c} = (\Phi^t\Phi)^{-1} \Phi^{t}y$ to estimate the coefficients.

# Least squares estimate
# estimate basis coefficient
M = ginv(t(PHI) %*% PHI) %*% t(PHI)
c_hat = M %*% Wiener

We compute $\hat{y}$, the estimates of our observed values, and plot.

y_hat = PHI %*% c_hat
# Augment data frame for plotting
df <- df %>% mutate(y_hat = y_hat)
p2 <- df %>% ggplot() + 
      geom_line(aes(x = time, y = Wiener), col = "grey") +
      geom_point(aes(x = time, y = y_obs)) +
      geom_line(aes(x = time, y = y_hat), col = "red")
p2 + ggtitle("Original curve and least squares estimate") + 
      xlab("time") + ylab("f(time)")

The gray curve in the plot represents the underlying Brownian motion process, the dots are the observed values (the same as in the first plot), and the red curve represents the least squares “smoothed” estimates.

Now, we work through the matrix calculations to estimate the variance of the noise and the error bars for $\hat{y}$.

# estimate the variance of noise
## SSE = (Y - Xb)'(Y - Xb)
SSE = t(y_hat-y_obs)%*%(y_hat-y_obs)
sigma2 = SSE/(n_obs-n_basis)

# estimate the variance of the fitted curve
# H is the Hat matrix H
# H = X*inv(X'X)*X``
H = PHI %*% M
varYhat = diag(H %*% H * matrix(sigma2,n_obs,n_obs))

# 95% confidence interval

y_hat025 = y_hat-1.96*sqrt(varYhat)
y_hat975 = y_hat+1.96*sqrt(varYhat)

And, we plot. We have a satisfying smoothed representation of our original curve that looks like it would be and adequate starting point for further study. Note that process of using regression to produce a curve from the basis functions is often referred to as “regression smoothing”

df <- mutate(df, y_hat025 = y_hat025,
                 y_hat975 = y_hat975)
#names(df) <- c("time","Wiener","y_hat", "y_hat025", "y_hat975")
p3 <- df %>% ggplot() + 
      geom_line(aes(x = time, y = Wiener), col = "grey") +
      geom_point(aes(x = time, y = y_obs)) +
      geom_line(aes(x = time, y = y_hat), col = "red") +
      geom_line(aes(x = time, y_hat025), col = "green") +
      geom_line(aes(x = time, y_hat975), col = "green") 
p3 + ggtitle("Estimated curve with error bars") + 
     xlab("time") + ylab("f(time)")

We finish for today, by showing how to do the hard work of estimating coefficients and function values with a single line of code using the fda function smooth.basis(). The function takes the arguments argvals the times we want to use for evaluation as a vector (or matrix or array), y the observed values, and fdParobj, an fda object containing the basis elements.

Wiener_obj <- smooth.basis(argvals = time, y = y_obs, fdParobj = basis)

Here we plot our “hand calculated” curve in red and show the smooth.basis() curve in blue. They are reasonably close, except at the end points, where there is not much data to construct the basis.

plot(time, Wiener, type = "l", xlab = "time", ylab = "f(time)", 
     main = "Comparison of fda package and naive smoothing estimates", col = "grey")
lines(time,y_hat,type = "l",col="red")
lines(Wiener_obj, lwd = 1, col = "blue")

Note that we have shown the simplest use of smooth.basis() which is capable of computing penalized regression estimates and more. The examples of using the smooth.basis() function in the fda pdf are extensive and worth multiple blog posts. In general, the pdf level documentation for fda is superb. However, the package lacks vignettes. For a price, the book Functional Data Analysis with R and Matlab$^{4}$ supplies the equivalent of several the missing vignettes.

Next Steps

Once you have a basis representation, what’s next? You may be interested in the following:

More exploratory work such as Functional Principal Components Analysis, the analog of principal components analysis.
Clustering curves. See the funHDDC package.
Setting up regression models where either the dependent variable, or some of the independent variables, or both are functional objects. See the refund package and the book by Kokoszka and Reimnerr$^{2}$
Studying the shape of the curves themselves. For example, the shape of a protein concentration curve may convey some clinical meaning. FDA permits studying the velocity and acceleration of curves, offering the possibility of obtaining more information than the standard practice of looking at the area under the curves. You can explore this with the fda package (But be sure to check that the order of your basis functions is adequate to compute derivatives.). See the book by Ramsay, Hooker and Graves$^{4}$.
Learning what to do when you have sparse data. See the paper by Yao et al. below$^{5}$ and look into the fdapace package.
Working with two and three dimensional medical images$^{6}$.

I would like to make Functional Data Analysis a regular feature on R Views. If you are working with FDA and would like to post, please let me (joseph.rickert@rstudio.com) know.

References

Books

$^{2}$Kokoszka, P. and Reimherr, M. (2017). Introduction to Functional Data Analysis. CRC.
$^{1}$Ramsay, J.O. and Silverman, B.W. (2005). Functional Data Analysis. Springer.
$^{4}$Ramsay, J.0., Hooker, G. and Graves, S. (2009) Functional Data Analysis with R and MATLAB Springer.

Online Resources

$^{3}$Cao, J. (2019). Functional Data Analysis Course
Staicu, A. and Park, Y. (2016) Short Course on Applied Functional Data Analysis

March 2021: "Top 40" New CRAN Packages

Thu, 22 Apr 2021 00:00:00 +0000

By my count, two hundred twenty-one new packages stuck to CRAN in March 2021.¹ Here are my “Top 40” selections in twelve categories: Computational Methods, Data, Engineering, Genomics, Machine Learning, Medicine, Music, Networks, Science, Statistics, Utility, and Visualization. Two of these categories Engineering and Music have only one entry each. However, I decided to give them their own category in order to draw attention to the use of R outside of the mainstream, and I have always lamented the fate of the Miscellaneous. In the same spirit, note that the complete works of the Bard appear in the Data category and that due to tidypaleo Paleoenvironmental is now a thing in R.

Computational Methods

gamlss v1.0-5: Implements computationally intensive calculations for Generalized Additive Models for location, scale, and shape as described in Rigby & Stasinopoulos (2005).

waydown v1.1.0: Implements an algorithm based on the classical Helmholtz decomposition to obtain an approximate potential function for non gradient fields. See Rodríguez-Sánchez (2020) for background and the vignette for examples.

Data

aopdata v0.2.1: Provides functions to download data from the Access to Opportunities Project (AOP) which includes annual estimates of access to employment, health and education services by transport mode, as well as data on the spatial distribution of population, schools and health-care facilities at a fine spatial resolution for all cities included in the study. There is an Introduction to the package, and there are vignettes on Analyzing Inequality, Mapping Urban Accessibility, and Mapping Pooulation and Land Use.

bardr v0.0.9: Provides R data structures for Shakespeare’s complete works, as provided by Project Gutenberg. See README.

metro v0.9.1: Provides access to the Metro Transparent Data Sets API published by the Washington Metropolitan Area Transit Authority, the government agency operating light rail and passenger buses in the Washington D.C. area. See README.

RAQSAPI v2.0.1: Provides functions to retrieve air monitoring data and associated metadata from the US Environmental Protection Agency’s Air Quality System Service. There are several short vignettes including an Introduction and a vignette on Usage tips and precautions.

troopdata v0.1.3: Provides access to U.S. Department of Defense data on overseas military deployments and includes functions for pulling country-year troop deployment and basing data. See README to get started

Engineering

pipenostics v0.1.7: Implements empirical and data-driven models of heat losses, corrosion diagnostics, reliability and predictive maintenance of pipeline systems which should be of interest to the engineering departments of heat generating and heat transferring companies. See Timashev et al. (2016) and Reddy (2017) for the methods used and README to get started.

Genomics

glmmSeq v0.1.0: Provides functions to fit negative binomial mixed effects models with matched samples to model expression data. See the vignette for examples.

ondisc v1.0.0: Implements a method to allow researchers to analyze large-scale single-cell data as and R object stored on disk. There is a tutorial on the the ondisc matrix class and another on Metadata.

SignacX v2.2.0: Implements a neural network trained with flow-sorted gene expression data to classify cellular phenotypes in single cell RNA-sequencing data. See Chamberlain et al. (2021) for background. There are seven vignettes including an Analysis of Kidney Lupus Data and an Analysis of PBMCs from 10X Genomics.

Machine Learning

opitools v1.0.3: Implements a tool to analyze opinions inherent in a text document relating to a specific subject (A) and assess how opinions expressed with respect to another subject (B) may affect the opinions on subject A. This package has been designed specifically for application to social media datasets, such as Twitter and Facebook. See Adepeju and Jimoh (2021) for an extended example that demonstrates the utility of the approach and the vignette to get started.

poems v1.0.1: Provides a framework of interoperable R6 classes for building ensembles of viable models via the pattern-oriented modeling (POM) approach. The package includes classes for encapsulating and generating model parameters, and managing the POM workflow which includes: model setup; generating model parameters via Latin hyper-cube sampling; running multiple sampled model simulations; collating summary results; and validating and selecting an ensemble of models that best match known patterns. There are two vignettes: Simple Example and Thylacine Example.

Medicine

dampack v1.0.0: Implements a suite of functions for analyzing and visualizing the health economic outputs of mathematical models. See Hunink et al. (2014) for the theoretical underpinnings. There are five vignettes including Basic Cost Effectiveness Analysis, Probabilistic Sensitivity Analysis: Analysis and Value of Information Analysis.

rdecision v1.0.3: Provides classes and functions for using decision trees to model health care interventions using cohort models. See Briggs et al. for theory and terminology. There are five vignettes including Elementary decision tree (Evans 1997) and Decision tree with PSA.

Music

gm v1.0.2: Implements a high-level language to create music including converting your music to musical scores and audio files. It works with R Markdown, R Jupyter Notebooks, and RStudio. There vignette is available in English and in Chinese.

Networks

sfnetworks v0.5.1: Provides a tidy approach to spatial network analysis in the form of classes and functions that enable a seamless interaction between the network analysis package tidygraph and the spatial analysis package sf. There are vignettes on sf network structure, Preprocessing, Spatial joins and filters, Routing, and Spatial morphers.

valhallr v0.1.0: Implements an interface to the Valhalla routing engine’s API for turn-by-turn routing, isochrones, and origin-destination analyses. See the vignette for examples.

Science

asteRisk v0.99.4: Provides functions to calculate the positions of satellites given a known state vector. It includes implementations of the SGP4 and SDP4 simplified perturbation models to propagate orbital state vectors. See Hoots et al. (1988), Vallado et al. (2012), and Hoots et al. (2014) for background and the vignette for examples.

forImage v0.1.0: Implements a tool to measure the size of foraminifera and other unicellulars and includes functions to guide foraminiferal test biovolume calculations and cell biomass estimations. The volume function includes several microalgae models geometric adaptations based on Hillebrand et al. (1999), Sun & Liu (2003), and Vadrucci et al. (2007). See the vignette to get started.

OpenSpecy v0.9.1: Provides functions to analyze, process, identify and share Raman and (FT)IR spectra with functions to implement Savitzky-Golay smoothing in accordance with Zhao et al. (2007) and identify spectra using an onboard reference library, see Cowger et al. 2020. Analyzed spectra can be shared via Shiny App. There is a vignette.

tidypaleo v0.1.1: Provides functions with a common framework for age-depth model management, stratigraphic visualization, and common statistical transformations with a focus on stratigraphic visualization using ggplot2. There are vignettes on Age-depth Models, Nested Analyses, and Stratigraphic Diagrams.

VulnToolkit v1.1.2: Provides functions to analyze and summarize tidal data sets and to access to NOAA mean sea level data. See Hill & Anisfeld (2015) for background and the vignette for examples.

Statistics

corncob v0.2.0: Implements functions for modeling correlated count data using the beta-binomial distribution, described in Martin et al. (2020). See the vignette for an introduction.

hawkesbow v1.0.2: Implements an estimation method for Hawkes processes when count data are only observed in discrete time, using a spectral approach derived from the Bartlett spectrum. See Cheysson and Lang (2020) for background and the vignette for examples.

LMMELSM v0.1.0: Implements two-level mixed effects location scale models on multiple observed or latent outcomes, and between-group variance modeling. See Williams et al. (2020) and Hedeker et al. (2008) for background and README for an example.

mixpoissinreg v1.0.0: Provides functions to fit mixed Poisson regression models (Poisson-Inverse Gaussian or Negative-Binomial) with count data response variables. See Barreto-Souza and Simas (2016) for background. There are five vignettes on Global and Local Influence, Confidence and Prediction Intervals, MLE, Tidy Methods, and Overdispersed Count Data.

ppdiag v0.1.0: Provides a suite of diagnostic tools for univariate point processes including tools for simulating and fitting both common and more complex temporal point processes and the diagnostic tools described in Brown et al. (2002) and Wu et al. (2020). There is a vignette on Markov Modulated Point Processes and another on Diagnostic Tools.

robustlm v0.1.0: Implements a computationally efficient exponential squared loss algorithm for variable selection proposed by Wang et al.(2013). See the vignette.

smmR v1.0.2: Provides functions to estimate and simulate multi-state semi-Markov models. The methods implemented are described in Barbu & Limnios (2008) and Trevezas & Limnios (2011). The vignette contains an extended example.

spotoroo v0.1.1: Implements an algorithm to cluster satellite hot spot data spatially and temporally. See the vignette.

Utilities

clock v0.2.0: Provides a comprehensive library for date-time manipulations using a new family of orthogonal date-time classes (duration, time points, zoned-times, and calendars) that partition responsibilities so that the complexities of time zones are only considered when they are really needed. There is a Getting Started guide, as well as vignettes on FAQ, and Examples and Recipies.

crosstable v0.2.1: Provides functions to create descriptive tables for continuous and categorical variables, apply summary statistics, and create reports using rmarkdown or officer. There is an Introduction, and vignettes on Troubleshooting, Making Automatic Reports, and Selecting Variables.

pkgdepends v0.1.0: Provides functions to find recursive dependencies for R packages from various sources including CRAN, Bioconductor, and GitHub enabling users to obtain a consistent set of packages to install. See README to get started.

pkglite v0.1.1: Implements a tool, grammar, and standard to represent and exchange R package source code as text files. Converts one or more source packages to a text file and restores the package structures from the file. There are vignettes on Generating File Specifications, Representing Packages, and Compact Package Representation.

Visualization

datplot v1.0.0: Provides tools to process and prepare data for visualization and employs the concept of aoristic analysis. See aorist and the vignettes Data Preparation and Visualization and Visualizing Chronological Distribution.

ferrn v0.0.1: Implements diagnostic plots for optimization, with a focus on projection pursuit which show paths the optimizer takes in the high-dimensional space. See README for examples.

funcharts v1.0.0: Provides functional control charts for statistical process monitoring of functional data, using the methods of Capezza et al. (2020) and Centofanti et al. (2020). There are vignettes on Capezza 2020, Centofanti 2020 and on the mfd class.

gghilbertstrings v0.3.3: Provides functions to plot Hilbert curves which are used to map one dimensional data into the 2D plane. A specific use case maps a character column in a data frame into 2D space allowing visually comparing long lists of URLs, words, genes or other data that has a fixed order and position. See README for examples.

mapsf v0.1.1: Provides functions to create and integrate thematic maps including functions to design various cartographic representations such as proportional symbols, choropleth or typology maps. Look here for examples.

¹ I have used phrases like By my count and stuck to CRAN in the past, but I do not believe that I have explained what I mean. For some time now, but I believe more frequently in recent months, packages will appear as new on CRAN, only to be removed within a relatively short period of time for failing to resolve check problems. If you happen to know about these packages and search for them by name on CRAN you will receive the message:

Package XXXX was removed from the CRAN repository. Formerly available versions can be obtained from the archive. Archived on 2021-04-17 as check problems remained after update. A summary of the most recent check results can be obtained from the check results archive. Please use the canonical form https://CRAN.R-project.org/package=XXXX to link to this page.

I did not include the ten packages that were identified as being new for March when I created my list of March packages on April 10, 2021, but were removed by the time I finalized my list for this post a week later, in my total count of new CRAN packages. So, there is some instability with the notion of counting new packages in a given month.

COVID-19 Data Forum: Data Journalism

Tue, 06 Apr 2021 00:00:00 +0000

The COVID-19 Data Forum, a joint project of the Stanford Data Science Institute and the R Consortium, is an ongoing series of multidisciplinary webinars where topic experts discuss data-related aspects of the scientific response to the pandemic. The most recent event, held on March 18, 2021, explored the role of data journalism in the pandemic. This was a bit of a departure from previous forum events¹ because it focused on issues relating to using and interpreting COVID-19 data, and not on the particular kinds of COVID-19 related data that are available.

I think you will find the webinar video worth watching. If you are a statistician or epidemiologist working on COVID-19, you may find the data journalists’ accounts of difficulties they faced working with COVID data and statistical models instructive. But, even if you are not directly working on COVID, you may find that listening to the journalists fills in some gaps between what you know about statistics and data visualizations and what you see in the news.

The data journalism event was moderated by Dr. Irena Hwang, a data reporter at ProPublica. Speakers included Dr. Mark Hansen, David and Helen Gurley Brown Professor of Journalism and Innovation at Columbia University; Ana Carolina Moreno, a senior data journalist at TV Globo in São Paulo, Brazil; and Meghan Hoyer, Director of Data Reporting at the Washington Post.

The video of the data journalism event is available here. The following short time map and the times referenced in my comments below should be helpful for browsing the ninety minute event.

2:37 Irena Hwang introduces Mark Hansen
3:50 Start of Mark’s talk
19:30 Irena introduces Ana Carolina Moreno (Carol)
21:10 Start of Carol’s talk
39:20 Irena introduces Meghan Hoyer
40:00 Start of Meghan’s talk
1:01:40 Start of discussion

Mark Hansen

In his talk, Mark offers an overview of the profession of data journalism that provides some historical context and emphasizes the hybrid nature of the practice which blends a hard nose detective’s drive to uncover facts with the empathy to tell stories “about who we are and how we live”.

7:00 Mark introduces Joseph Pulitzer’s 1904 paper The College of Journalism in which Pulitzer includes Statistics as a subject journalists should study. On page 673, Pulitzer writes:

You want statistics to tell you the truth. You can find truth there if you know how to get at it, and romance, human interest, humor and fascinating revelations as well.

10:19 Mark describes a piece, An ode to reporter’s notebooks, published by Philip Eil in the Columbia Journalism Review that offers a personal account of reporting: Eil writes:

To report is to be alert and alive at a particular time and place.

11:00 Mark remarks:

when we’re thinking about bringing computation to journalism we are taking that basic curiosity that we are cultivating in our students minds … and adding computational lines of inquiry to that habit of mind, that questioning why things look the way they do…

12:08 Mark calls attention to the report by Charles Berret and Cheryl Phillips Teaching Data And Computational Journalism and describes some recent activities of the Brown Institute at the Columbia School of Journalism.

Ana Carolina Moreno

22:32 Carol introduces Brazil’s universal healthcare system and shows a schematic of the available official and unofficial COVID-19 data sources.

26:00 Carol notes that a platform originally built to track SARS data was adapted to track COVID.

27:38 Carol explains that, in practice, there are many obstacles making it difficult to obtain the data necessary to understand how the pandemic is developing. Some of these are called out in the following slide:

30:37 Carol remarks that hospital data seems to be the most reliable.

31:06 Carol describes how the government changed its policy for reporting deaths. The new scheme of only reporting deaths that have been confirmed in the past twenty-four hours vastly undercounts the current death rate.

31:57 In an effort to obtain more reliable data, a consortium of competing journalists at local news organizations began cooperating by sharing information directly obtained from hospitals every day.

32:35 Carol provides a view of day-to-day journalism at the local news organizations and describes how the data journalist scrape data on a daily basis to populate dashboards showing rolling averages and daily indicators. By focusing on the more reliable hospitalization data journalists are doing their best to track the spread of the pandemic an expose inequities in the health care system.

Meghan Hoyer

40:07 Meghan begins her walk through of what last year was like for data journalists who were trying to tell the story of the pandemic in real time as it was happening.

41:22 Meghan recounts her experiences trying to make sense of COVID-19 models and expresses the frustration she and other data journalists felt with the multitude of contradictory predictive models.

44:03 In a memorable quote, Meghan remarks:

Models were inherently problematic and yet they were being forced upon us by society…

Consequently journalists at the AP agreed and decided that they were not going to base stories on models.

In absence of reliable case data, and wanting nothing to do with the models, Meghan explains that data journalists turned to whatever data they could get their hands on to quantify the story of the pandemic.

46:00 Meghan recounts how journalists used garbage pickup data as a proxy for population density to estimate where people were living in NYC and correlate it with case data.

47:30 Journalists struggled to find data to verify the anecdotal stories they were hearing about the the disparities in who was being affected by virus. Finding that one quarter to one third of the COVID case data was missing information on race, data journalists “hand collected” data by looking city by city to find the missing data.

50:30 Meghan recounts how they turned to age adjusted data to determine the impact of the virus on communities of color.

52:10 Data journalists find that excess deaths is a reliable metric for determining the impact of what is happening on the ground.

54:18 Journalists developed a survey which was returned by seven hundred schools to investigate how going back to school might be affecting students. Among their findings was that districts serving students of color were more likely to start online.

56:35 Meghan discusses the COVID-19 Tracking Project and the effort to sort out the impact of test positivity rates. She reports that because not all states measure the number of people who test in the same way, correctly comparing test positivity rates among states remains an unsolved problem.

58:33 Meghan shares the need to “flip the numbers” to help people understand the meaning of statistics stated in terms of very large numbers. For example, saying that “Since January of last year at least 1 in 15 people who live in Alexandria, Virginia have been infected by the virus” is easier for people to understand than something like: “On March 17th there were 14 cases per 100,000 in Alexandria”.

59:53 Vaccination tracking is another problematic data reporting area. Not only are vaccinations reported differently from state-to-state, but the data that is reported is changing from day-to-day. The CDC is apparently still adding new fields to the vaccination data sets.

The Q & A Discussion

1:02 The question and answer discussion begins.

1:02:56 Mark talks about how visualizations evolved over the course of the pandemic.

1:06:08 Carol and then Meghan talk how the lessons the pandemic taught data journalists about competition and collaboration.

1:10:04 Meghan describes how during the pandemic data journalists became advocates for public data.

1:11:21 Carol answers a question about the opportunities for data journalism in Brazil.

1:15:50 Answers a question of how academia is supporting data journalism during the pandemic and mentions an effort to have statistical and scientific experts collaborate with data journalists.

1:20:19 Meghan responds to a question about technical and social challenges for data journalists during the pandemic.

1:23:10 Carol talks about the difference between reporting online news and television news.

1:26:01 Mark answers a question about communicating emotional impact in COVID reporting and ends with emphasizing the importance of communicating honestly about what we do, and do not know.

¹The first forum on May 14, 2020 focused on the data needs and challenges of modeling and controlling the spread of COVID-19, The second forum on August 13, 2020 explored what was being done to make clinical data available and useful. The third forum on December 10, 2020 discussed the role of mobility data.

February 2021: "Top 40" New CRAN Packages

Fri, 19 Mar 2021 00:00:00 +0000

In February, two hundred forty-three new packages made it to CRAN, many of them very interesting and at least one entertaining. It was exceptionally difficult to pick the “Top 40”, but here they are, more or less, in eleven categories: Computational Methods, Data, Finance, Games, Genomics, Machine Learning, Mathematics, Medicine, Networks and Graphs, Statistics, Utilities, and Visualization. iconr in the Networks and Graphs section is a package for doing computational archaeology, a relatively new field that I hope will dig R. I also hope that sassy in the Statistics sections helps some statisticians find their way to R.

Computational Methods

blaster v1.0.3: Implements an efficient BLAST-like sequence comparison algorithm, written in C++11 and using native R data types. See Schmid et al. (2018) for background and README for an example.

rando v0.2.0: Provides random number generating functions that are much more context aware than the built-in functions. The functions are also safer, as they check for incompatible values, and reproducible.

Data

AWAPer 0.1.46: Provides catchment area weighted climate data NetCDF files from the Bureau of Meteorology Australian Water Availability Project for all of Australia. There is a vignette on Daily Area Weighted PET and Precipitation and another on Daily Point Precipitation

caRecall v0.1.0: Provides API access to the Government of Canada Vehicle Recalls Database used by the Defect Investigations and Recalls Division for vehicles, tires, and child car seats. See the vignette.

geofi v1.0.0: Provides tools for reading Finnish open geospatial data in R. There are vignettes on Datasets, Joining Attributes, Making Maps, Data Manipulation, and Color-coded Maps.

hockeystick v0.4.0: Provides easy access to essential climate change data sets for non-climate experts. Users can download the latest raw data from authoritative sources and view it via pre-defined ggplot2 charts. Data sets include atmospheric CO2, instrumental and proxy temperature records, sea levels, Arctic/Antarctic sea-ice, and Paleoclimate data. Sources include: NOAA Mauna Loa Laboratory, NASA GISTEMP, National Snow and Sea Ice Data Center, CSIRO, NOAA Laboratory for Satellite Altimetry, and Vostok Paleo carbon dioxide and temperature data. See README for examples.

votesmart v0.1.0: Implements a wrapper to the Project VoteSmart API. See the vignette.

Finance

PriceIndices v0.0.3: Provides functions to compute bilateral and multilateral indexes. For details, see: de Haan and Krsinich (2017) and Diewert and Fox (2020). The vignette offers examples.

treasuryTR v0.1.1: Generates Total Returns (TR) from bond yield data with fixed maturity (e.g. reported treasury yields) which may provide an alternative to commercial products. See Swinkels (2019) for background and the vignette for examples.

Games

pixelpuzzle v1.0.0: Implements a puzzle game that can be played in the R console. Restore the pixel art by shifting rows. Learn how to play here.

Genomics

CDSeq v1.0.8: Provides functions to estimate cell-type-specific gene expression profiles and sample-specific cell-type proportions simultaneously using bulk sequencing data. See Kang et al. (2019) for the theory and the vignette for examples.

ClusTorus v0.0.1: Provides various tools for clustering multivariate angular data on the torus including angular adaptations of usual clustering methods such as the k-means clustering, pairwise angular distances. See the vignette for examples.

dsb v0.1.0: Provides a method for normalizing and denoising protein expression data from droplet based single cell experiments. See the vignette for tutorials on how to integrate dsb with Seurat, Bioconductor and the AnnData class in Python. The preprint Mulè et al. (2020) describes the details.

Machine Learning

besridge v1.0.4: Provides functions to perform ridge regression in complex situations on high dimensional data using the primal dual active set algorithm proposed in Wen et al. (2020). Functions support regression, classification, count regression and censored regression, group variable selection and nuisance variable selection. See the vignette for examples.

ROCket v1.0.1: Provides functions for estimating receiver operating characteristic (ROC) curves and area under the curve (AUC) calculation which distinguish two types of ROC curve representations: 1) parametric curves - the true positive rate (TPR) and the false positive rate (FPR) are functions of a score parameter and 2) function curves - TPR is a function of FPR. See Gonçalves et al. (2014) and Cai & Pepe (2004) for background and README to get started.

wordpiece v1.0.2: Provides functions to apply Wordpiece tokenization to input text, given an appropriate vocabulary. The BERT tokenization conventions are used by default. See the vignette for an example.

Mathematics

fractD v0.1.0: Estimates the of fractal dimension of a black area in 2D and 3D (slices) images using the box-counting method. See Klinkenberg (1994) for background and the vignette for examples.

spacefillr v0.2.0: Generates random and quasi-random space-filling sequences including Halton, Sobol and other sequences with errors distributed as various types of jittered blue noise. See Joe and Kuo (2018), Christensen et al. (2018) and Heitz et al. (2019) for background and look here for examples.

tensorsign v0.1.0: Provides an efficient algorithm for nonparametric tensor completion via sign series. The algorithm which employs the alternating optimization approach to solve the weighted classification problem is described in Lee and Wang (2021)

Medicine

bp v1.0.1: Provides functions to aid in the analysis of blood pressure data of all forms by providing both descriptive and visualization tools for researchers. There is a vignette.

CHOIRBM v0.0.2: Provides functions for visualizing body map data collected with the Collaborative Health Outcomes Information Registry CHOIR). See the vignette.

QDiabetes v1.0-2: Calculates the risk of developing type 2 diabetes using risk prediction algorithms derived by ClinRisk. Look here for information and examples.

Networks and Graphs

bnmonitor v0.1.0. Implements sensitivity and robustness methods for Bayesian networks including methods to perform parameter variations via a variety of co-variation schemes, to compute sensitivity functions and to quantify the dissimilarity of two Bayesian networks via distances and divergences. See Chan and Darwiche (2002), Cowell et al. (2007), and Goergen and Leonell (2020) for background and README for examples.

iconr v0.1.0: Provides formal methods for studying archaeological iconographic data sets (rock-art, pottery decoration, stelae, etc.) using network and spatial analysis See Alexander (2008) and Huet (2018) for background and the vignette for examples.

MLVSBM 0.2.1: Provides functions for simulation, inference and clustering of multilevel networks using a stochastic block model framework as described in Chabert-Liddell et al. (2021). There is a tutorial.

motifr v1.0.0: Provides tools to analyze motifs(small configurations of nodes and edges) in multi-level networks (networks which combine multiple networks in one, e.g. social-ecological networks.) See The motif zoo and Baseline model comparisons.

Statistics

cfda v0.9.9: Provides functions to encode categorical data as functional data and perform basis statistical analysis. See Preda et al. (2020) for background and the vignette to get started.

cvCovEst v0.3.4: Implements an efficient cross-validated approach for covariance matrix estimation, particularly useful in high-dimensional settings. See the vignette for background and examples.

flipr v0.2.1: Implements a permutation framework point estimation, confidence intervals or hypothesis testing for multiple data types. There is a Tour of Permutation Inference, and vignettes on Alternative Hypothesis Testing, the Exactness of Permutation Tests, and Calculating p-value Functions.

ipmr v0.0.1: implements integral projection models using an expression based framework that handles density dependence and environmental stochasticity and provides tools for diagnostics, plotting, simulations, and analysis. See Easterling et al. (2000) for an in depth description of integral projection models. There is an Introduction and vignettes on Age-Size IPMS, Density Dependent IPMS, Hierarchical Notation, and Data Structures.

metapack v0.1.1: Provides functions performing Bayesian inference for meta-analytic and network meta-analytic models through Markov chain Monte Carlo algorithm. See Yao et al. (2015) for the theory, the vignette for an introduction and the online documentation.

sassy v1.0.4: Loads a collection of packages that collectively aim to make R easier for SAS® programmers. Functions bring many familiar SAS® concepts to R, including data libraries, data dictionaries, formats and format catalogs, a data step, and a traceable log. There is an Introduction, and vignettes with example Figures, Listings, and Tables, as well as a few Disclaimers which include a statement indicating that the packages were developed in the context of the pharmaceutical industry but should be generally helpful.

Utilities

gargoyle v0.0.1: Implements an event-Based framework for building Shiny apps. Instead of relying on standard Shiny reactive objects, this package allow to relying on a lighter set of triggers, so that reactive contexts can be invalidated with more control. See the vignette.

multidplyr Provides simple multicore parallelism through functions that partition a data frame across multiple worker processes. See the vignette.

quarto v0.1: Provides an interface to the Quarto markdown publishing system and allows converting R Markdown documents and Jupyter Notebooks to a variety of output formats.

var v0.0.2: Provides functions to manage, provision and use virtual machines pre-configured for R, and develop, test and build package in a clean environment. Vagrant and a provider such as Virtualbox must be installed.

Visualization

ggh4x v0.1.2.1: Extends ggplot2 facets by setting individual scales per panel, resizing panels, providing nested facets, and allowing multiple colour and fill scales per plot. See the Introduction, and the vignettes Facets, Misc, Position Guides, and Statistics.

tastypie v0.0.3: Provides functions and templates for making pie charts even though you probably shouldn’t. See the vignettes available templates and Your favorite template, and look here for examples.

terrainr v0.3.1: Provides functions to retrieve, manipulate, and visualize geospatial data, with an aim towards producing ‘3D’ landscape visualizations in the Unity 3D rendering engine. Functions are also provided for retrieving elevation data and base map tiles from the USGS National Map. There is an Introduction and a vignette on importing terrain tiles.

Cheat Sheets

Wed, 10 Mar 2021 00:00:00 +0000

In a previous post, I described how I was captivated by the virtual landscape imagined by the RStudio education team while looking for resources on the RStudio website. In this post, I’ll take a look at Cheatsheets another amazing resource hiding in plain sight.

Apparently, some time ago when I wasn’t paying much attention, cheat sheets evolved from the home made study notes of students with highly refined visual cognitive skills, but a relatively poor grasp of algebra or history or whatever to an essential software learning tool. I don’t know how this happened in general, but master cheat sheet artist Garrett Grolemund has passed along some of the lore of the cheat sheet at RStudio. Garrett writes:

One day I put two and two together and realized that our Winston Chang, who I had known for a couple of years, was the same “W Chang” that made the LaTex cheatsheet that I’d used throughout grad school. It inspired me to do something similarly useful, so I tried my hand at making a cheatsheet for Winston and Joe’s Shiny package. The Shiny cheatsheet ended up being the first of many. A funny thing about the first cheatsheet is that I was working next to Hadley at a co-working space when I made it. In the time it took me to put together the cheatsheet, he wrote the entire first version of the tidyr package from scratch.

It is now hard to imagine getting by without cheat sheets. It seems as if they are becoming expected adjunct to the documentation. But, as Garret explains in the README for the cheat sheets GitHub repository, they are not documentation!

RStudio cheat sheets are not meant to be text or documentation! They are scannable visual aids that use layout and visual mnemonics to help people zoom to the functions they need. … Cheat sheets fall squarely on the human-facing side of software design.

Cheat sheets live in the space where human factors engineering gets a boost from artistic design. If R packages were airplanes then pilots would want cheat sheets to help them master the controls.

The RStudio site contains sixteen RStudio produced cheat sheets and nearly forty contributed efforts, some of which are displayed in the graphic above. The Data Transformation cheat sheet is a classic example of a straightforward mnemonic tool. It is likely that even someone who just beginning to work with dplyr will immediately grok that it organizes functions that manipulate tidy data. The cognitive load then is to remember how functions are grouped by task. The cheat sheet offers a canonical set of classes: “manipulate cases”, “manipulate variables” etc. to facilitate the process. Users that work with dplyr on a regular basis will probably just need to glance at the cheat sheet after a relatively short time.

The Shiny cheat sheet is little more ambitious. It works on multiple levels and goes beyond categories to also suggest process and workflow.

The Apply functions cheat sheet takes on an even more difficult task. For most of us, internally visualizing multi-level data structures is difficult enough, imaging how data elements flow under transformations is a serious cognitive load. I for one, really appreciate the help.

Cheat sheets are immensely popular. And even in this ebook age where nearly everything you can look at is online, and conference attending digital natives travel light, the cheat sheets as artifacts retain considerable appeal. Not only are they useful tools and geek art (Take a look at cartography) for decorating a workplace, my guess is that they are perceived as runes of power enabling the cognoscenti to grasp essential knowledge and project it in the world.

When in-person conferences resume again, I fully expect the heavy paper copies to disappear soon after we put them out at the RStudio booth.

January 2020: "Top 40" New CRAN Packages

Wed, 24 Feb 2021 00:00:00 +0000

Two hundred thirty new packages made it to CRAN in January. Here are my “Top 40” selections in ten categories: Data, Finance, Genomics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Data

igoR v0.1.1: Provides tools to extract information from the Intergovernmental Organizations (‘IGO’) Database , version 3, provided by the Correlates of War Project. See Pevehouse et al. (2020) for information from 1815 to 2014, and get started with the vignette.

OTrecord v0.1.0: Uses optimal transportation theory as described in Gares, Guernec & Savy (2019) and Gares & Omer (2020) to solve recoding problems. Given two databases that share a subset of variables, package functions assist users in obtaining a unique synthetic database with complete information. See the vignette for examples.

pwt10 v10.0-0: Interfaces to the Penn World Table 10.x which provides information on relative levels of income, output, input, and productivity for 183 countries between 1950 and 2019.

trainR v0.0.1: Interfaces to the the National Rail Enquiries systems, including Darwin which provides real-time arrival and departure predictions, platform numbers, delay estimates, schedule changes and cancellations. Look here for examples.

Finance

LSMRealOptions v0.1.0: Provides an implementation of the least-squares Monte Carlo simulation method to value American option products and capital investment projects through real options analysis. Cash flows are modeled as being dependent upon underlying state variables that evolve stochastically. See the vignette for examples.

Genomics

AlleleShift v0.9-2: Provides methods for calibrating and predicting shifts in allele frequencies through redundancy analysis (vegan::rda()) and generalized additive models (mgcv::gam()) and functions to visualize the predicted changes in frequencies. See README for examples.

GenomeAdmixR v1.1.3: Provides tools to simulate how patterns in ancestry along the genome change after admixture. Se Janzen (2020) for the details and the vignettes Isofemales, Joyplot, Visualization, and Walkthrough.

MOSS v0.1.0: Implements an omics integration method based on sparse singular value decomposition to deal with the challenges of high dimensionality, noise and heterogeneity among samples and features in omics data. See (Gonzalez-Reymundez & Vazquez, 2020) for background and the vignette for examples.

Machine Learning

autoMrP v0.98: Implements a tool that improves the prediction performance of multilevel regression with post-stratification (MrP) by combining a number of machine learning methods. For information on the method, refer to Broniecki, Wüest, Leemann (2020) and the vignette.

aweSOM v1.1: Implements Self-organizing maps, a method for dimensionality reduction and clustering of continuous data, as well as interactive graphics to assist analysis. See Kohonen (2001) for background and the vignette for an overview of the package.

RandomForestsGLS v0.1.2: Fits non-linear generalized least square regression models with Random Forests as described in Saha, Basu & Datta (2020).

torchaudio v0.1.1.0: Provides access to datasets, models and preprocessing facilities for deep learning in audio. See the vignette.

vimpclust v0.1.0: Implements functions to perform sparse k-means clustering with a group penalty and variable selection on mixed categorical and numeric data. See Chavet et al. (2020) for background. There are vignettes on numeric and mixed data sparse k-means clustering.

Medicine

dataQuieR v1.0.4: Provides functions to assess data quality issues in studies. See the TMF Guideline and the DFG Project for background, and the vignette for examples.

Science

LPDynR v1.0.1: Implements methods that use phenological and productivity-related variables derived from time series of vegetation indexes to assess ecosystem dynamics. Functions compute an indicator with five classes of land productivity dynamics. Look here for background. See the vignette for an example.

rgee v1.0.8: Provides an Earth Engine client library for R that includes all Earth Engine API classes, modules, and functions, as well as additional functions for importing spatial objects, extracting time series, and displaying metadata and interactive maps. Look here for further details. Read the Introduction and the vignette on Best Practices to get started.

SAMtool v1.1.1: Provides tools for simulating the MSEtool operating model to inform data-rich fisheries. It includes a conditioning model, tools for assessing models of varying complexity and comparing models, and diagnostic tools for evaluating assessments inside closed-loop simulations. There is a User Guide and a series of seven more vignettes including an overview of the Rapid Conditioning Model (RCM) for conditioning MSEtool operating models, and a mathematical description of RCM.

Statistics

circularEV v0.1.0: Provides functions for performing extreme value analysis on a circular domain. See the local methods example and the spline example.

ghcm v1.0.0: Implements a statistical hypothesis test for conditional independence which can be applied to both discretely observed functional data and multivariate data. See Lundborg et al. (2020) for details and the vignette for an overview of the generalized Hilbert Covariance measure with examples.

gplite v0.11.1: Implements the most common Gaussian process models using Laplace and expectation propagation approximations, maximum marginal likelihood inference for the hyperparameters, and sparse approximations for larger datasets. See the vignette for a quick start.

multibridge v1.0.0: Implements functions to evaluate hypotheses concerning the distribution of multinomial proportions using bridge sampling. Functions are able to compute Bayes factors for hypotheses that entail inequality constraints, equality constraints, free parameters, and mixtures of all three. See Sarafoglou et al. (2020) for background and the examples: Memory of Lifestresses, Mendelian Laws of Inheritance, and Prevalence of Statistical Reporting Errors.

partR2 v0.9.1: Provides functions to to partition the variance explained in generalized linear mixed models (GLMMs) into variation unique to predicators and variation shared among predictors. This can be done using semi-partial R² and inclusive R². See Nakagawa & Schielzeth (2013) and Nakagawa, Johnson & Schielzeth (2017) for the theory and the vignette for examples.

spNetwork v0.1.1: Provides tools to perform spatial analysis on network including estimating network kernel density, building spatial matrices. See Okabe et al. (2019) for background and the vignettes: Network k Functions, Network Kernel Density Estimate, Details about NKDE, and Spatial Weight Matrices.

ubms v1.0.2: Provides functions to fit Bayesian hierarchical models, including single-season occupancy, dynamic occupancy, and N-mixture abundance models, of animal abundance and occurrence with the rstan package. See Carpenter et al. (2017) and Fiske and Chandler (2011) for background. There is a package Overview, a vignette on Random Effects, and another on Comparing ubms with JAGS.

Time Series

autostsm v1.2: Provides functions to automate the decomposition of structural time series into trend, cycle, and seasonal components using the Kalman filter. See Koopman et al. (2012) for the theory and the vignette for an overview with examples.

bayesforecast v0.0.1: Provides functions to fit Bayesian time series models using Stan for full Bayesian inference. It includes seasonal ARIMA, ARIMAX, dynamic harmonic regression, GARCH, t-student innovation GARCH models, asymmetric GARCH, random walks, stochastic volatility models for univariate time series. See Hyndman (2017) and Carpenter et al. (2017) for background and README for examples.

Utilities

autoharp v0.0.5: Implements customizable tools for assessing and grading R or R-markdown scripts from students which allow for checking correctness of code output, runtime statistics and static code analysis. There is a User Manual and a vignettes on the S4 class treeharp.

cachem v1.0.4: Provides functions to cache R objects with automated pruning. Caches can limit either their total size or the age of the oldest object (or both), automatically pruning objects to maintain the constraints. See README for examples.

eList v0.2.0: Provides list compression functions to convert for loops into vectorized lapply() functions which support loops with multiple variables, parallelization, and loops across non-standard objects. See the vignette for examples.

Microsoft365R v1.0.0: Builds on AzureGraph to implement and interface to Microsoft365 and enables access to data stored in SharePoint Online and OneDrive. See the vignette to get started.

rtables v0.3.6: Provides a framework for declaring complex multi-level tabulations and then applying them to data. Tables are modeled as hierarchical, tree-like objects which support sibling sub-tables, arbitrary splitting or grouping of data in row and column dimensions, cells containing multiple values, and the concept of contextual summary computations. There is a Introduction and a series of vignettes on comparing against baseline or control, a clinical trials example, constructing tables manually, pruning and sorting tables, subsetting tables, Tabulation concepts, and a comparison with dplyr tabulation.

targets v0.1.0: Brings together function-oriented programming and make- like declarative workflows in toolkit for building statistics and data science pipelines in R. The methodology borrows from GNU make and drake. See the vignette and the reference website.

Visualization

ggmulti v0.1.0: Provides tools such as serial axes objects, Andrew’s plot, various scatter plot glyphs to visualize high dimensional data. There are vignettes on visualizing high dimensional data, adding glyphs to scatter plots, and creating histograms with density.

ggOceanMaps v1.0.9: Allows plotting data on bathymetric maps using ggplot2 using data that contain geographic information from anywhere around the globe. There is a User Manual and a vignette on pre-made shape files.

pacviz v1.0.0.5: Provides functions to map data onto a radial coordinate system and visualize the residual values of linear regression and Cartesian data in the defined radial scheme. See the pacviz documentation for more information.

parallelPlot v0.1.0: Provides functions to create parallel coordinates plots using the htmlwidgets package and d3.js. The vignette provides multiple examples.

thematic v0.1.1: Provides tools to “theme” ggplot2, lattice, and base graphics using a small set of choices that include foreground color, background color, accent color, and font family. See README for examples.

Some thoughts on rstudio::global talks

Thu, 04 Feb 2021 00:00:00 +0000

The fifty-five videos from last month’s rstudio::global conference are now available online. You will find them at the link above arranged in ten categories (Keynotes, Data for Good, Language Interop, Learning, Modeling, Organizational Tooling, Package Dev, Programming, Teaching, and Visualization) that reflect fundamental areas of technical and community infrastructure and R applications. Theses talks were selected from hundreds of submissions, many of which were really very good. I participated in the first selection round and found it impossible to make some choices, so I am certain that it must have been agonizingly difficult for the program committee to pare down to the final selections.

I believe that you will find the content of most of these talks to be nothing less than compelling. The themes and moods of the talks range from informative and deeply technical R issues to data science, journalism, art, education and public service. A few talks transcend the parochial concerns of the R community and address issues that are important to society at large. It is gratifying to see that in the hands of committed people R is helping to make the world just a little bit better. The videos themselves are high quality and a pleasure to watch. Unlike typical conference videos recorded in real time, all of these were produced with excellent lighting, good audio, and were rehearsed, pre-recorded, and edited. Except for the keynotes, the talks are shorter than twenty minutes.

In the remainder of this post, I will highlight just five talks that I personally found compelling. I have arranged them in an order that I think makes sense to view them. But, you might do just as well to sample talks from the categories listed above that organize the talks on the conference page. My selections do not cover the whole range of topics submitted, and they certainly do not include all of the good stuff. I do think, however, that they reflect the quality of the talks, and I hope that if you watch these five you will be motivated to watch the rest too.

My first three selections are by data journalists who are out to make the world a better place. The Opioid Files: Turning big pharmacy data over to the public by Washington Post data journalist Andrew Ba Tran demonstrates the scale of the opiod scandal. I had the opportunity to meet Andrew at the NICAR conference for Investigative Reporters and Editors in 2019 where he was speaking and teaching R. NICAR opened my eyes to the discipline and tradition of Data Journalism and the efforts of data journalists to harness technology for the public good. Andrew’s conference talk represents this tradition and illustrates the data crunching skills, persistence, and unvarnished storytelling necessary to illuminate a dark topic.

Next, I recommend watching Trial and Error in Data Viz at the ACLU. Sophie Beiers is a data journalist whose work for the ACLU involves discovering and visualizing data with sufficient rigor and clarity to support arguments that will hold up in court. Sophie describes the messy work of iterating through visualizations in a process built around candid feedback from colleagues and stakeholders. Driving the process is a determination to make charts that effectively communicate key points to the intended audience. Sophie’s talk hints at the emotional toll caused by striving to see the people behind the data, and the satisfaction that comes from making a difference. The ACLU analytics team coined a word for expressing the excitement at being able to prove terrible news with quantitative evidence.

The third talk on my list by John Burn Murdoch on Reporting on and visualizing the pandemic continues the theme of polishing visualizations until they work for the target audience. John is a data journalist with the Financial Times who has garnered quite a following for producing data visualizations that command attention. John’s talk dives deeply into his process of evolving a visualization until it not only illustrates what he wants to show but also shows that it is resonating with his mass audience.

In thinking about Sophie and John’s work, the Japanese word kaizen (making something good for the good of other people) comes to mind. There are many R visualization experts who can lay down the basic principles of making a good data visualization, but few I think, that have Sophie and John’s empathy and capacity to listen and process criticism.

The final two talks on my list are by R developers who are concerned with the big picture of sustaining the R package ecosystem. Hadley Wickham’s Keynote Maintaining the house the tidyverse built addresses a fundamental challenge encountered by all complex software projects that develop over time. How do you maintain functional stability while coping with growth and change? Hadley’s solution for the tidyverse encapsulated in the following figure:

may not be the right solution for all subsystems within the R ecosystem, but surely something like it must evolve to reach into all of the corners of the R Universe. For example, consider the CRAN Task Views. When they were assembled, each of these curated lists of R packages represented the cutting edge of software for a particular functional area. The Task View maintainers do a mostly unacknowledged, essential service in keeping these up to date. Nevertheless, it is not difficult to discover Task View packages that have not changed significantly in five or ten years or more. To my knowledge, there are no standards for retiring packages and integrating new work. The future of R depends on systems thinking and the development of new ideas and tools for open source development.

These considerations lead to Jeroen Ooms’ talk: Monitoring health and impact of open-source projects which describes how ROpenSci is taking on the immense challenge of measuring the quality of R packages according to technical, social and scientific indicators, while building out the infrastructure to improve the entire R package ecosystem. Some of the tools for monitoring the status and “health” of open source software are already in place. ROpenSci is offering R-universe, a platform based in git for managing personal R repositories. Once a “universe” of packages is registered with R-universe every time an author pushes an update the platform will automatically build binaries and documentation.

Enjoy the videos!!

Dec 2020: "Top 40" New CRAN Packages

Fri, 29 Jan 2021 00:00:00 +0000

One hundred twenty-three new packages made it to CRAN in December. Here are my “Top 40” selections in nine categories: Computational Methods, Data, Genomics, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

FKF.SP v0.1.0: Provides a fast and flexible Kalman filtering implementation utilizing sequential processing, designed for efficient parameter estimation through maximum likelihood estimation. See the vignette.

rminizinc v0.0.4: Implements an interface to MiniZinc, a free and open-source constraint modeling language which is used to identify feasible solutions out of a very large set of candidates when the problem can be modeled in terms of arbitrary constraints. See the vignette.

nosiySBM v0.1.4: Implements the variational expectation-maximization algorithm to fit a noisy stochastic block model to an observed dense graph and to perform node clustering. See Rebafka & Villers (2020) for background and the vignette to get started.

qsimulatR v1.0: Implements a quantum computer simulator with up to 24 qubits which provides many common gates and allows users to define general single qubit gates and general controlled single qubit gates. The package supports plotting circuits and exporting circuits to Qiskit, a Python package which can be used to run on IBM’s Quantum hardware. There is an Introduction, and vignettes on Exponentiation modulo n, Addition by Fourier transform, the Deutsch-Sozsa Algorithm, the Phase Estimation Algorithm and Quantum Fourier Trafo.

Data

eyedata v0.1.0: Contains anonymized real life, open source data sets from patients treated in Moorfields Eye Hospital, London and includes data about people who received intravitreal injections with anti-vascular endothelial growth factor due to age-related macular degeneration or diabetic macular edema. See README for the list of medical publications associated with the data sets.

rgugik v0.2.1: Automates open data acquisition including raster and vector data from the Polish Head Office of Geodesy and Cartography. See the vignettes Digital Eelvation Model, Orthophotomap, and Topographic Database.

readrba v0.1.0: Provides tools to download current and historical statistical tables and forecasts from the Reserve Bank of Australia Data which comprise a broad range of Australian macroeconomic and financial time series. See the vignette to get started.

threesixtygiving v0.2.2: Provides access to open data from 360Giving, a database of charitable grant giving in the UK. See the vignette.

USgas Links to the US Energy Information Administration to provide and overview of natural gas demand at the county level. See the vignette.

Genomics

polyqtlR v0.0.4: Provides functions for quantitative trait loci (QTL) analysis in polyploid bi-parental F1 populations. See the vignette for background and examples.

RPPASPACE v1.0.7: Provides tools for the analysis of reverse-phase protein arrays (RPPAs), which are also known as tissue lysate arrays or simply lysate arrays. See Hu (2007) for background and the Guide to for examples.

RVA v0.0.3: Provides functions to automate downstream visualization & pathway analysis in RNAseq analysis. See the vignette.

Machine Learning

comparator v0.0.1: Implements functions for comparing strings, sequences and numeric vectors for clustering and record linkage applications. It includes generalized edit distances for comparing sequences/strings, Monge-Elkan similarity for fuzzy comparison of token sets, and L-p distances for comparing numeric vectors. See README to get started.

DoubleML v0.1.1: Implements the double/debiased machine learning framework of Chernozhukov et al. (2018) for partially linear regression models, partially linear instrumental variable regression models, interactive regression models and interactive instrumental variable regression models. There are guides on Installation and Getting Started.

functClust v0.1.6: Provides functions to cluster the components that make up an interactive system on the basis of their functional redundancy for one or more collective, systemic performances. There are six vignettes including and Overview, a simple Use Case, and Multi Fuctionality.

mlpack v3.4.2.1: Implements bindings to the mlpack C++ machine learning library. See Curtin et al (2018) for background and look here for documentation.

RFCCA v1.0.3: Implements Random Forest with Canonical Correlation Analysis, a method for estimating the canonical correlations between two sets of variables depending on the subject-related covariates. The method is described in Alakus et al. (2020). See the vignette for examples.

Medicine

metaSurvival v0.1.0: Provides a function to assess information from a summary survival curve and test the between-strata heterogeneity. See the GitHub repo for an example.

Science

cmcR v0.1.3: Implements the congruent matching cells method for cartridge case identification as proposed by Song (2013) as well as an extension of the method proposed by Tong et al. (2015). There is a vignette on Decision Rules and another vignette reproducing the study by Song et al.

envi v0.1.6: Provides tools for environmental interpolation using occurrence data, covariates, kernel density-based estimation, and spatial relative risk. See Davies et al. (2018) for details on spatial relative risk, Bithell (1990) for kernel density estimation and Bithell (1991) for estimating relative risk. The vignette provides background and examples.

PAMpal v0.9.14: Provides tools for loading and processing passive acoustic data, including functions to read Pamguard data, process, and export data. See Oswald et al (2007), Griffiths et al (2020), and Baumann-Pickering et al (2010) for background. Look here for the installation guide and tutorial.

Statistics

bpcs v1.0.0: Implements models for the analysis of paired comparison data using Stan including random effects, generalized model for predictors and order effect Bayesian versions of the Bradley-Terry model. See Bradley & Terry (1952), Davidson (1970), and Carpenter et al. (2017) for background and the vignette for an overview.

brolgar v0.1.0: Provides a framework of tools to summarise, visualise, and explore longitudinal data and includes methods for calculating features and summary statistics and sampling individual series. See Tierney, Cook & Prvan and the Getting Started Guide to get going. There are also vignettes exploratory modelling, finding features, identifying interesting observations, data structures, mixed effects models, and visualisation.

MASSExtra v1.0.2: Provides enhancements, extensions and additions (such as Gramm-Schmidt orthogonalisation and generalised eigenvalue problems) to the MASS package with convenient default settings and user interfaces. See the vignette.

motifr v1.0.0: Provides tools for motif analysis in multi-level networks to visualize multi-level networks, count multi-level network motifs and compare motif occurrences to baseline models. See the motif zoo and Baseline model comparisons to get started.

OptCirClust v0.0.3: Provides fast (runtime = O(K N log^2 N), optimal, reproducible clustering algorithms for circular, periodic, or framed data based on a core algorithm for optimal framed clustering. There are vignettes on Circular genome clustering, Performance, Circular Clustering, and Framed Clusterine.

pflamelet v0.1.1: Provides functions to compute the persistence flamelets, a statistical tool for exploring the Topological Invariants of Scale-Space families introduced in Padellini and Brutti (2017).

PRDA v1.0.0: Implements the Design Analysis proposed by Gelman & Carlin (2014) which combines the evaluation of Power-Analysis with other inferential-risks. See also Altoè et al. (2020) and Bertoldo et al. (2020) for background and the vignettes PRDA, Prospective and Retrospective.

puls v0.1.1: Supplements the fda and fda.use packages by providing a method for clustering functional data using subregion information of the curves. See the vignette for an example and references.

Utilities

coro v1.0.1: Provides coroutines, a family of functions that can be suspended and resumed later on. This includes async functions (which await) and generators (which yield). See the vignette.

dataReporter v1.0.0: Provides functions to auto generate a customizable data report showing potential errors in a data set. See Petersen & Ekstrøm (2019) for background, and the vignette for examples.

DescrTab2 v2.0.3: Provides functions to create descriptive statistics tables for continuous and categorical variables. There are vignettes on Maintenance, Usage, and Validation.

libr v1.1.1: Provides functions to create data libraries, generate data dictionaries, and simulate a data step. There is an Introduction, and vignettes on library operations and management, and Data Step operations and the enhanced equality operator.

outsider v0.1.1: Allows users to install and run external command-line programs in R through use of Docker and online repositories. Look here for package information.

srcr v1.0.0: Provides a simple tool to abstract connection details, including secret credentials, out of your source code and manage configurations for frequently-used database connections. See the vignette.

Visualization

ComplexUpset v1.0.3: Provides functions to create Upset plots which offer improvements over Venn Diagrams for set overlap visualizations.

nmaplateplot v1.0.0: Provides a graphical display of results from network meta-analysis (NMA) which is suitable for outcomes like odds ratios, risk ratios, risk differences, and standardized mean differences. See the vignette for examples. vignette for examples.

PantaRhei v0.1.2: Provides functions to produce Sankey diagrams which are used to visualize the flow of conservative substances through a system. See the vignette.

COVID-19 Data: The Long Run

Wed, 06 Jan 2021 00:00:00 +0000

The world seems to have moved to a new phase of paying attention to COVID-19. We have gone from pondering daily plots of case counts, to puzzling through models and forecasts, and are now moving on to the vaccines and the science behind them. For data scientists, however, the focus needs to remain on the data and the myriad issues and challenges that efforts to collect and curate COVID data have uncovered. My intuition is that not only will COVID-19 data continue to be important for quite some time in the future, but that efforts to improve the quality of this data will be crucial for successfully dealing with the next pandemic.

An incredible amount of work has been done by epidemiologists, universities, government agencies and data journalists to collect, organize, and reconcile data from thousands of sources. Nevertheless, the experts caution that there is much yet to be done.

Roni Rosenfeld, head of the Machine Learning Department of the School of Computer Science at Carnegie Mellon University and project lead for the Delphi Group put it this way in a recent COPSS-NISS webinar:

Data is a big problem in this pandemic. Availability of high quality, comprehensive, geographically detailed data is very far from where it should be.

There are over 6,000 hospitals in the United States, and over 160,000 hospitals worldwide. Many of these are collecting COVID-19 data yet there few standards for recording cases, dealing with missing data, updating case count data, and coping with the time lag between recording and reporting cases. Nowcasting epidemiological and heath care data has become a vital field of statistical research.

The following slide from the COPSS-NISS webinar shows a hierarchy of relevant COVID data organized on the Severity Pyramid that epidemiologists use to study disease progression.

The Delphi Group is making fundamental contributions to the long term improvement of COVID data by archiving the data shown in such a way that versions can be retrieved by date, and also by collecting massive data sets of leading indicators.

The webinar is well worth watching, and I highly recommend listening through the Q&A session at the end. The speakers explain the importance of nowcasting and Professor Rosenfeld presents a vision of making epidemic forecasting comparable to weather forecasting. It seems to me that this would be a worthwhile project to help advance.

Note that the Delphi’s COVID-19 indicators, probably the nation’s largest public repository of diverse, geographically-detailed, real-time indicators of COVID activity in the US, are freely available through the public API which is easily accessible to R and Python users.

Also note that R users can contribute to R Consortium sponsored COVID related projects that include the COVID-19 Data Hub an organized archive of global COVID-19 case count data, and the RECON COVID-19 Challenge, an open source project to improve epidemiological tools.

Learn and Teach R

Wed, 02 Dec 2020 00:00:00 +0000

If you haven’t explored the RStudio website in a while, your next visit may include a pleasant surprise. I recently went to the Tidymodels page, just to see what was new and was immediately drawn into the new landscape imagined by the RStudio education team. Clicking on Get Started I came to a fork and a choice of going farther with Tidymodels or backing up a bit. I went down the beginners path Finding Your Way to R and found myself in a well-lit wood, and I was not lost.

Like a park with well-marked trails, the Learning R section of the RStudio Education site branches off to R excursions graded to match the “hikers” experience. The Beginners trail starts in a safe place that should make even the absolutely terrified feel comfortable. It offers different modes of learning including videos, tutorials, books and even excursions to trusted third party sites that have their own feel.

The Intermediates section emphasizes learning how to get help, suggests some basic tools that should be useful no matter what path you select, and then points to places where you may already know you want to go: bioconductor, financial models or Spark clusters for example. The basic idea is that at this stage you know enough R to get some real work done, and a good path to making further progress might be to follow what interests you.

The Experts trail is for the intrepid who are ready to venture past terra firma and take a deep dive

into the foundations of R, or package building, or using Python, or exploring deep learning with Tensorflow. As with the others, this trail is well marked and offers tools and suggestions for making progress.

Perhaps the most pleasant surprise for an educator coming to the RStudio Education site is to see that the trails don’t stop with Expert. There is an entire section of the “park” marked off for teaching with trails that offer essential material for supporting both educators and online education. Learn to teach offers advice on how to develop as a teacher based on practical experience with the Carpentries. The section on Materials for teaching offers a panpoply of courses and workshops developed at RStudio that teachers can freely adapt to their needs. There is material here relevant to data wrangling, data science, R, tidyverse, shiny and more. The third section, Tools for teaching describes RStudio Cloud and other RStudio tools for creating a modern, interactive teaching infrastructure.

Whether you are just thinking about learning R or about to teach an advanced course, I think you might enjoy walking around the RStudio Education website. Maybe, I’ll get to Tidymodels next time.

October 2020: "Top 40" New CRAN Packages

Thu, 19 Nov 2020 00:00:00 +0000

One hundred fifty-six new packages made it to CRAN in October. Here are my “Top 40” selections in eight categories: Computational Methods, Data, Epidemiology, Mathematics, Machine Learning, Statistics, Utilities, and Visualization.

Computational Methods

mcmcensemble v 2.0: Provides ensemble samplers for affine-invariant Monte Carlo Markov Chain algorithms which allow a faster convergence for badly scaled estimation problems. Two samplers are included: the ‘differential.evolution’ sampler from the Braak and Vrugt (2008) and the ‘stretch’ sampler from Goodman and Weare (2010). Look here for examples.

psqn v0.1.3: Provides quasi-Newton methods to minimize partially separable functions. The methods are described by Nocedal and Wright (2006). There is an Introduction and a vignette on the Partially Separable Quasi-Newton method.

Data

AirSensor v1.0.2: Allows processing and displaying data from air quality sensors. Initial focus is on PM2.5 measurements from sensors produced by PurpleAir.

fflr v0.3.12: Provides functions to format the raw data from the ESPN fantasy football API into tidy tables. See the vignette.

pmlbr v0.2.0: Provides access to more than 150 classification and regression data sets in the University of Pennsylvania’s PMLB repository. See README to get started.

podr v0.0.5: Allows users to to connect, access and review over 250 datasets in Pharmaceutical User Software Exchange (PHUSE) open data repository (PODR). See the vignette for details.

starwarsdb v0.1.2: Provides the data from the Star Wars API as a set of relational tables or a DuckDB database. Look here for an example.

Epidemiology

anovir v0.1.0: Implements maximum likelihood techniques to estimate virulence in population dynamics models. See the pre-print and the eleven vignettes including Introduction, Confidence intervals and Worked examples I and II.

epifitter v0.1.0: Provides functions for fitting two-parameter population dynamics models to proportion data for single or multiple epidemics using either linear or non-linear regression. See Madden et al. (2007) for background and the vignettes on fitting models and simulating disease progress.

epitweetr v0.1.24: Allows users to automatically monitor trends of tweets by time, place and topic aiming at detecting public health threats early through the detection of signals (e.g. an unusual increase in the number of tweets). It was designed to focus on infectious diseases, but can be adapted to other applications by modifying the topics and keywords. See the vignette.

i2extras v0.0.2: Provides functions to work with ‘incidence2’ objects, including a simplified interface for trend fitting and peak estimation. This package is part of the RECON toolkit for outbreak analysis. There are vignettes on Fitting epicurves and Peak Estimation.

IBMPopSim v0.3.1: Provides functions to simulate the random evolution of structured population dynamics, called stochastic Individual Based Models (IBMs). Users can simulate the random evolution of a population in which individuals are characterized by their date of birth, a set of attributes, and their potential date of death. See Ferrière and Tran (2009) and Bansaye and Méléard (2015) for background. There is a package overview and vignettes on C++ essentials, Human populations, Human populations with changing characteristics, Insurance portfolio, and populations with genetically variable traits.

MGDrivE2 v1.0.1: Provides a simulation modeling framework which significantly extends capabilities of the MGDrivE package with a new mathematical and computational framework based on stochastic Petri nets. To get started with this package see the vignettes on SEIR dynamics, Meta population dynamics, One node dynamics, Inhomogenous stochastic processes, Life-cycle dynamics, One node lifecycle dynamics, data analysis, and advanced topics

msce v1.0.1: Provide functions to calculate hazard and survival functions for multi-stage clonal expansion models used in cancer epidemiology. There is a vignette on fitting incidence data.

Machine Learning

acumos v0.4-1: Provides access to the Linux Foundation Acumos open source framework intended to make it easy to build, share, and deploy AI apps. Look here for information on the Acumos R CLient.

bigSurvSGD v0.0.1: Provides a function to fit Cox models via stochastic gradient descent which avoids the computational instability of the standard Cox Model. Functions scales up with large datasets and accommodate datasets that do not fit the memory. See Aliasghar et al. (2020) for details.

deepredeff v0.1.0: Implements a tool that contains trained deep learning models for predicting effector proteins using a set of known experimentally validated effectors from either bacteria, fungi, or oomycetes. Kristianingsih and MacLean (2020) for background, and the overview and vignette on prediction.

MKclass v0.3: Implements performance measures and scores for statistical classification including accuracy, sensitivity, specificity, recall, similarity coefficients, AUC, GINI index, Brier score and more. It calculates optimal cut-offs and decisions stumps according to (Iba and Langley (1991) follows Lemeshow and Hosmer and Hosmer et al. (1997) for goodness of fit tests and Porta (2014) for epidemiological risk measures. See the vignette to get started.

mlr3hyperband v0.1.0: Implements the bandit-based hyperparameter optimization method of Li et al. (2016). Look here for an example.

Mathematics

tdaunif Provides functions to randomly sample from simple manifolds. See Arvo (1995) and Diaconis, Holmes, and Shahshahani (2013) for the theory, and the vignette to get started.

Science

SoundShape v1.0: Implements the eignensound method of MacLeod et al. (2013) to compare stereotyped sound from different species. The vignette will make you want to start comparing sound waves.

Statistics

anscombiser v1.0.0: Provides functions to create data sets with identical summary statistics: i.e. identical marginal sample means and sample variances, sample correlation, least squares regression coefficients and coefficient of determination, that that look amusingly different. See the vignette for examples.

gglm v0.1.0: Extends ggplot2 for creating common diagnostic plots associated with linear models. Look here for examples.

microcluster v0.1.0: Implements the method in Betancourt et al. (2020) to perform microclustering models for categorical data. The vignette offers an example.

PressPurt v1.0.2: Provides functions to identify the most sensitive interactions within a network which can be described by differential equations, in order to produce qualitatively robust predictions to a press perturbation. See Koslicki & Novak (2017) for background. There a tutorial and a vignette for set up.

quantdr v1.0.0: Provides functions to perform dimension reduction for conditional quantiles by determining the directions that span the central quantile subspace (CQS). The vignette contains the details.

singcar v0.1.1: Implements frequentist and Bayesian methods to compare single cases to control populations. See Crawford and Howell (1998) and Crawford and Garthwaite (2005) for background, and the vignette for an introduction.

statgenGxE v1.0.3: Provides functions to facilitate the analysis of multi-environment data from plant breeding experiments following the analyses described in Malosetti et al. (2013). See the vignette for examples.

survivalMPLdc v0.1.1: Implements functions to fit Cox proportional hazard models under dependent right censoring using copula and maximum penalized likelihood methods. See the vignette for examples.

Time Series

FoReco v0.1.1: Provides bottom-up, optimal and heuristic combination forecast reconciliation procedures for cross-sectional, temporal, and cross-temporal linearly constrained time series. There is an introduction and another vignette on average relative accuracy indices.

modeltime.ensemble v0.3.0: Implements time series ensemble forecasting methods including model averaging, weighted averaging, and stacking. See Pavlyshenko (2019) for the theory and the vignette for an introduction.

Utilities

findInFiles v0.1.2: Enables users to search for a pattern in a folder and display the results in the RStudio viewer pane or in an R Markdown file or Shiny app.

groundhog v1.0.0: Assists with reproducibility by providing functions for version specific package loading. See the vignette.

monaco v0.1.0: Implements an HTML widget rendering of the Monaco editor which is useful for JavaScript. Look here for examples.

oskeyring v0.1.1: Aims to support all features of the system credential store, including non-portable ones. It supports Keychain on macOS, and Credential Manager on Windows. See the keyring package if you need a portable API. README describes how to get started.

PDE v1.1.1: Enables extracting information and tables from pdf files based on search words. The vignette has several examples.

representr v0.1.1: Implements the record linkage methods of Kaplan et al. (2020) to create representative records for use in downstream tasks after entity resolution is performed. See the vignette.

Visualizations

imbibe v0.1.0: Provides a set of fast, chainable image-processing operations which are applicable to images of two, three or four dimensions, particularly medical images. Look here for an example.

r3dmol v0.1.0: Provides functions to create and manipulate rich and fully interactive 3D visualizations of molecular data that can be included in Shiny apps and R markdown documents. See the vignette for an introduction.

visualpred v0.1.0: Provides 2D point and contour plots for visualizing binary classification models. There is an introduction and vignettes on plotting outliers, comparing algorithms, and advanced settings.

Sept 2020: "Top 40" New CRAN Packages

Fri, 23 Oct 2020 00:00:00 +0000

Two hundred thirty-six new packages made it to CRAN in September. Here are my “Top 40” picks in eleven categories: Computational Methods, Data, Finance, Genomics, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Utilities and Visualization. The large number of packages and, in my opinion, the high percentage of high quality work made choosing only forty more difficult than for most months.

Computational Methods

pmwg v0.1.9: Provides an R implementation of the Particle Metropolis algorithm within a Gibbs sampler for model parameter. Covariance matrix and random effect estimation are described in Gunawan et al. (2020). There is a Tutorial.

sanic v0.0.1: Provides access to Eigen C++ library routines for solving large systems of linear equations. Direct and iterative solvers available include Cholesky, LU, QR, and Krylov subspace methods.

Data

cmsafops v1.0.0: Provides functions for the analysis and manipulation of CM SAF climate monitoring data. Detailed information and test data are available here.

friends v0.1.0: PRovides complete scripts from the American sitcom Friends in tibble format. Use this package to practice data wrangling, text analysis and network analysis. See README for examples.

nflfastR v3.0.0: Provides functions to access National Football League play-by-play data. Look here for examples.

od v0.0.1: Provide tools and example datasets for working with origin-destination (‘OD’) datasets of the type used to describe aggregate urban mobility patterns Carey et al. (1981) and supports the sf class system of Pebesma (2018). See the vignette to a brief introduction to OD data.

Finance

GARCHIto v0.1.0: Provides functions to estimate model parameters and forecast future volatilities using the Unified GARCH-Ito Kim and Wang (2016) and Realized GARCH-Ito Song et. al. (2020) models. See the vignette for an introduction.

LifeInsuranceContracts v0.0.2: Provides a framework for modeling traditional life insurance contracts such as annuities, whole life insurances or endowments and includes modeling profit participation schemes, dynamic increases or more general contract layers, as well as contract changes. See the vignette for details.

Genomics

dPCP v1.0.3: Implements the automated clustering and quantification of the digital PCR data is based on the combination of DBSCAN (Hahsler et al. (2019) and c-means (Bezdek et l. (1981) algorithms. See the vignette for examples.

MAPITR v1.1.2: Implements the algorithms described in Turchin et al. (2020) for identifying marginal epistasis between pathways and the rest of the genome. See the vignette for an example with simulated data.

Machine Learning

FuncNN v1.0: This Allows the user to build models of the form: f(z, g(x) | θ) where f() is a neural network, z is a vector of scalar covariates, and g(x) is a vector of functional covariates. The package is built on top of the Keras/Tensorflow architecture. See Thind et al. (2020) for information on the methodology, and README for an example.

shapr 0.1.3: Implements the method for computing Shapley Values which accounts for feature independence as described in Aas et al. (2019) to help interpret machine learning models. See the vignette for details.

rMIDAS v0.1.0: Implements the method for multiple imputation using denoising autoencoders described in Lall & Robinson (2020) that has advantages for large data sets.

Mathematics

Riemann v0.1.0: Provides algorithms for manifold-valued data, including Fréchet summaries, hypothesis testing, clustering, visualization, and other learning tasks. Look here for the math.

simplextree v1.0.1: Provides an interface to a Simplex Tree data structure which enables efficient manipulation of simplicial complexes of any dimension. See Boissonnat & Maria (2014) for background and look here for a quickstart.

topsa v0.1.0: Provides functions to estimate geometric sensitivity indices reconstructing the embedding manifold of a data set. Detailed information of can be found in Hernandes et al.

Medicine

Statistics

bkmrhat v1.0.0: Extends the Bayesian kernel machine regression package bkmrto allow multiple-chain inference and diagnostics by leveraging functions from the future, rstan, and coda package. See Bobb et al. (2018) for background and the vignette for examples.

densEstBayes v1.0-1: Provides functions for density estimation via Bayesian inference engines including Hamiltonian Monte Carlo, the no U-turn sampler, semiparametric mean field variational Bayes and slice sampling. The methodology is described in Wand and Yu (2020). The vignette has several examples.

EquiSurv v0.1.0: Provides both a non-parametric and a parametric approach to investigating the equivalence (or non-inferiority) of two survival curves obtained from two given datasets. Tests are based on the creation of confidence intervals at pre-specified time points. see Möllenhoff &Tresch (2020) for all of the details.

gmGeostats v0.10-7: Provides functions to support the geostatistical analysis of multivariate data, in particular data with restrictions. See Tolosana-Delgado et al. (2018) for background and the vignette for the basics.

hermiter v1.0.0: Provides functions for estimating the full probability density function, cumulative distribution function and quantile function using Hermite series based estimators which are particularly useful in the sequential setting (both stationary and non-stationary) and one-pass batch estimation setting for large data sets. See Stephanou et al. (2017) and Stephanou et al. (2020) for background and the vignette for examples.

ivreg v0.5-0: Implements instrumental variable estimation for linear models by two-stage least-squares (2SLS) regression. Several methods are provided for fitted ivreg model objects, including extensive functionality for computing and graphing regression diagnostics in addition to other standard model tools. There is an overview and a vignette on diagnostics.

mcmcsae v0.5.0: Provides functions to fit multi-level models with possibly correlated random effects using Markov Chain Monte Carlo simulation. There are vignettes on Area-level models, Linear Regression, and Unit-level models.

rater v1.0.0: Provides functions to fit models based on Dawid & Skene (1979) to repeated categorical data. The vignette describes the modeling workflow.

testtwice v1.0.3: Implements the method of Rosenbaum (2012) to test one hypothesis with several test statistics while correcting for multiple testing.

txshift v0.3.4: Provides functions to estimate the population-level causal effects of stochastic interventions on a continuous-valued exposure. The causal parameter and estimation methodology are described in Díaz & van der Laan (2013). There is an Introduction to Targeted Learning and an additional vignette with a more advanced example.

vacuum v0.1.0: Implements Tukey’s FUNOP (FUll NOrmal Plot), FUNOR-FUNOM (FUll NOrmal Rejection-FUll NOrmal Modification), and vacuum cleaner procedures to identify, treat, and analyze outliers in contingency tables. See Tukey (1962). There is a vignette on the vacuum.

Time Series

localFDA v1.0.0: Implements a theoretically supported alternative to k-nearest neighbors for functional data to solve problems of estimating unobserved segments of a partially observed functional data sample, functional classification and outlier detection. The methodology and details are in Elías et al. (2020). Look here for some examples.

onlineforecast v0.9.3: Implements a framework for fitting adaptive forecasting models that provides a way to use forecasts as input to models, e.g. weather forecasts for energy related forecasting. There are vignettes on Forecast Evaluation, Model Setup, and Data Setup.

Utilities

cmdfun v1.0.2: Provides a framework for building function calls to interface with shell commands by allowing lazy evaluation of command line arguments. It is intended to enable package builders to wrap command line software, and to help analysts stay inside the R environment. Full documentation is on the package website.

ducdb v0.2.1-2: The DuckDB project is an embedded analytical data management system with support for the Structured Query Language (SQL). This package includes all of DuckDB and a R Database Interface (DBI) connector.

path.chain v0.2.0: Provides path_chain class and functions which facilitates loading and saving directory structure in YAML configuration files via config package. There is a vignette on Path Validation and another on Config Files.

procmaps v0.0.3: Provides functions to determine which library or other region is mapped to a specific address of a process. It is the equivalent of /proc/self/maps as a data frame, and is designed to work on all major platforms.

robservable v0.2.0: Enables loading and displaying online JavaScript Observable notebook. Have a look at the Gallery, the Introduction, and the vignette on Shiny Applications.

Visualization

catmaply v0.9.0: Implements methods and plotting functions for displaying categorical data on an interactive heatmap using plotly. In addition to the viewer pane, resulting plots can be saved as a standalone HTML file, embedded in R Markdown documents or in a Shiny app. The vignette offers examples.

diffviewer v0.1.0: Implements an HTML widget that shows differences between files (text, images, and data frames).

ggip v0.2.0: Extends ggplot2 to enable the visualization of IP (Internet Protocol) addresses and networks using space filling curves that map the address space onto Cartesian coordinates. It offers full support for both IPv4 and IPv6 address spaces. There is an Introduction and a vignette on Visualizing IP Data.

Some Thoughts on R / Medicine 2020

Wed, 16 Sep 2020 00:00:00 +0000

The third annual R / Medicine Conference was held online this year from August 27th to August 29th and was an unqualified success. The last minute pivot from small, in-person conference, which was to be held onsite at the Children’s Hospital of Philadelphia, CHOP, to a virtual event turned out to be a catalyst for positive change. Under the imaginative, and tireless leadership of conference chair Dr. Stephan Kadauke, R / Medicine grew from a small R conference to become a medical conference with international reach. The map below shows that the conference attracted international attendees from forty-three countries. (Click on the markers to see country and number of attendees.)

Four hundred fifty-two people attended the live event and four hundred thirty-one watched the replay. Equally impressive is that approximately sixteen percent of conference registrants supplied titles which suggest that they are medical doctors. This figure does not include medical students, nurses, or ancillary clinical staff.

Much of the success in attracting the target audience this year was very likely attributable to the effort that the organizers made to reach out to sponsors familiar to clinicians. In addition to the R Consortium, R Consortium member companies Procogia and RStudio, The Yale School of Public Health (the conference host for its first two years), this year R / Medicine attracted the Children’s Hospital of Philadelphia, the American Association of Clinical Chemistry, AACC, and the Association for Mass Spectrometry & Advances in the Clinical Lab, MSACL. The panel discussion Integrating R into Clinical Practice, held during virtual US MSACL 2020 Conference in July was particularly effective in spreading the word among clinicians.

Another key element of the success of the R / Medicine 2020 was that the Program Committee chaired by Beth Atkinson looked beyond the usual cadre of R luminaries and assembled a roster of speakers with deep experience in medical related applications. Keynote talks from Daniela Witten, Robert Gentleman, Ewin Harrison, and Patrick Mathias ranged from a sophisticated analysis of a common mistake in predictive modeling and a glimpse into the future of computational genetics to recent work related to the COVID-19 pandemic. Pre-conference workshops were delivered by Stephan Kadauke: (Intro to R for Clinicians), and Alison Hill (Intro to Machine Learning with Tidymodels). Other talks covered R education, reproducible research, operational clinical workflows, clinical reporting, and statistical analyses. My favorite talk title The MD in .rmd: Teaching Clinical Data Analytics with R by Ted Laderas perfectly captured the spirit of R / Medicine 2020.

All of the keynote and regular talks are available on the R Consortium’s Youtube Channel. Click on “PLAYLISTS” and select the first on the left.

The big takeaways from R / Medicine 2020 are that R is an established tool in clinical applications. Doctors are teaching doctors about R. And, as knowledge about R propagates, R use in clinical workflows is increasing..

Be sure to mark your 2021 calendar with reminder about R / Medicine for sometime in late August. Also note that sixty-eight percent of the folks who filled out the post conference survey indicated that they would be interested in a hybrid event next year even if progress with mitigating the risks of COVID-19 permits an in person event next year. So the R / Medicine team is going to be thinking hard about how to keep their international following. Moreover, the team hopes to be able to offer R / Medicine branded events throughout the year. Please watch the R Consortium Blog for news and updates.

Fake Data with R

Wed, 09 Sep 2020 00:00:00 +0000

Simulation is the foundation of computational statistics and a fundamental organizing principle of the R language. For example, few complex tasks are more compactly expressed in any programming language than rnorm(100). But while many simulation tasks are trivial in R, simulating adequate and convincing synthetic or “fake” data is a task whose cognitive demands increases quickly as complexity moves beyond independent random draws from named probability distributions. In this post, I would like to highlight a few of the many R packages that are useful for simulating data.

Before we begin, let’s just list a few reasons why you may want to make fake data. Having some of these in mind should be useful for exploring the tools provided in the R packages listed below. Bear in mind that no one package is going to cover everything. But before simulating data on you own, it may be helpful to see where others thought it was worthwhile make the effort to write a package.

Why simulate data?

Simulation is part of exploratory data analysis. For example, when you may have data that comes from some kind of arrival process, it may be helpful to simulate Poisson arrivals to see if counts and arrival time distributions match with your data set. Having a generative model might really simplify your analysis, and knowing that you don’t have one may be critical too.
Carefully constructed fake data may be helpful in testing the limits of an algorithm or analysis. Everybody wants to publish an algorithm that really seems to explain some data set. Not too many people show you the edge conditions where their algorithm breaks down.
Making fake data may be a good way to begin a project before you even have any data. There is no better way to expose you assumptions than to write them down in code.
Fake data can be helpful when there are privacy concerns. Building a data set that reproduces some of the statistical properties of the real data while passing the eyeball test for being convincing may make all the difference in communicating your ideas.

R Packages for Simulating Data

Here are a few of R package that ought to be helpful in nearly every project where you need to manufacture fake data.

bindata is an “oldie but goodie” from R Core members Friedrich Leisch. Andreas Weingessel, and Kurt Hornik that goes back to 1999 and still gets about twenty-five downloads a day. The package does one really nice trick: it provides multiple ways to create columns of binary random variables that have a pre-specified correlation structure.

Suppose you want to create a matrix whose columns are draws from binary random variables. You can begin by setting up a matrix, M, of “common probabilities”. In the simple example below M[1,1] specifies the probability that the random variable V1 will be a 1, while M[2,2] does the same for variable V2. The off diagonal elements specify the joint probabilities that V1 and V2 are both one.

M <-  matrix(c(.5, .2, .2, .5), nrow = 2, byrow = TRUE)
colnames(M) <- c("V1","V2")
M

##       V1  V2
## [1,] 0.5 0.2
## [2,] 0.2 0.5

res <- rmvbin(100,commonprob = M)
head(res)

##      [,1] [,2]
## [1,]    0    0
## [2,]    0    1
## [3,]    0    1
## [4,]    0    0
## [5,]    1    1
## [6,]    1    1

For this example, it’s easy to check the results by plotting a couple of histograms. The tricky part is deciding on the correlation structure. There are some limitations on what the common probabilities can be, but the authors provide a helper function: check.commonprob(M) so you can check ahead of time.

charlatan from Scott Chamberlain and the rOpenSci team allows you to make convincing fake addresses, person names, dates, times, coordinates, currencies, DOIs’, jobs, phone numbers, DNA sequences and more.

Here is a small example of simulating gene sequences.

ch_gene_sequence(n = 10)

##  [1] "CTNNTTCTNCACNTCCNNATCGGNGACNCG" "CAGGACCAGNCCNNNTGAAAGTCANCNTCT"
##  [3] "CGNTTNAGTGATGCANGNCGCAAACCGTGC" "TGCNAACTATTCGGANNTCAGNCTTCTTNC"
##  [5] "NNCGGAGTTGNGTNAGNCNCGCGTTCGGNT" "NTCCCCTACACNNTTAANTTGNTATNTCGG"
##  [7] "GNGACNAAAGCANNNGGTAAGGTACTNNNA" "AGCGGNTNCNGGCGGCATNAGNCCNCNCTC"
##  [9] "AGAANNCNTGTGGACCGNNCNGNNAGCNTA" "TANCNCTCTTNNGNAANGNNTNNANACAGC"

And here is another small example showing how to simulate random longitude and latitude coordinates. (These very likely point to some places you’ve never been that might make good vacation destinations when you can actually go somewhere again.)

set.seed(1234)
locations <- as.data.frame(cbind(ch_lon(n=10),ch_lat(n=10)))
names(locations) <- c("lon", "lat")
leaflet(locations) %>% addProviderTiles("Stamen.Watercolor")  %>%
  addMarkers(~lon, ~lat)

fabricatr is part of the DeclareDesign suite of packages from Graeme Blair and his colleagues for “formally ‘declaring’ the analytically relevant features of a research design”. As the package authors put it: “(the package) helps you imagine your data before you collect it”, and provides functions for building hierarchical data structures and correlated data. To see some of what it can do, please have a look at this R Views post by the package authors that works through examples of hierarchical data, longitudinal data and intra-class correlation. There is also a tutorial.

fakeR from Lily Zhang and Dustin Tingley helps to solve the problem of making fake data that matches your real data. It simulates data from a data set with various data types. It randomly samples character and factor data from contingency tables and numeric and ordered data from multivariate distributions that account for the between column correlations among the variables. There are also functions to simulate stationary time series.

In this example, we simulate data from the USArrests data set.

set.seed(1234)
state_names <- rownames(USArrests)
df <- tibble(state_names)  
data <- USArrests 
rownames(data) <- NULL
sim_data <- simulate_dataset(data)

## [1] "Numeric variables. No ordered factors..."

fake_arrests <- cbind(df, sim_data)
head(fake_arrests)

##   state_names Murder Assault UrbanPop  Rape
## 1     Alabama   3.96   179.8    77.89  7.28
## 2      Alaska  10.64   209.3    57.91 19.32
## 3     Arizona   3.01    87.8    54.51  7.86
## 4    Arkansas   5.48   175.9    79.31 22.17
## 5  California   4.81   104.5    55.52 31.75
## 6    Colorado   6.88   131.7    58.60 21.03

GenOrd by Bapiero and Ferrari implements gaussian copula based procedure for generating samples from discrete random variables with prescribed correlation matrix and marginal distributions.

The following is a slightly annotated version of Example 2 given on page nine of the package pdf. The problem is to draw samples from four different discrete random variables with different numbers of categories that have different specified uniform marginal distributions but conform to a specified correlation structure.

The example begins by specifying the marginal distributions and then running the function corrcheck() to get the upper and lower ranges for a feasible correlation matrix. Note that the random variables are set to have respectively 2, 3, 4, and 5 categories and that it is not necessary to specify the final 1 in each vector of cumulative marginal probabilities.

k <- 4 #number of random variables
marginal <- list(0.5, c(1/3,2/3), c(1/4,2/4,3/4), c(1/5,2/5,3/5,4/5))

corrcheck(marginal)

## [[1]]
## 4 x 4 Matrix of class "dsyMatrix"
##         [,1]    [,2]    [,3]    [,4]
## [1,]  1.0000 -0.8165 -0.8944 -0.8485
## [2,] -0.8165  1.0000 -0.9129 -0.9238
## [3,] -0.8944 -0.9129  1.0000 -0.9487
## [4,] -0.8485 -0.9238 -0.9487  1.0000
## 
## [[2]]
## 4 x 4 Matrix of class "dsyMatrix"
##        [,1]   [,2]   [,3]   [,4]
## [1,] 1.0000 0.8165 0.8944 0.8485
## [2,] 0.8165 1.0000 0.9129 0.9238
## [3,] 0.8944 0.9129 1.0000 0.9487
## [4,] 0.8485 0.9238 0.9487 1.0000

Given the bounds, we select a feasible correlation matrix, Sigma, and generate n samples of the four random variables.

Sigma <- matrix(c(1,0.5,0.4,0.3,0.5,1,0.5,0.4,0.4,0.5,1,0.5,0.3,0.4,0.5,1),
                k, k, byrow=TRUE)
Sigma

##      [,1] [,2] [,3] [,4]
## [1,]  1.0  0.5  0.4  0.3
## [2,]  0.5  1.0  0.5  0.4
## [3,]  0.4  0.5  1.0  0.5
## [4,]  0.3  0.4  0.5  1.0

set.seed(1)
n <- 1000 # sample size
m <- ordsample(n, marginal, Sigma)
head(m,10)

##       [,1] [,2] [,3] [,4]
##  [1,]    2    3    4    2
##  [2,]    1    2    3    1
##  [3,]    2    3    2    5
##  [4,]    1    1    1    1
##  [5,]    2    1    2    2
##  [6,]    1    3    3    5
##  [7,]    1    3    1    1
##  [8,]    1    1    3    3
##  [9,]    1    2    3    2
## [10,]    2    3    3    2

Finally, check how well the simulated correlation structure agrees with the specified correlation matrix, and compare the empirical cumulative marginal probabilities with the specified probabilities. Things look pretty good.

cor(m) # compare it with Sigma

##        [,1]   [,2]   [,3]   [,4]
## [1,] 1.0000 0.5199 0.4352 0.3097
## [2,] 0.5199 1.0000 0.4907 0.3812
## [3,] 0.4352 0.4907 1.0000 0.4690
## [4,] 0.3097 0.3812 0.4690 1.0000

cumsum(table(m[,4]))/n # compare it with the fourth marginal specified above

##     1     2     3     4     5 
## 0.189 0.392 0.592 0.793 1.000

MultiOrd by Amatya, Demirtas and Gao generates multivariate ordinal data given marginal distributions and a correlation matrix using the method proposed in Demirtas (2006).

PoisBinOrdNonNor by Demirtas et al. uses the power polynomial method described in Demirtas (2012) to generate count, binary, ordinal, and continuous random variables, with specified correlations and marginal properties.

SimMultiCorrData by Allison Cynthia Fialkowski is a tour de force of a package that was built using ideas pioneered in some of the packages listed above, and was part of her award winning PhD thesis. A motivating goal of the package is to enable users to make synthetic data useful in clinical applications. This includes generating data to match theoretical probability distributions, and also to mimic empirical data sets. The package is notable not only for the sophisticated capabilities it provides, but also for the technical documentation and references to source materials.

SimMultiCorrData provides tools to simulate random variables from continuous, ordinal, categorical, Poisson and negative binomial distributions with precision and error estimates. It exhibits attention to numerical detail that is rare in the machine learning literature, and impressive even by R’s standards. For example, there are several functions to evaluate and visualize the quality of the synthetic data.

The package provides three main simulation functions. nonnormvar1() is preferred for simulating a single, continuous random variable, rcorrvar() and rcorrvar2() are two different methods for generating correlated ordinal, continuous, Poisson, and negative binomial random variables that match a specified correlation structure.

The following example is a stripped-down, annotated version of the example provided in the package pdf for the rcorrvar() function. The example simulates one ordinal random variable (k_cat), two continuous random variables (k_cont one is logistic and the other Weibull), one Poisson random variable (k_pois), and one negative binomial random variable (k_nb). The portion replicated below concludes with generating the data. The version in the package pdf also works through the process of evaluating the fake data.

# Binary, Ordinal, Continuous, Poisson, and Negative Binomial Variables
options(scipen = 999) # decides when to switch between fixed or exponential notation
seed <- 1234
n <- 10000 # number of random draws to simulate
Dist <- c("Logistic", "Weibull") # the 2 continuous rvs to be simulated 
#Params list: first element location and scale for logistic rv
#             second element shape and scale for Weibull rv
Params <- list(c(0, 1), c(3, 5))
# Calculate theoretical parameters for Logistic rv including 4th, 5th and 6th moments
Stcum1 <- calc_theory(Dist[1], Params[[1]]) 
# Calculate theoretical parameters for a Weibull rv 
Stcum2 <- calc_theory(Dist[2], Params[[2]])
Stcum <- rbind(Stcum1, Stcum2)
rownames(Stcum) <- Dist
colnames(Stcum) <- c("mean", "sd", "skew", "skurtosis", "fifth", "sixth")
Stcum # matrix of parameters for continuous random variables

##           mean    sd   skew skurtosis  fifth sixth
## Logistic 0.000 1.814 0.0000    1.2000  0.000 6.857
## Weibull  4.465 1.623 0.1681   -0.2705 -0.105 0.595

# Six is a list of vectors of correction values to add to the sixth cumulants
# if no valid pdf constants are found
Six <- list(seq(1.7, 1.8, 0.01), seq(0.10, 0.25, 0.01))
marginal <- list(0.3) # cum prob for Weibull rv, Logistic assumed = 1
lam <- 0.5 # Constant for Poisson rv
size <- 2 # Size parameter for Negative Binomial RV
prob <- 0.75 # prob of success
Rey <- matrix(0.4, 5, 5)  # target correlation matrix: the order is important
diag(Rey) <- 1
# Make sure Rey is within upper and lower correlation limits
# and that a valid power method exists
valid <- valid_corr(k_cat = 1, k_cont = 2, k_pois = 1, k_nb = 1,
                    method = "Polynomial", means = Stcum[, 1],
                    vars = Stcum[, 2]^2, skews = Stcum[, 3],
                    skurts = Stcum[, 4], fifths = Stcum[, 5],
                    sixths = Stcum[, 6], Six = Six, marginal = marginal,
                    lam = lam, size = size, prob = prob, rho = Rey,
                    seed = seed)

## 
##  Constants: Distribution  1  
## 
##  Constants: Distribution  2  
## All correlations are in feasible range!

Simulate the data

Sim1 <- rcorrvar(n = n, k_cat = 1, k_cont = 2, k_pois = 1, k_nb = 1,
                 method = "Polynomial", means = Stcum[, 1],
                 vars = Stcum[, 2]^2, skews = Stcum[, 3],
                 skurts = Stcum[, 4], fifths = Stcum[, 5],
                 sixths = Stcum[, 6], Six = Six, marginal = marginal,
                 lam = lam, size = size, prob = prob, rho = Rey,
                 seed = seed)

## 
##  Constants: Distribution  1  
## 
##  Constants: Distribution  2  
## 
## Constants calculation time: 0.141 minutes 
## Intercorrelation calculation time: 0.008 minutes 
## Error loop calculation time: 0 minutes 
## Total Simulation time: 0.149 minutes

Unpack the simulated data

ord_rv <- Sim1$ordinal_variables; names(ord_rv) <- "cat_rv"
cont_rv <- Sim1$continuous_variables; names(cont_rv) <- c("logistic_rv", "Weibull_rv")
pois_rv <- Sim1$Poisson_variables; names(pois_rv) <- "Poisson_rv"
neg_bin_rv <- Sim1$Neg_Bin_variables; names(neg_bin_rv) <- "Neg_Bin_rv"
fake_data <- tibble(ord_rv, cont_rv, pois_rv, neg_bin_rv)
fake_data

## # A tibble: 10,000 x 5
##    cat_rv logistic_rv Weibull_rv Poisson_rv Neg_Bin_rv
##     <dbl>       <dbl>      <dbl>      <dbl>      <dbl>
##  1      1      -0.496       2.29          0          3
##  2      2       1.67        4.39          0          0
##  3      2       2.48        5.43          0          1
##  4      2      -1.02        3.86          2          2
##  5      2       4.26        5.98          1          2
##  6      2       0.945       2.33          0          1
##  7      1      -4.02        3.45          0          0
##  8      2      -1.31        7.78          1          1
##  9      1      -3.87        1.85          0          0
## 10      1       1.01        6.28          0          1
## # … with 9,990 more rows

Well, that’s it for now. My short list of R packages for simulating data is far from being exhaustive and does not include some really good stuff, but it covers the basics. Hopefully, it will motivate you to acquire the good habit of using simulation to learn more about your data, or make your life easier if you have already acquired this habit. I hope to return to this topic again in the near future, and explore simulating survival data.

If you have any favorite R packages for simulating data please let me know.

July 2020: "Top 40" New CRAN Packages

Thu, 27 Aug 2020 00:00:00 +0000

One hundred sixty-one new packages made it to CRAN in July. Here are my “Top 40” picks in seven categories: Computational Methods, Data, Genomics, Machine Learning, Science, Statistics, and Utilities.

Computational Methods

libgeos v3.8-1-3: Implements API access the Open Source Geometry Engine GEOS which can be used to write high-performance C and C++ geometry operations. Look here for help.

gms v0.4.0: Implements a collection of tools to create and maintain modularized model written in the GAMSmodeling language.

LoopDetectR v0.1.2: Provides functions to detect feedback loops (cycles, circuits) between species (nodes) in ordinary differential equation (ODE) models. See Thomas & Kaufman (2002) for background and the vignette for information on how to use the package.

mrgsim.parallel v0.1.1: Provides a parallel backend for the mrgsolve ODE solver. Look here for an example.

paropt v0.1: Uses the SUNDIALS suite of nonlinear differential/algebraic equation solvers to optimize the parameters of ordinary differential equations. There is a vignette.

Data

chirps v0.1.2: Implements an API client for the Climate Hazards Group InfraRed Precipitation with Station CHIRPS data: 35+ years of satellite imagery, and in-situ station data used to create gridded rainfall time series for trend analysis and seasonal drought monitoring. See the vignette for an example.

covid19mobility v0.1.1: Provides COVID-19 mobility data scrapped from different sources including Google, Apple and others. There are vignettes on Animating Covid-19 Mobility Data, How mobility data has changed in cities, Work versus Home, and US mobility trends.

covidregionaldata v0.5.0: Provides access to daily COVID-19 time series data including cases, deaths, hospitalizations, and tests for several countries and subnational regions. See README to get started.

fec16 v0.1.1: Provides access to relational data from the United States 2016 federal election cycle as reported by the Federal Election Commission including information about candidates, committees, and a variety of different financial expenditures. See the vignette for details.

oxcovid19 v0.1.1: Provides an interface to the OxCOVID19 Database. There are vignettes on Database Access, The R API, and Visualization for China.

palmerpenguins v0.1.0: Provides size measurements, clutch observations, and blood isotope ratios for adult foraging Adélie, Chinstrap, and Gentoo penguins observed on islands in the Palmer Archipelago near Palmer Station, Antarctica. Look here to get started.

Genomics

bioseq v0.1.1: Provides a toolbox for manipulating DNA, RNA and amino acid sequences including functions for detection, selection, replacement, transciption and translation. There is an Introduction and a vignette on Database Preparation.

singleCellHaystack v0.3.2: Implements the singleCellHaystack algorithm as described in Vandenbon & Diez (2019) for finding differentially expressed genes in single-cell transcriptome data. The vignette offers and example.

Machine Learning

image.binarization v0.1.1: Implements algorithms to improve optical character recognition by binarizing images (turn a color or gray scale image into a black and white image). Look here for and example.

image.ContourDetector v0.1.0: Implements the unsupervised smooth contour detection algorithm described in von Gioi & Randall (2016). Look here for an example.

image.CornerDetectionF9 v0.1.0: Implements the FAST-9 corner detection algorithm explained in Rosten et al. (2008). Look here for an example.

image.CornerDetectionHarris v0.1.1: Implements the Harris Corner Detection algorithm described in Sánchez et al (2018). Look here for background, and here for an example.

image.LineSegmentDetector v0.1.0: Implements the line segment detector algorithm described in von Gioi et al (2012). Look here for an example.

Medicine

freqtables v0.1.0: Provides functions to make tables of descriptive statistics (i.e., counts, percentages, confidence intervals) for categorical variables. Designed for the Tidyverse pipeline, it also provides functions to write results into Microsoft Word ® documents. There is a vignette on the Tidyverse pipeline and another on using the freq_test() function.

nbTransmission v1.1.1: Provides functions to estimate the relative transmission probabilities between cases in an infectious disease outbreak. See Leavitt et al. (2020) for details, and the vignette for an Introduction.

precautionary v:0.1-2: Provides functions that enhance the design and simulation of phase 1 dose-escalation trials by adding diagnostics to examine the safety characteristics of these designs in light of expected inter-individual variation in pharmacokinetics and pharmacodynamics. See Norris (2020) for background and the vignette for an Introduction.

pspline.inference v1.0.2: Provides tools for making inferences about infectious disease outcomes using generalized additive (mixed) models with penalized basis splines (P-Splines). See Weinberger et al. (2020) for background and the vignette to get started.

SITH v1.0.1: Implements a three-dimensional stochastic model of cancer growth and mutation similar to the one described in Waclaw et al. (2015) and allows for interactive 3D visualizations of the simulated tumor. See the vignette for an Introduction.

Science

apsimx v1.946: Implements an interface to the APSIM framework for agricultural systems modeling and simulation. There is an Introduction and a vignette on Writing Scripts .

cmstatr v0.7.0: Implements the statistical methods commonly used for advanced composite materials in aerospace applications focusing on calculating basis values (lower tolerance bounds) for material strength properties, as well as performing the associated diagnostic tests. See Kloppenborg (2020) for details and the vignettes for a Tutorial, examples of Plotting Composite Material Data and the Anderson-Darling Test.

hadron v3.1.0: Provides a tool kit to perform statistical analyses of correlation functions generated from Lattice Monte Carlo simulations including functions to extract hadronic quantities from Lattice Quantum Chromodynamics simulations (Boucaud et al. (2008)), to determine energy eigenvalues of hadronic states (Blossier et al. (2009), and Fischer et al. (2020)). There are vignettes on the Two Amplitudes Model, GEVP energy level extraction, The Hankel Method, Jackknife Covariance and Missing Values, and Jackknife Error Normalization.

sarp.snowprofile v1.0.0: Provides analysis and plotting tools for snow profile data produced from manual snowpack observations and physical snowpack models. The functions read multiple data formats, manipulate data, and produce stratigraphy and time series profiles. See the vignette for more information.

Statistics

fddm v0.1-1: Implements the Diffusion Decision Model of Ratcliff & McKoon (2008) with across tial variable drift rate. It includes C++ implementations of the approximations of Navarro & Fuss (2009) and Gondan et al. (2014). There are vignettes on Benchmark Testing, Model Fitting, Mathematical Methods, and Validation.

GGMncv v1.1.0: Provides functions to estimate Gaussian graphical models with non-convex penalties, including atan, seamless L0, exponential, smooth integration of counting and absolute deviation, logarithm, Lq, smoothly clipped absolute deviation, minimax concave penalty, Lasso, and Adaptive lasso. See README for examples.

LTRCforests v0.5.0: Implements the conditional inference forest and random survival forest algorithm to modeling left-truncated right-censored data with time-invariant covariates, and (left-truncated) right-censored survival data with time-varying covariates. See Yao et al. (2020).

MoMPCA v1.0.0: Implements a method to cluster any count data matrix with a fixed number of variables, such as document/term matrices. Inference is done by means of a greedy Classification Variational Expectation Maximisation (C-VEM) algorithm. See Jouvin et. al. (2020) for more details and the vignette for an example..

nortsTest v1.0.0: Implements four tests, Lobato and Velasco’s, Epps, Psaradakis and Vavra, and the random projections tests for assessing the normality of stationary process. See README for details.

sptotal v0.1.0: Provides functions for predicting totals and weighted sums, or finite population block kriging, on spatial data using the methods in Ver Hoef (2008). See the vignette to get started.

ztpln v0.1.0: Provides functions for obtaining the density, random variates, and maximum likelihood estimates of the Zero-truncated Poisson lognormal distribution and its mixture distributions. See the vignette for details.

Utilities

cpp11 v0.2.1: Provides a header only, C++ 11 interface to R’s C interface and strives to be safe against long jumps from the C API and C++ exceptions, and to conform to normal R function semantics and support interactions with ALTREP vectors. There are vignettes on Motivations, Getting Started, cpp11 Internals, and Converting from Rcpp.

listdown: Provides functions to programmatically create R Markdown documents from lists. There is a vignette.

oysteR v0.0.3: Provides functions to discover third party packages used in an R package and scan them for vulnerabilities using the sonatype OSS INDEX. Look here for information to get started.

rbibutils v1.0.3: Provides functions to convert between a number of bibliography formats, including BibTeX, BibLaTeX and Bibentry, and includes a port of the bibutils utilities by Chris Putnam. Look here for examples.

stickr v0.3.1: Lets users download and use R hex stickers available in the hex-stickert repository.

supreme v1.1.0: Implements a tool to help developers to visualize and understand the structure of Shiny applications. Look here for examples.

Monitor COVID-19 at the COVID-19 Forecast Hub

Mon, 10 Aug 2020 00:00:00 +0000

If you are looking for a place to monitor expert forecasts for United States weekly and cumulative COVID-19 deaths, you can’t do any better than the Reich Lab (University of Massachusetts) COVID-19 Forecast Hub. The same is true for data scientists who may be seeking to publish their own forecasts and compare them with the work of their peers. Every Tuesday morning, the Hub publishes four-week, national and state level forecasts from over thirty-five different groups along with its own ensemble forecast. These forecasts can be examined via an effective D3 interactive visualization, and are also available in a public GitHub repository. The ensemble forecast and all of the submitted forecasts are also passed on to the CDC and the FiveThirtyEight forecast tracker.

The visualization lets users select form a menu of forecasts to compare. By clicking on points marking the actual observed values on past dates, users can compare the accuracy of the various forecasts.

In addition to the visualization, the COVID-19 Forecast Hub has several notable features:

The COVID-19 Forecast Hub team encourages participation from any group that meets its standards, and provides a showcase for the Community.
It conducts several automated tests, as well as some “human in the loop” tests, to screen forecasts for inclusion in the ensemble. For this reason, you are unlikely to see forecasts that appear to float untethered from the actual data, or take off on precipitous increases or drastic decreases that appear to be “dramatically out of line with the historical data”.
In addition to being available in the GitHub repository mentioned above, the details of the forecasts can also be downloaded programmatically from the Zoltar API.
The ensemble is created by algorithms that score each model and average twenty-three quantiles of the predictive distribution produced by each included forecast.
The submitted forecasts included in the ensemble cover a wide range of modeling methodologies and model types. There are epidemiological models conditioned on various assumptions about disease transmission and public behavior, as sell as unconditional time series and “curve fitting” models. There are SEIR models, deep learning models, agent based simulations and unique hybrid models.
In addition to the ensemble, the COVID-19 Forecast Hub team also produces a simple, but surprisingly accurate baseline forecast. (The median predictions of incidence at future time points is the most recently observed incidence.)

For a detailed overview of the statistics and data science underlying the ensemble forecast and COVID-19 Forecast Hub have a look at video of the presentation Nicholas Reich recently gave at the ASA-JDS Webinar Series.

Finally, when you visit the COVID-19 Forecast Hub, please plan to spend some time examining the individual forecasts. The statistical imagination and technique on display is astounding, and there is quite a bit of data science to be learned. This effort is representative of the thousands of statisticians, data scientists, programmers and researchers worldwide who are giving their best to help control this pandemic.

June 2020: "Top 40" New CRAN Packages

Mon, 27 Jul 2020 00:00:00 +0000

Two hundred ninety new packages made it to CRAN in June. Here are my “Top 40” picks in ten categories: Computational Methods, Data, Genomes, Machine Learning, Medicine, Science, Statistics, Time Series, Utilization, and Visualization.

Computational Methods

Rfractran v1.0 Implements the esoteric, Turing complete FRACTRAN programming language invented by John Horton Conway.

QGameTheory v0.1.2: Provides a general purpose toolbox for simulating quantum versions of game theoretic models See Flitney and Abbott (2002) for background. Models include the Penny Flip Game Meyer (1998), the Prisoner’s Dilemma Grabbe (2005), Two Person Duel Flitney and Abbott (2004), Battle of the Sexes Nawaz and Toor (2004), Hawk and Dove Game Nawaz and Toor (2010), Newcomb’s Paradox Piotrowski and Sladkowski (2002) and the Monty Hall Problem Flitney and Abbott (2002). Look here for and introduction to the package.

Data

covid19dbcand v0.1.0: Provides access seventy-five Drugbank data sets containing information about possible treatment for COVID-19.

tidytuesdayR v1.0.1: Provides functions for downloading the Tidy Tuesday data sets from R for Data Science Online Learning Community repository

us.census.geoheader v1.0.2: Implements a simple interface to the Geographic Header information from the 2010 US Census Summary File 2. See the vignette for details.

Genomics

dnapath v0.6.4: Provides functions to integrate pathway information into the differential network analysis of two gene expression datasets as described in Grimes et al. (2019). There is an Introduction and a vignette on Datasets.

TreeDist v1.1.1: Implements measures of tree similarity, including the information-based generalized Robinson-Foulds distances Smith, (2020), the Nye et al. (2006) metric and other additional metrics. There are several vignettes: Generalized Robinson-Foulds distances, Extending the Robinson-Foulds metric, Calculate tree similarity, Comparing splits using information theory, and Contextualizing tree distances.

volcano3D v1.0.1: Implements interactive plotting for three-way differential expression analysis which is useful for discovering quantitative changes in expression levels between experimental groups. See Lewis et al. (2019) for background and the vignette.

Machine Learning

boundingbox v1.0.1: Provides functions to generate bounding boxes for image classification. See Ibrahim et al. (2012) for background and the vignette for and introduction to the package.

corels v0.0.2: Implements the Certifiably Optimal RulE ListS (Corels)’ learner described in Angelino et al. (2017) which provides interpretable decision rules with an optimality guarantee. README contains an example.

nntrf v0.1.0: Implements non-linear dimension reduction by means of a neural network with hidden layers which can be useful as data pre-processing for machine learning methods that do not work well with many irrelevant or redundant features. See Rumelhart et al. (1986) for background and the vignette.

permimp v1.0-0: Implements an add-on to the party package, with a faster implementation of the partial-conditional permutation importance for random forests. See Strobl et al. (2007) for background and the vignette for an introduction.

pluralize v0.2.0: Provides tools based on a JavaScript library to create plural, singular, and regular forms of English words along with tools to augment the built-in rules to fit specialized needs. See the vignette for examples.

triplot v1.3.0: Provides model agnostic tools for exploring effects of correlated features in predictive models and calculating the importance of the groups of explanatory variables. Biecek (2018) for details and look here for an example.

tfaddons v0.10.0: Provides and interface to TensorFlow Addons. See the vignette for an example.

Medicine

BayesianReasoning v0.3.2: Provides functions to plot and help understand positive and negative predictive values (PPV and NPV), and their relationship with sensitivity, specificity, and prevalence. See Akobeng (2007) for a theoretical overview and Navarrete et al. (2015) for a practical explanation. There is an Introduction and a vignette on Screening Tests.

riskCommunicator v0.1.0: Provides functions to estimate flexible epidemiological effect measures including both differences and ratios using the parametric G-formula. See Robbins (1986) and Ahern et al. (2009) for background. There is an Introduction and a vignette for newbie R users.

Science

actel v1.0.0: Designed for studies where fish tagged with acoustic tags are expected to move through receiver arrays, this package combines the advantages of automatic sorting and checking of fish movements with the possibility for user intervention on tags that deviate from expected behavior. Calculations are based on Perry et al. (2012). There are an astounding seventeen vignettes including: Preparing your data, Structruing the Study Area, and Explore.

safedata v1.0.5: Provides access to data from the SAFE Project, a large scale ecological experiment in Malaysian Borneo that explores the impact of habitat fragmentation and conversion on ecosystem function and services. There is an Overview and an Introduction.

Statistics

causact v0.3.2: Built on greta and TensorFlow, this package enables users to define probabilistic models using directed acyclic graphs. See README for examples.

frechet v0.1.0: Provides implementation of statistical methods for random objects lying in various metric spaces, which are not necessarily linear spaces including Fréchet regression for random objects with Euclidean predictors. See Petersen and Müller (2019) for the theory.

hmclearn v0.0.3: Provide a framework for learning the intricacies of the Hamiltonian Monte Carlo. See Michael (2017) and Thomas and Tu (2020) for background. There are vignettes on Linear mixed effects, linear regression, logistic mixed effects, logistic regression, and poisson regrression.

mashr v0.2.38: Implements the multivariate adaptive shrinkage (mash) method of Urbut et al. (2019) for estimating and testing large numbers of effects in many conditions (or many outcomes) There is an Introduction and vignettes on eQTL studies, Correlations, Covariances, Common Baseline, No Common Baseline, Sampling from Posteriors, and Simulation.

molic v2.0.1: Implements the method of Lindskou et al. (2019) to detect outliers in high dimensional, categorical data. There are vignettes on the Outlier Model, Detecting Skin Diseases, and Genetic Data.

multinma v0.1.3: Uses Stan to fit network meta-analysis and network meta-regression models for aggregate data, individual patient data, and mixtures of both. See Phillippo et al. (2020) for background and the vignettes for examples: Stroke prevention, BCG Vaccine for Tuberculosis, Beta Blockers, Diabetes, Dietary Fat, Parkinson’s disease, Plaque Psoriasis, Smoking Cessation, Statins, Thrombolytic Treatments, and neutropenia or neutrophil dysfunction.

SCOUTer v1.0.0: Offers a new approach to simulating outliers by generating new observations defined by the statistics: Squared Prediction Error (SPE) and Hotelling’s $T^{2}$ statistic. See the vignette.

Time Series

bootUR v0.1.0: Provides functions to perform various bootstrap unit root tests for both individual time series (including augmented Dickey-Fuller test and union tests), multiple time series and panel data. See Palm et al. (2008) for background, and the vignette for an introduction and extensive references.

ChangePointTaylor v0.1.0: Implements the change in mean detection method described in Taylor (2000). See the vignette.

LOPART Implements the change point detection algorithm described in Hocking and Srivastava (2020).

modeltime v0.0.2: Implements a time series forecasting framework for use with the tidymodels ecosystem, and ARIMA, Exponential Smoothing, and time series models from the forecast and prophet See Forecasting Principles & Practice, and Prophet: forecasting at scale for background. These is a Getting Started Guide and vignettes describing Extension and the Model List.

Utilities

knitrdata v0.5.0: Implements a data language engine for incorporating data directly in ‘rmarkdown’ documents so that they can be made completely standalone. See the vignette for details.

lazyarray v1.1.0: Implements multi-threaded, serialized compressed arrays that fully utilizes modern solid state drives that allow users to quickly store large data while using limited memory. A lazy-array can be shared across multiple R sessions and multiple R sessions can simultaneously write to a same array. For more information, look here.

officedown v0.2.0: Provides functions to produce Microsoft Word documents from R Markdown. There are vignettes on Captions and References, Lists, officer Support, Data Frame Printing, and YAML Headers.

r2dictionary v0.1: Allows users to directly search for definitions of terms from within the R environment. The source dictionary is an original work of The Online Plain Text English Dictionary (OPTED). See the vignette.

rmdpartials v0.5.8: Enables the use of rmarkdown partials (knitr child documents) for making components of HTML, PDF and Word documents. See the vignette to get started.

tidycat v0.1.1: Provides functions to create additional rows and columns on broom::tidy() output to allow for easier control on categorical parameter estimates. The vignette contains examples

Visualization

ggdist v2.2.0: Provides primitives for visualizing distributions using ggplot2 that are tuned for visualizing uncertainty in either a frequentist or Bayesian mode. Primitives include points with multiple uncertainty intervals, eye plots Spiegelhalter (1999), density plots, gradient plots, dot plots Wilkinson (1999), quantile dot plots Kay et al. (2016),, complementary cumulative distribution function barplots, Fernandes et al. (2018), and fit curves with multiple uncertainty ribbons.

loon.ggplot v1.0.1: Provides a bridge between the loon and ggplot2 packages. Users can turn static ggplot2 plots into interactive loon plots and vice versa. There are vignettes on ggplots -> loon, loon -> ggplots, and on u sing pipes.

treeheatr v0.1.0: Provides interpretable decision tree visualizations with the data represented as a heatmap at the tree’s leaf nodes. There is a vignette.

tilemaps v0.2.0: Implements the algorithm of McNeill and Hale (2017) for generating tilemaps. See the vignette.

wrGraph v1.0.2: Provides enhancements to base R graphics for plotting high-throughput data including automatic segmenting of the current device (e.g. window) to accommodate multiple new plots, automatic checking for optimal location of legends in plots, small histograms inserted as legends, the generation of mouse-over interactive html pages and more. See the vignette.

Open-Source Authorship of Data Science in Education Using R

Wed, 01 Jul 2020 00:00:00 +0000

Joshua M. Rosenberg, Ph.D., is Assistant Professor of STEM Education at the University of Tennessee, Knoxville.

In earlier posts, we shared how we wrote Data Science in Education Using R as an open book (Post 1, Post 2). In this post, we describe what we consider to be the open-source authorship process we took to write the book.

We think of open-source authorship as a broader—and perhaps better—term for describing what authors of some open books undertake. In our characterization, open-source authorship draws upon:

parts of open-source software (OSS) values and tools
parts of open science that establish the importance of scholarly work beyond original, discovery research
the values surrounding the creation of open educational resources (OER)

We believe that combining elements from OSS, open science, and OER is notable because while OSS and open science emphasize the sharing of technical work (including technology and code) and OER emphasizes the sharing of resources, technical books have not been as much a focus of the conversation. Moreover, the way in which the conversation about open books has taken place in different communities and contexts means that some books that are open do not fully receive the attention (for being openly available) that they merit from those interested in OER. This also might mean that those involved with OSS development and open science may fail to recognize the creation of a book as a substantial contribution.

In this way, we argue for open-source authorship as an important, new type of work, one that we increasingly see by the authors of other books, especially in the R community¹ ² ³ ⁴.

After describing how we wrote our book in an open way, we elaborate on these ideas and draw connections to the process we undertook.

How We Wrote Data Science in Education Using R as an Open Book

Early in our process, we determined that we wanted to share the book in an open way. Since we were using GitHub as a repository for the book, it was easy for the contents of the book to be available for anyone to view–even before and as the book was being written. Despite the benefits of using GitHub, GitHub can be difficult to navigate for those who are unfamiliar with it, and so sharing the book in a more widely-accessible way was also important. To do this, we used {bookdown} and Netlify to share the book as a website. Additionally, we chose an easy-to-remember URL (http://datascienceineducation.com/) to help others (and us!) to be able to access it easily.

Being available for others to contribute was important. Because we used GitHub, we were able to receive feedback at a very early-stage on issues such as how we referred to data (as data or datum). Other issues (by non-authors) raised questions about whether certain content was in scope—such as content on gradebooks, which we included a chapter on. We found that apart from the five of us as authors, fifteen individuals made contributions, and another one hundred forty-four individuals starred the repository⁵. Moreover, we received feedback through Twitter and an email account we created for the book for those unfamiliar with GitHub (or Twitter) to be able to provide feedback directly to us. In this way, making the book available to others to contribute made the book better, and points to the importance of sharing work at only one stage of the writing process.

Lastly, we shared products that could be seen as tangential to the book, but which were important given its focus on data science and R. Namely, we created an R package, {dataedu}, to accompany the book. This package includes code to install the packages necessary to reproduce the book as well as all of the data sets used in it. By doing so, we invited others to contribute to the book in ways not related to its prose. This also led to (pleasantly) surprising contributions, including the creation of an iPython Notebook with python code to carry out comparable steps as those carried in a walkthrough chapter of our book.

Collectively, these practices—involving not only making the book open, but also planning for others to contribute and creating other, shared (open) products— comprise what we think of as the results of open-source authorship.

Originally a niche effort, open-source software (OSS) and OSS development are (likely not to the surprise of R users!) now widespread ⁶. There are some insights that can be gained from efforts to understand how OSS development proceeds. For example, in foundational, work Mockus et al. found that OSS is often characterized by a core group of 10-15 individuals contributing around 80% of the code, but that a group around one order of magnitude larger than that core will repair specific problems, and a group another order of magnitude larger will report issues⁷; proportions (generally) similar to those we found for those who contributed to our book.

Second, open science is both a perspective about how science should operate and a set of practices that reflect a perspective about how science should proceed ⁸ ⁹. Related to open science are open scholarly practices. Others trace the origin of the idea of open scholarly practices to a book by Boyer, who shared a broad description of intellectual (especially academic) work. This suggests that scholarly work is not only original, discovery research; it also includes the applications of advances in one’s own discipline (or “translational research”) and sharing the results of research with multiple stakeholders. Open science and open scholarly practices point to the scientific or scholarly contributions of open books; while different from original, scientific research, books such as our own—which focused on providing a language for data science in education—may serve as helpful examples (of open science) or forms of a broader view of scholarship.

Last, OER are “teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others” ¹⁰. These resources range from courses and books to tests and technologies. By being open, they are not only available to others to use, but also to reuse, redistribute (or share), revise (adapt or change the work), and remix (combining existing resources to create a new one) ¹¹. OER can serve as an inspiration for authors of open books, especially those who see their books as being used to teach and learn from. At the moment, OER and traditional publishing modes are largely separate: For most books that are published, the publisher retains the copyright, and authors are typically not allowed to share their book in the open, though this may be changing. Many authors of books about R have negotiated with their publisher to share their books in the open (often only as a website, as we have) in addition to sharing them through print and e-book formats. In addition, a number of platforms for creating books that are OER are emerging; one example is EdTech Books. There are increasing conversations related to making materials, resources, and even education as an enterprise more open; OER may be an area in which authors of books about R and other technical books can both learn from the work of authors as well as advance the conversation.

fin

This post was an effort to step back from what we did to write our book to reflect on what we meant by open-source authorship and to attempt to situate what we did (and what others have done) in broader conversations about OSS, open science, and OER. In this open mode, we invite others to revise or remix these ideas to advance other, new forms of authorship of books.

You can reach us on Twitter: Emily @ebovee09, Jesse @kierisi, Joshua @jrosenberg6432, Isabella @ivelasq3, and me @RyanEs.

See you in two weeks for our next post! Josh, with help from Ryan, Emily, Jesse, Joshua, and Isabella

Ryan A. Estrellado is a public education leader and data scientist helping administrators use practical data analysis to improve the student experience.
Emily A. Bovee, Ph.D., is an educational data scientist working in dental education.
Jesse Mostipak, M.Ed., is a community advocate, Kaggle educator, and data scientist.
Isabella C. Velásquez, MS, is a data analyst committed to nonprofit work with the aim of reducing racial and socioeconomic inequities.

May 2020: "Top 40" New CRAN Packages

Wed, 24 Jun 2020 00:00:00 +0000

One hundred eighty-four new packages stuck to CRAN in May. The following are my “Top 40” picks in eleven categories: Data, Finance, Genomics, Marketing, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Data

covid19nytimes v0.1.3: Provides accesses the NY Times Covid-19 county-level data for the US that is also available here. There is a vignette.

geodata v0.1.0: Contains small spatial datasets used to teach basic spatial analysis concepts. Datasets are based on of the GeoDa software workbook and data site.

GermaParl v1.4.2: Provides access to the GermaParl corpus of parliamentary debates of the German Bundestag maintained by the PolMine Project. The vignette introduces the corpus and package.

nhlapi v0.1.2: Retrieves and processes the data exposed by the open NHL API, including information on players, teams, games, tournaments, drafts, standings, schedules and other endpoints. There are vignettes on a Low-level API, Retrieving Player Data, and Retireving Team Data.

polAr v0.1.3: Implements a toolbox for the analysis of political and electoral data from Argentina. There are vignettes on Computing, Data Access, and Displaying Results.

rKolada v0.1.3: Provides methods for downloading and processing data and metadata from Kolada, the official Swedish regions and municipalities database. There is an Introduction and a Quick Start Guide.

Finance

strand v0.1.3: Provides a framework for performing discrete (share-level) simulations of investment strategies. Simulated portfolios optimize exposure to an input signal subject to constraints such as position size and factor exposure. The vignette on Backtesting with strand is nicely done.

TwitterAutomatedTrading v0.1.0: Provides access to the MetaTrader 5 platform that enables users to carry out automated trading using sentiment indexes computed from twitter and/or stockwits. See Godeiro (2018) for background, and the vignette for how to use the package.

Genomics

immunarch v0.6.5: Provides a framework for bioinformatics exploratory analysis of bulk and single-cell T-cell receptor and antibody repertoires that includes data loading, analysis and visualization for bulk and single-cell AIRR (Adaptive Immune Receptor Repertoire) data. There is an Introduction and a vignette on Working with Data.

SubtypeDrug v0.1.0: Implements a tool to prioritize cancer subtype-specific drugs by integrating genetic perturbation, drug action, biological pathway, and cancer subtype. See Han et al. (2019) for background and the vignette for details on the package.

TransPhylo v1.4.4: Provides functions to reconstruct infectious disease transmission using genomic data. See Didelot et. al (2014) and Didelot et. al (2017) for background. See the Introduction and the vignettes: Inference of transmission tree from a dated phylogeny, Simultaneous Inference of Multiple Transmission Trees and Simulation of outbreak data.

Marketing

CLVTools v0.5.0: Implements various probabilistic latent customer attrition models for non-contractual settings (e.g., retail business) with and without time-invariant and time-varying covariates. See Schmittlein et al. (1987) and Fader et al. (2005) for background and the vignette to get started.

grizbayr v1.2.2: Provides functions to implement Bayesian A / B and Bandit marketing tests. See Stucchio (2015) for background and the vignette to get started.

Machine Learning

applicable v0.0.1.1: Provides functions that measure the amount of extrapolation new samples can have from the training set which are based on the concept of applicability domains. See Netzeva et al (2005). There are vignettes for binary and continuous data.

piRF v0.1.0: Implements multiple state-of-the-art prediction interval methodologies for random forests including quantile regression intervals, out-of-bag intervals, bag-of-observations intervals, one-step boosted random forest intervals, bias-corrected intervals, high-density intervals, and split-conformal intervals. Look here for an example.

rules v0.0.2: Provides bindings that allow prediction rule ensembles, C5.0 rules, and Cubist to be used with the parsnip package.

Medicine

AdhereRViz v0.1.0: Implements a Shiny based GUI to the AdhereR package to allow users to access different data sources, explore patterns of medication, and compute various measures of adherence. See the vignette for details.

MrSGUIDE v0.1.1: provides functions to facilitate subgroup analysis for single and multiple responses in both randomized trials and observational studies based on the GUIDE algorithm. See the Vignette.

Science

ldsr v0.0.2: Provides functions to reconstruct streamflow and climate information using linear dynamical systems. See Nguyen and Galelli (2018) for background and the vignette for examples.

rties v5.0.0: Provides tools for investigating temporal processes in bivariate (e.g., dyadic) systems. The theoretical background can be found in Butler (2011) and Butler & Barnard (2019). There is an Overview, and vignettes on Intertia Coordination, Data Preparation, and System Varibles.

Statistics

Compack v0.1.0: Implements regression methodologies with compositional covariates, including sparse log-contrast regression with compositional covariates proposed by Lin et al. (2014), and sparse log-contrast regression with functional compositional predictors proposed by Sun et al. (2020). There is a vignette.

ghypernet v1.0.0: Provides functions for model fitting and selection of generalized hypergeometric ensembles of random graphs (gHypEG). The package is based on the research by Casiraghi and collaborators. For example, see Casiraghi et al. (2016), Casiraghi (2017) and Casiraghi and Nanumyan (2018). There is an Introduction, a short Tutorial, and a vignette on Finding Significant Links.

motifcluster v0.1.0: Provides tools for spectral clustering of weighted directed networks using motif adjacency matrices. These methods, which perform well on large and sparse networks, are based on the methodology described in Underwood et al. (2020). See the vignette.

regmedint v0.1.0: Implements the regression-based causal mediation analysis with a treatment-mediator interaction term, as originally implemented in the SAS macro described in Valeri and VanderWeele (2013) and Valeri and VanderWeele (2015). There is an Introduction, and vignettes on Implementing Formulas, Bootstrapping, and Multiple Imputation.

Time Series

DeCAFS v3.1.5: Provides functions to detect abrupt changes in time series with local fluctuations as a random walk process and autocorrelated noise as an AR(1) process. See Romano et al. (2020) for the theory.

Rdrw v1.0.1: Provides functions to fit and simulate a univariate or multivariate damped random walk process (also known as an Ornstein-Uhlenbeck process or a continuous-time autoregressive model of the first order) which is suitable for analyzing time series data with irregularly-spaced observation times and heteroscedastic measurement errors. See Hu and Tak (2020) for background.

statespacer v0.1.0: Provides functions for estimating time series using the state space method. For background see JSS Vol 41. The package has an Introduction, a Dictionary for the model object and vignettes on Fitting and ARIMA Model, an Example and on Specifying a new model component.

Utilities

almanac v0.1.1: Provides tools for implementing recurrence rules, i.e. functions for defining recurring events. There is an Introduction and vignettes on Adjusting and Shifting Dates, iCalendar Specification, and Quarterly Rules.

gdiff v0.2-1: Provides functions for performing graphical difference testing. Look here for more information.

i2dash v0.2.1: Provides functions for creating web-based dashboards. See the vignette.

pkgndep v1.0.0: Provides functions to check and visualize the “heaviness” of R packages. See the vignette.

presser v1.0.0: Implements the httpbin.org web service and functions to test web clients without using the internet.

stringfish v0.12.1: Implements a framework for performing string and sequence operations using the alt-rep system to speed up the computation of common string operations. See the vignette.

worcs Implements the Workflow for Open Reproducible Code in Science, WORCS. There is an Introduction, and vignettes on citing, git_cloud, and setup.

Visualization

ggpacman v0.1.0: Reproduces the game Pac-Man using ggplot2 and gganimate. Look here for more information.

iNZightTS v1.5.2: Provides tools for working with time series data, including functions for drawing, decomposing, and forecasting, comparing multiple series, and fitting both additive and multiplicative models. Look here for more information.

prismadiagramR v1.0.0: Provides functions to create PRISMA diagrams used to track the identification, screening, eligibility, and inclusion of studies in a systematic review. See the vignette.

sketcher v0.1.3: Implements image processing effects that convert a photo into a line drawing image. See Tsuda (2020) for background and look here for examples.

upsetjs v1.3.1: Provides an htmlwidget wrapper for the JavaScript UpSet.js library. There is an Introduction, and vignettes on Coloring, Combination Modes, and Venn and Euler Diagrams.

xaringanthemer v0.3.0: Provides functions to create custom CSS themes. There is and Overview, and vignettes on ggplot2 Themes, and Template Variables.

Learning R With Education Datasets

Thu, 11 Jun 2020 00:00:00 +0000

Ryan A. Estrellado is a public education leader and data scientist helping administrators use practical data analysis to improve the student experience.

Timothy Gallwey wrote in The Inner Game of Tennis:

…There is a natural learning process which operates within everyone, if it is allowed to. This process is waiting to be discovered by all those who do not know of its existence … It can be discovered for yourself, if it hasn’t been already. If it has been experienced, trust it.

Discovering a new R concept like a function or package is exciting. You never know if you’re about to learn something that fundamentally changes the way you code or solve data science problems. But I get even more excited when I see somebody use new R concepts. For example, I learned about random forest models when I read about them in An Introduction to Statistical Learning (ISL). Then I imagined myself using them when I watched Julia Silge fit a random forest model to predict attendance at NFL games. I need the reading to give me language for what I see data scientists do. Then I need to see what data scientists do for me to imagine myself doing what I’ve read.

Still, for most people using R in their jobs, there’s another step. They have to imagine how to apply what they’ve read and seen to the problems they’re solving at work. But what if we used education datasets to help them imagine using R on the job, just as the authors of ISL use words and code to teach about models and Julia Silge uses video to inspire coding?

We learned from writing Data Science in Education Using R (DSIEUR) that we can combine words, code, and professional context. Professional context includes scenarios, language, and data that readers will recognize in their education jobs. We wanted readers to feel motivated and engaged by seeing words and data that reminds them of their everyday work tasks. This connection to their professional lives is a hook for readers as they engage R syntax which is, if you’ve never used it, literally a foreign language.

Let’s use pivot_longer() as an example. We’ll describe this process in three steps: discovering the concept, seeing how the concept is used, and seeing how the concept is used in education.

Step 1: See the concept

When I read something like “Use pivot_longer() to transform a dataset from wide to long”, I can imagine the shape of a dataset changing. But it’s harder to imagine what happens with the variables and their contents as the dataset’s shape changes. I’ve been using R for over five years and I still struggle to visualize the contents of many columns rearranging themselves into one.

Step 2: See how the concept is used

The concept gets much clearer when you add an example—even one with little context—to the explanation. Here’s one from the pivot_longer() vignette, which you can view with vignette("pivot"):

library(tidyverse)

# Simplest case where column names are character data
relig_income

#> # A tibble: 18 x 11
#>    religion `<$10k` `$10-20k` `$20-30k` `$30-40k` `$40-50k` `$50-75k` `$75-100k`
#>    <chr>      <dbl>     <dbl>     <dbl>     <dbl>     <dbl>     <dbl>      <dbl>
#>  1 Agnostic      27        34        60        81        76       137        122
#>  2 Atheist       12        27        37        52        35        70         73
#>  3 Buddhist      27        21        30        34        33        58         62
#>  4 Catholic     418       617       732       670       638      1116        949
#>  5 Don’t k…      15        14        15        11        10        35         21
#>  6 Evangel…     575       869      1064       982       881      1486        949
#>  7 Hindu          1         9         7         9        11        34         47
#>  8 Histori…     228       244       236       238       197       223        131
#>  9 Jehovah…      20        27        24        24        21        30         15
#> 10 Jewish        19        19        25        25        30        95         69
#> 11 Mainlin…     289       495       619       655       651      1107        939
#> 12 Mormon        29        40        48        51        56       112         85
#> 13 Muslim         6         7         9        10         9        23         16
#> 14 Orthodox      13        17        23        32        32        47         38
#> 15 Other C…       9         7        11        13        13        14         18
#> 16 Other F…      20        33        40        46        49        63         46
#> 17 Other W…       5         2         3         4         2         7          3
#> 18 Unaffil…     217       299       374       365       341       528        407
#> # … with 3 more variables: `$100-150k` <dbl>, `>150k` <dbl>, `Don't
#> #   know/refused` <dbl>

relig_income %>%
 pivot_longer(-religion, names_to = "income", values_to = "count")

#> # A tibble: 180 x 3
#>    religion income             count
#>    <chr>    <chr>              <dbl>
#>  1 Agnostic <$10k                 27
#>  2 Agnostic $10-20k               34
#>  3 Agnostic $20-30k               60
#>  4 Agnostic $30-40k               81
#>  5 Agnostic $40-50k               76
#>  6 Agnostic $50-75k              137
#>  7 Agnostic $75-100k             122
#>  8 Agnostic $100-150k            109
#>  9 Agnostic >150k                 84
#> 10 Agnostic Don't know/refused    96
#> # … with 170 more rows

Sharing an idea by pairing an abstract programming concept with a reproducible example is a common practice for experienced R programmers. Community guidelines for Stack Overflow posts and the {reprex} package are two artifacts of a popular R community norm: help folks understand an idea by using words and code.

Step 3: See how the concept is used in education

Combining the explanation with a reproducible example makes pivot_longer() more concrete by showing how it works. What happens when we connect the explanation and reproducible example to the everyday work of a data scientist in education?

In chapter seven of DSIEUR, we use pivot_longer() to transform a dataset of coursework survey responses from wide to long. Before using pivot_longer(), the dataset had a column for each survey question. When we use pivot_longer(), the name of each survey question moves to a new column called “question”. Another new column is added, “response”, which contains the corresponding response to each survey question.

To run this code, you’ll need the DSIEUR companion R package, {dataedu}:

# Install the {dataedu} package if you don't have it
# devtools::install_github("data-edu/dataedu")
library(dataedu)

Here’s the survey data in its original, wide format:

# Wide format
pre_survey

#> # A tibble: 1,102 x 12
#>    opdata_username opdata_CourseID Q1Maincellgroup… Q1Maincellgroup…
#>    <chr>           <chr>                      <dbl>            <dbl>
#>  1 _80624_1        FrScA-S116-01                  4                4
#>  2 _80623_1        BioA-S116-01                   4                4
#>  3 _82588_1        OcnA-S116-03                  NA               NA
#>  4 _80623_1        AnPhA-S116-01                  4                3
#>  5 _80624_1        AnPhA-S116-01                 NA               NA
#>  6 _80624_1        AnPhA-S116-02                  4                2
#>  7 _80624_1        AnPhA-T116-01                 NA               NA
#>  8 _80624_1        BioA-S116-01                   5                3
#>  9 _80624_1        BioA-T116-01                  NA               NA
#> 10 _80624_1        PhysA-S116-01                  4                4
#> # … with 1,092 more rows, and 8 more variables: Q1MaincellgroupRow3 <dbl>,
#> #   Q1MaincellgroupRow4 <dbl>, Q1MaincellgroupRow5 <dbl>,
#> #   Q1MaincellgroupRow6 <dbl>, Q1MaincellgroupRow7 <dbl>,
#> #   Q1MaincellgroupRow8 <dbl>, Q1MaincellgroupRow9 <dbl>,
#> #   Q1MaincellgroupRow10 <dbl>

The third through eighth columns are named after each survey question—“Q1MaincellgroupRow1”, “Q1MaincellgroupRow2”, “Q1MaincellgroupRow3”, etc. These are the column names we’ll be moving to a single column called “question” when the dataset transforms from wide to long.

Here’s the new dataset, where a column called “question” contains the question names and a column called “response” contains the corresponding responses:

# Pivot the dataset from wide to long format
pre_survey %>%
  pivot_longer(cols = Q1MaincellgroupRow1:Q1MaincellgroupRow10,
               names_to = "question",
               values_to = "response")

#> # A tibble: 11,020 x 4
#>    opdata_username opdata_CourseID question             response
#>    <chr>           <chr>           <chr>                   <dbl>
#>  1 _80624_1        FrScA-S116-01   Q1MaincellgroupRow1         4
#>  2 _80624_1        FrScA-S116-01   Q1MaincellgroupRow2         4
#>  3 _80624_1        FrScA-S116-01   Q1MaincellgroupRow3         4
#>  4 _80624_1        FrScA-S116-01   Q1MaincellgroupRow4         1
#>  5 _80624_1        FrScA-S116-01   Q1MaincellgroupRow5         5
#>  6 _80624_1        FrScA-S116-01   Q1MaincellgroupRow6         4
#>  7 _80624_1        FrScA-S116-01   Q1MaincellgroupRow7         1
#>  8 _80624_1        FrScA-S116-01   Q1MaincellgroupRow8         5
#>  9 _80624_1        FrScA-S116-01   Q1MaincellgroupRow9         5
#> 10 _80624_1        FrScA-S116-01   Q1MaincellgroupRow10        5
#> # … with 11,010 more rows

When you put it all together, the learning thought process is something like this:

There’s a function called pivot_longer(), which turns a wide dataset into a long dataset
pivot_longer() does this by putting multiple column names into its own column, then creating a new column that pairs each column name with a value
I can use pivot_longer() to change an education survey dataset that has question names for columns into one that has a “question” column and a “response” column

We’ll be back with the next post in about two weeks. Until then, do share with us about the people and tools that inspire you to work on collaborative projects. You can reach us on Twitter: Emily @ebovee09, Jesse @kierisi, Joshua @jrosenberg6432, Isabella @ivelasq3 and me @RyanEs.

More Select COVID-19 Resources

Wed, 03 Jun 2020 00:00:00 +0000

We are over five months into this pandemic, and it is pretty clear that almost everyone is really tired of hearing about it. I myself am totally zoomed out and have already seen too many dashboards. Nevertheless, we are in this for the long run. So from time-to-time, I think it worthwhile to continue to look for tools that can help us make some sense of the continuing stream of incoming data.

First, I would like to draw your attention to the Covid Trends animated dashboard from Physics teacher Aatish Bhatia. The epidemiologists are the experts in this domain, but it is just like a physicist to deliver on insight.

What’s unique about this dashboard is how it beautifully illustrates the consequence of exponential growth. Notice that there is no time axis on the graph. Total confirmed cases are plotted on the x axis, and new confirmed cases in the past week are plotted on the y axis. In this setup, doubling times are represented as straight lines. As you run the animation, time passes and you observe the various countries hugging the seven day doubling time line and then dropping down as the whatever counter measures they are taking get the epidemic under control. This plot makes it clear that while things are opening up in the U.S. we do not quite have the disease under control. Please do watch the short video explaining the graph.

Next, please have a look at the COVID-19 Data Hub, an open source project started by Finance Ph.D. student Emanuele Guidotti with initial IVADO arranged by David Ardia that may very well become the main repository for epidemiologists working with COVID-19 case data. Currently over sixty data sets are available.

All data sets are in a standardized format, and are well documented. Additionally, the site provides R, Python, MatLab, Julia, Node.js, Scala and Excel code to access the data. This project is an extraordinary effort that deserves community support.

Finally for today, I recommend the video recording from the first COVID-19 Data Forum webinar held on May 14, 2020. After the opening remarks by Michael Kane, Assistant Professor, Department of Biostatistics, Yale University which begin at one minute and forty seconds (1:40) into the video, there are four talks, each approximately fifteen minutes long.

The first talk: Modeling COVID19 spread and control: Data needs and challenges by Alison Hill of the Department of Organismic & Evolutionary Biology, Harvard University begins at (5:33). The second talk: Collecting and Visualizing COVID-19 Case Count Data from Multiple Open Sources by independent consultant Ryan Hafen begins at (21:26). The third talk: Spatial and Space-Time Data on COVID-19 by Orhun Aydin of esri and the Environmental Systems Research Institute University of Southern California begins at (38:49). The final talk by Noam Ross of the EcoHealth Alliance and rOpenSci which begins at (56:23), focuses on the genomic data that enables scientists to study the emergence of new diseases.

Enjoy the videos.

April 2020: "Top 40" New CRAN Packages

Thu, 28 May 2020 00:00:00 +0000

One hundred forty-eight new packages made it to CRAN in April. Here are my “Top 40” picks in nine categories: Computational Methods, Data, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

JuliaConnectoR v0.6.0: Allows users to import Julia packages and functions in such a way that they can be called directly as as R functions.

RcppBigIntAlgos: v0.2.2: Implements the multiple polynomial quadratic sieve (MPQS) algorithm for factoring large integers and a vectorized factoring function that returns the complete factorization of an integer. See Pomerance (1984) and Silverman (1987) for background and this Microsoft post for an explanation.

smoothedLasso v1.0: Implements the smoothed LASSO regression using the method of Nesterov (2005).

Data

daqape v0.3.0: Provides a variety of methods to identify data quality issues in process-oriented data. There is an Introduction.

DSOpal v1.1.0: is the DataShield implementation of Opal, the data integration application for biobanks by OBiBa, open source software for epidemiology.

epuR v0.1: Provides functions to collect data from the the economic policy uncertainty website. See the vignette.

hystReet v0.0.1: Implements an API wrapper for the Hystreet project which provides pedestrian counts for various cities in Germany. See the vignette to get started.

rGEDI v0.1.7: Provides a set of tools for downloading, reading, visualizing and processing GEDI Level1B, Level2A and Level2B data. see the vignette to get started.

Machine Learning

catsim v0.2.1: Computes structural similarity metrics for binary and categorical 2D and 3D images including Cohen’s kappa, Rand index, adjusted Rand index, Jaccard index, Dice index, normalized mutual information, or adjusted mutual information. See Thompson & Maitra (2020) for background and the vignette for an introduction.

klic v1.0.2: Implements a kernel learning integrative clustering algorithm which allows combining multiple kernels, each representing a different measure of the similarity between a set of observations. There is an Introduction.

MIDASwrappeR V0.5.1: Provides a wrapper for the C++ implementation of the MIDAS algorithm described in Bhatia et al. (2020) for graph like data. See the Introduction.

VUROCS v1.0: Calculates the volume under the ROC surface and its (co)variance for ordered multi-class ROC analysis as well as certain bivariate ordinal measures of association.

WeightSVM v1.7-4: Provides functions for subject/instance weighted support vector machines (SVM). It uses a modified version of libsvm and is compatible with e1071 package. Look here for some background.

Medicine

covid19.analytics v1.1: Provides functions to load and analyze COVID-19 data from the Johns Hopkins University CSSE data repository. It includes functions to visualize cases for specific geographical locations, generate interactive visualizations and produce a SIR model. See the vignette for an introduction.

covid19france Provides functions to import, clean and update French COVID-19 data from opencovid19-fr.

interactionR v0.1.1: Produces a publication-ready table that includes all effect estimates necessary for full reporting effect modification and interaction analysis as recommended by Knol & Vanderweele (2012), estimates confidence interval additive interaction measures using the delta method Hosmer & Lemeshow (1992), the variance recovery method Zou (2008), or percentile bootstrapping Assmann et al. (1996).

RCT v1.0.2: Provides tools to facilitate the process of designing and evaluating randomized control trials, including methods to handle misfits, power calculations, balance regressions, and more. For background see Athey et al. (2017). The vignette describes how to use the package.

Science

rasterdiv: Provides functions to calculate indices of diversity on numerical matrices based on information theory. The rationale behind the package is described in Rocchini et al. (2017). See the vignette for an extended example.

SSHAARP v1.0.0: Processes amino acid alignments from the IPD-IMGT/HLA database to identify user-defined amino acid residue motifs shared across HLA alleles, calculate the frequencies of those motifs, and generate global frequency heat maps that illustrate the distribution of each user-defined map around the globe. See the vignette for an introduction.

Statistics

BayesSampling v1.0.0: Provides functions for applying the Bayes Linear approach to finite populations with the simple random sampling, stratified simple random sampling designs, and to the ratio estimator. See Gonçalves et al. (2014) for background and the vignettes: BLE_Ratio, BLE_Reg, BLE_SRS, BLE_SSRS, and BayesSampling.

cort v0.3.1: Provides S4 classes and methods to fit several copula models including empirical checkerboard copula Cuberos et. al (2019) and the Copula Recursive Tree algorithm proposed by Laverny et. al (2020). There are vignettes on the Empirical Checkerboard Copula, the Copula Recursive Tree, the Empirical Checkerboard Copula with known margins, and the convex mixture of m-randomized checkerboards.

ExpertChoice v0.2.0: Implements tools for designing efficient discrete choice experiments. See Street et. al (2005) for some background. There is an Practical Introduction and a vignette with some theory.

genscore v1.0.2: Implements the generalized score matching estimator from Yu et al. (2019) for non-negative graphical models with truncated distributions, and the estimator of Lin et al. (2016) for untruncated Gaussian graphical models. See the vignette.

hmma v1.0.0: Provides functions to fit Bayesian asymmetric hidden Markov models. HMM-As are similar to regular HMMs, See Bueno et al. (2017) for background and the vignette for and introduction.

lmeInfo v0.1.1: Provides analytic derivatives and information matrices for fitted linear mixed effects models and generalized least squares models estimated using lme() and gls() as well as functions for estimating the sampling variance-covariance of variance component parameters and standardized mean difference effect sizes. See Pustejovsky et al. (2014) and the vignette.

metapower v0.1.0: Implements a tool for computing meta-analytic statistical power for main effects, tests of homogeneity, and categorical moderator models. Have a look at Pigott (2012), Hedges & Pigott (2004), or Borenstein et al. (2009) for background and the vignett to get started.

sasLM v0.1.3: Implements the SAS procedures for linear models: GLM, REG, ANOVA. The sasLM functions produce the same results as the corresponding SAS procedures for nested and complex models.

sdglinkage 0.1.0: Provides a tool for synthetic data generation that can be used for linkage method development. There is an Overview and vignettes on Real and Synthetic Identifiers, Gold Standard File and Linkage Files, Synthetic Data Generation and Evaluation.

starm v0.1.0: Estimates the coefficients of the two-time centered autologistic regression model described in Gegout-Petit et al. (2019). The vignette describes the theory.

Time Series

ConsReg v0.1.0: Provides functions to fit regression and generalized linear models with autoregressive moving-average (ARMA) errors for time series data. There is a vignette.

simITS v0.1.1: Implements the method of Miratrix (2020) to create prediction intervals for post-policy outcomes in interrupted time series. It provides methods to fit ITS models with lagged outcomes and variables to account for temporal dependencies and then to simulate a set of plausible counterfactual post-policy series to compare to the observed post-policy series. See the vignette.

Utilities

dreamerr v1.1.0: Implements tools to facilitate package development by providing a flexible way to check the arguments passed to functions. See the vignette for details.

flair v0.0.2: Facilitates formatting and highlighting of R source code in a R Markdown based presentation. The vignette shows how.

J4R v1.0.7: Makes it possible to create Java objects and to execute Java methods from the R environment. The JVM is handled by a gateway server which relies on the Java library j4r.jar.

waldo v0.1.0: Provides functions to compare complex R objects and reveal the key differences. It was designed primarily for use in testing packages.

Visualization

anglr v0.6.0: Extends rgl conversion and visualization functions to mesh3d to give direct access to generic 3D tools and provide a full suite of mesh-creation and 3D plotting functions. See the vignette

brickr v0.3.4: Uses tidyverse functions to generate digital LEGO models and convert image files into 2D and 3D LEGO mosaics. There are vignettes for building mosaics and for generating models from mosaics, programs, tables, and by piece type.

survCurve v1.0: Provides functions to enhance plots created with the survival and mstate packages. See the vignette for examples.

textplot v0.1.2: Provides functions to visualize complex relations in texts by displaying text co-occurrence networks, text correlation networks, dependency relationships and text clustering. The vignette provides examples.

Community and Collaboration: Writing Our Book in the Open

Tue, 26 May 2020 00:00:00 +0000

Ryan A. Estrellado is a public education leader and data scientist helping administrators use practical data analysis to improve the student experience.

Chicken Farm in the Open

In 2017, Emily Bovee, Jesse Mostipak, Joshua Rosenberg, Isabella Velásquez, and I started work on our book, Data Science in Education Using R (DSIEUR). We had two goals for DSIEUR. First, we aimed to write a practical reference for data scientists in education that helps them learn and apply R skills in their jobs. Second, we wanted to share the process with the R community by writing the book in the open on GitHub. After working together for almost three years, my co-authors and I submitted the manuscript for DSIEUR to Routledge and are now gearing up to begin editing the print version. The print version will be out from Routledge in late 2020, but you can read the online version of DSIEUR now at https://datascienceineducation.com/.

With the writing done, we’re reflecting on lessons we’ve learned from writing DSIEUR. In the coming weeks, we’ll share these reflections on R Views as a series of blog posts. These posts are about the people and tools in the R community that inspired us to do a book like DSIEUR. Think of these as our personal notes, typed up to help us organize our thoughts about what made this project possible. We’ll share in four parts:

Part 1: Teaching R Using Everyday Examples in Education

Learning R on the job presents many challenges, but one in particular sticks out. Once you start coding, it’s not obvious how to apply that code in everyday tasks at the office. We wrote DSIEUR to answer the question, “How would it feel to have a book that taught programming concepts, provided reproducible code, and used scenarios that data scientists in education recognize?” In this first post, we’ll explore how we put these elements together and what we learned in the process.

Part 2: How the R Community Inspired Us to Write About Data Science in Education

It wasn’t long before our team encountered our first writing challenge: do we describe our audience as “data scientists in education” or “education data scientists?”. The debate was a symbol for a larger dilemma–what common language do you use when projects like ours aren’t yet common? It helped that the community we were writing for inspired us to explore the topic. The things we love the most about the R community–welcoming folks from different backgrounds, a collective love of side projects, and a willingness to work in the open–made it safe for us to try new things and learn. We listened to stories from data scientists in education, spent a lot of time reading the Twitter #rstats hashtag, and invited community members to join the conversation. In this post, we’ll explore how community participation empowered our writing process.

Part 3: Writing In the Open

This post is about coordinating people and tools to write an open book, a challenging proposition for five writers who had only just met on Twitter. For instance, how would five people in different time zones write instructional materials and code together? And if coordinating five authors wasn’t hard enough, how would they invite the rest of the community to join the mission? Fortunately, people and programming tools encouraged us to believe that this project was possible. R, RStudio, {bookdown}, and Git had already solved publishing and collaboration problems for many. Except for some initial coding gaffes you’d expect from a team finding their feet and the occasional burned down fork, these tools freed us to focus on the larger task at hand: finding a common language for data science in education. We’ll close this post by discussing how books, authored through an open-source approach, can serve as an innovative platform for sharing knowledge with a wider audience.

Conclusion: One Writer, Five Authors.

How do you get five points of view to sound like a single voice? You’ll need a flexible sense of clarity, which I think is what Jesse meant when she said in a recent team call, “I have strong opinions, loosely held.” And it helps to have some basic rules as guardrails to flank your team as you march towards your writing deadlines. In this last post, we’ll share the workflows and processes we leaned on to discover what we wanted this book to be. We’ll also share our go-to tactics to keep the work going for the long haul, like managing meeting agendas, creating flexible norms for participation, and playing to individual strengths.

We’ll be back with that first post in about two weeks. Until then, do share with us about the people and tools that inspire you to work on collaborative projects. You can reach us on Twitter: Emily @ebovee09, Jesse @kierisi, Joshua @jrosenberg6432, Isabella @ivelasq3, and me @RyanEs.

See you in two weeks!

Ryan, with help from Emily, Jesse, Joshua, and Isabella

Emily A. Bovee, Ph.D., is an educational data scientist working in dental education.
Jesse Mostipak, M.Ed., is a community advocate, Kaggle educator and data scientist.
Joshua M. Rosenberg, Ph.D., is Assistant Professor of STEM Education and the University of Tennessee, Knoxville.
Isabella C. Velásquez, MS, is a data analyst committed to nonprofit work with the aim of reducing racial and socioeconomic inequities.

Congratulations Class of 2020!

Sat, 16 May 2020 00:00:00 +0000

Yesterday, along with R-Ladies Founder Gabriela de Queiroz I was honored to be asked to give a short talk at an online graduation ceremony for the BSc and MSc Statistics Graduates of the class of 2020 (all R users) at the California State University, East Bay. My talk contains some statistics jargon, but I thought it might be helpful to Class of 2020 graduates in other disciplines who have made some serious effort to acquire critical thinking skills. I know that not everyone thinks a graduation ceremony is important, but it seems to me that the Class of 2020 is getting seriously short changed, and could benefit from some attention. If any readers of this post would like to address the Class of 2020, I would be happy to publish their remarks in R Views.

Address to California State East Bay BSc and MSc Statistics Graduates: May 15, 2020

Dear Fellow Alumni: Congratulations!! It is difficult enough to make that final push to graduate under normal circumstances, but you have done it in the middle of a nightmarish doomsday scenario - Well done! You have demonstrated that you have what it takes to think clearly under pressure.

Your families need you, your friends need you, your communities need you, and it is not an exaggeration to say that the world needs you. You have demonstrated some skill at reasoning about uncertainty, at a time when fearful men and women are anxious about walking out their front doors. They want bright line rules for how to avoid the virus, and they want absolute certainty that they will be safe.

“If I wear a mask will I be protected? If a stay six feet apart does that mean everything will be OK? How about three feet? Well then, can I get my toenails done?”

The role of the statistician is to help people cope with the irreducible uncertainty in the world. Irreducible uncertainty - say that phase to yourself. Let it roll around in your head, but don’t let it scare you. You know that even when you have data that you believe in, data that you have cleaned and curated yourself; when you know something about the application you are working on and you have good reasons to use an informative prior - even then there is uncertainty. There is no way to escape the bias variance trade off - no way to be absolutely certain.

You have acquired a probabilistic mindset that should help you to help others: to help them navigate the narrow path flanked by despair on one side and magical thinking on the other. When you look at the news, you can see how the nuanced explanations of the health care experts get watered down into oversimplified statements that are mostly misleading. It is now an everyday thing to see some authoritative sounding news person or public official show a forecast with enormous error bands and just ignore them. You are now in the group with some responsibility to stand up and say: “Whoa! Hold on here!

Alright, so there’s one minor problem - the world, your community, your prospective employers may not yet know that they need you. The only way to help them to see the light in you is for you to be the person you want to be. If you want to be a statistician or data scientist, start to be one right away. Pick a problem, get the data, visualize it, analyze it, write about it. Put everything on your GitHub page. Talk about it to everyone who will listen.

You don’t have to want to be a data scientist, but you do have to want to grow in some direction. Listen to yourself, trust yourself. Decide what it will take for you to flourish and go for it. Choose your friends carefully, and get involved in some cause bigger than yourself.

Go for it. And wherever you go, walk with humility, but lead the way. Use your hard earned skills to navigate the random and the unpredictable. Take some reasonable risks, but consider the consequences for others.

You can do it! Good luck to you all. Stay safe and prosper.

And, because a graduation event is a good time to remember one’s own teachers, I would like to express my gratitude to Cal State East Bay Professors, Eric Suess and Bruce Trumbo, and also to Donald Lewis, Professor of Philosophy at Cal State Dominguez Hills.

Some Upcoming R Related, Virtual Events

Fri, 08 May 2020 00:00:00 +0000

COVID-19 Data Forum Webinar - Next Thursday (5/14/20) at Noon Pacific time, the COVID-19 Data Forum, sponsored by R Consortium and the Stanford Data Science Institute will open with a public webinar. The purpose of the Forum is to provide a way for experts to focus on data-related aspects of the scientific response to the pandemic, including data access and sharing, essential data resources for analysis, and how data scientists can best support decision making. The webinar speakers, all R savvy researchers, are:

Orhun Aydin, Researcher and Product Engineer, ESRI
Ryan Hafen, data scientist consultant with Preva Group, and adjunct assistant professor, Purdue University
Alison L. Hill, Research Fellow and independent principal investigator at Harvard’s Program for Evolutionary Dynamics
Noam Ross, Senior Research Scientist, EcoHealth Alliance

e-Rum2020!, The European R Users Meeting, will be online from June 17th through June 22nd. This free conference is shaping up to be an outstanding event. Keynote speakers include: Sharon Machlis, Director of Editorial Data & Analytics at IDG Communications; Jared Lander, Chief Data Scientist at Lander Analytics; Stephanie Hicks, Assistant Professor Johns Hopkins Bloomberg School of Public Health; Francesco Bartolucci, Professor of Statistics at the University of Perugia; Kelly O’Briant, solutions engineer at RStudio; and Tomas Kalibera, researcher at the Czech Technical University. Note that the e-Rum2020! CovidR Pre-conference Event will be held on May 29th. Be sure to “purchase” a free ticket before May 22nd.

The third R / Medicine conference will be held online from August 27th through the 29th. The keynote speakers will be Robert Gentleman, Daniela Witten, Ewen Harrison, and Patrick Mathias. The workshops scheduled for the first day are Introduction to R for Clinicians taught by Stephan Kaduake and an Introduction to Machine Learning with Tidymodels taught by Alison P. Hill. The call for papers is open.

The Why R? 2020 conference which will also begin on August 27th may turn out to be a virtual conference. The keynote speakers are Frank Harrell Professor of Biostatistics, Vanderbilt University; Roger Bivand, Professor at Norwegian School of Economics; Riinu Ots Senior Data Manager, University of Edinburgh; and Jan Vitek Professor of Computer Science, Northeastern University. As a run up to the conference, Why R? broadcasts webinars every Thursday at 8pm CEST / 2pm US Eastern Time. You can view the recordings of past events here.

Several other conferences including the R Conference, useR!2020, BioC 2020 and the JSMare also making plans to go virtual.

Greg Wilson Wins ACM Influential Educator Award

Mon, 04 May 2020 00:00:00 +0000

Recently, the Association for Computing Machinery’s (ACM’s) Special Interest Group on Software Engineering (SIGSOFT) recognized Greg Wilson as the 2020 recipient of its prestigious Influential Educator Award which is awarded annually to individuals or groups who have made significant contributions to software engineering through education, mentoring or policy.

JBR: Greg Congratulations, and thank you for agreeing to this interview.

Greg, you have been an educator for quite some time, first as a professor of computer science, then with Software Carpentry and now RStudio. During your career technology has affected the educational experience for both students and teachers alike. What do you think have been the most dramatic changes?

Greg: Technically, Stack Overflow is the biggest change in computing education in the last 20 years. It has a lot of problems—women frequently have to post under pseudonyms in order to avoid harassment, for example—but every professional programmer I know consults it many times a day, and I think educators have an obligation to include it rather than ignore it.

But the more important change has been cultural. Guys like me didn’t pay much attention to equity or inclusion 20 years ago, and when we did, it was to make snarky jokes. Today, most communities have a code of conduct, all-white or all-male panels at conferences are called out, and pay disparities are discussed openly. (This article by Julia Silge is a great example.) We still have a long way to go, but at least we’re now talking about things that really matter.

JBR: Economic pressures over recent years have led many colleges and universities to explore online learning as a cost effective alternative to the classroom experience, how do you see these efforts going? Have there been any notable successes? Do you think that online experience is changing the relationship between teachers and students?

Greg: There’s a joke that online learning is what everyone wants for someone else’s kids. Distance learning can be just as effective as in-person learning—the Open University in the UK, for example, has been proving that annually since the 1970s—but MOOCs were driven by “what technology do we have?” and “can we scale quickly enough to keep the VCs interested?” rather than “how do we teach effectively?” I think the next decade will see us backing away from recorded video and robo-graded exercises and emphasizing real-time peer-to-peer interaction instead. It’s pedagogically more effective, and learners who have grown up playing Minecraft with each other will wonder why we ever did anything else.

JBR: The COVID-19 pandemic is forcing educators at all levels from pre-school teachers to college professors to adapt, “cold turkey” to the reality of completely immersive online education. What advice do you have for teachers and students?

GREG: The first is “only change what you absolutely have to”. The second is to ask learners rather than making assumptions: their devices may be a lot less powerful than yours, or shared with other family members, or they may not have bandwidth or a quiet place for meetings, and so on. The third, which is probably most important, is to be kind: if there was ever a time to give learners partial marks or extensions on homework, it’s now.

JBR: Do you distinguish between education and training, and if so do, you think the online experience can be effective for both tasks?

Greg: I don’t think there’s a meaningful distinction. I don’t believe people can learn to think about computing in any meaningful way without learning how to program, or vice versa. I think (at least I hope) the biggest change we’ll see from moving programming education online is a much wider adoption of live coding, and with it, much more emphasis on how programs are written, not just what they look like when they’re done.

JBR: Is there anything special about teaching R? Does the deep association of R with statistical inference and methods pose any particular challenges for educators?

Greg: I think the biggest challenge in teaching R is getting people with computer science backgrounds like mine to take it seriously. Nobody would design a language that way today, but then, nobody would design C or JavaScript as they are either, and they’re all extraordinarily useful tools.

I think the other challenge is that we don’t really know how to test data analyses. The unit testing paradigm of software engineering isn’t really appropriate: it doesn’t tell us how close is close enough when we’re checking answers, and its focus is building reusable products, not answering questions. A data scientist constructing an R Markdown notebook step by step is doing something very different from what a developer building an e-commerce site does; I hope to learn a lot more about those differences in the next few years so that we can teach data scientists more effectively.

JBR: Is there anything we did not touch on that you would like to communicate to our R Views readers before we wrap up here?

Greg: Yes: please wash your hands, and please vote.

March 2020: "Top 40" New CRAN Packages

Mon, 27 Apr 2020 00:00:00 +0000

Two hundred ninety-six new packages made it to CRAN in March. Here are my “Top 40” picks in ten categories: Computational Methods, Data, Machine Learning, Mathematics, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

celltrackR v0.3.1: Provides a methodology to analyze cells that move in a two- or three-dimensional space. While the methodology has been developed for cell trajectory analysis, it is applicable to anything that moves including animals, people, or vehicles. For background see Beauchemin et al. (2007), Beltman et al. (2009), Gneiting & Schlather (2004) and additional papers listed by the package authors. There are There is an Introduction and vignettes on Track Analysis Methods, Clustering, Quality Control and Preprocessing, and Simulating Tracks.

collapse v1.1.0: Implements C/C++ based functions for advanced data transformations including statistical functions supporting grouped and/or weighted computations on vectors, matrices and data.frames and more. There is an Introduction and vignettes on collapse and dplyr, and Collapse and plm.

graphsim v0.1.1: Provides functions to simulate continuous data (e.g., gene expression) from a sigma covariance matrix derived from a graph structure in igraph objects. It extends mvtnorm to take igraph structures as input. There are vignettes on simulating gene expression data, convergent graph structures, divergent graph structures, network graph structures, inhibiting network graph structures, reconvergent graph structures, and vignettes on Directed Plots and Viral Plots.

rkeops Provides an interface to the KeOps library which computes generic reductions of very large arrays whose entries are given by a mathematical formula, and is suited to the computation of kernel dot products and the associated gradients, even when the full kernel matrix does not fit into the GPU memory. There is an Introduction and a vignette on Bindings.

simhelpers v0.1.0: Provides functions to help run simulation studies and calculate performance measures and associated Monte Carlo standard errors for simulation results. The general simulation workflow is closely aligned with the approach described by Morris et al. (2019). There are vignettes on Performance Criteria, Workflow, and Visualization.

Data

COVID19 v1.0.0: Provides COVID-19 datasets from several sources in a unified tidy format. The data are downloaded in real-time, cleaned and matched with exogenous variables. Vintage databases are also supported. There is a vignette.

ReDaMoR v).4.2: Implements methods to manipulate relational data models, including functions to create, modify and export data models in json format, importing models created with MySQL Workbench, and a shiny app. See the Tutorial.

Machine Learning

kerasTuneR v0.1.0.1: Implements an interface to the Keras Tuner hypertuning framework. There are vignettes on Bayesian Optimization, the HyperModel subclass, the R Interface, and MNIST Hypertuning.

rainette v0.1: Implements the Reinert text clustering method. There is an introduction, a description of the algorithm, and a vignette on utilization.

SSLR v0.9.1: Implements techniques for semi-supervised (both labeled and unlabeled data are used to train a classifier) classification and regression. There is an [Introduction]() and vignettes on Model Fitting, Models, Classification and Regression.

Mathematics

actuaryr v1.1.1: Provides functions to refer to the first or last (working) day within a specific period relative to a base date to facilitate actuarial reporting and to compare results. See the vignette for details.

stokes v1.0-5: Provides functionality for working with differentials, k-forms, wedge products, Stokes’s theorem, and related concepts from the exterior calculus and also Grassman algebra. See Calculus on Manifolds for the math, and the vignette for an introduction.

Medicine

covid19us v0.1.2: Implements wrapper around the COVID Tracking Project API providing data on cases of COVID-19 in the US.

Science

eicm v1.0.0: Implements model fitting and species biotic interaction network topology selection for explicit interaction community models. See the vignette for details.

leastcostpath v1.2.1: Provides functions to calculate cost surfaces based on slope. See Llobera & Sluckin (2007) and White & Barber (2012) for background and the vignette for examples.

Statistics

AQuadtree v1.0.0: Implements an automatic aggregation tool to manage point data privacy using the methodology described in Lagonigro et al. (2017). The algorithm seeks data accuracy at the smallest possible areas preventing individual information disclosure. The vignette provides examples.

bbricks v0.1.2: Provides tools to fit Bayesian parametric and nonparametric models including Gaussian and Normal-Inverse-Wishart conjugate structure, Gaussian and Normal-Inverse-Gamma conjugate structure, Categorical and Dirichlet conjugate structure and other Dirichlet processes. See the vignette.

causaloptim v0.6.5: Implements an interface to specify causal graphs and compute bounds on causal effects by extending and generalizing the approach taken in Balke & Pearl (1994). There are vignettes on Examples, Computing Causal Bounds, and on the Using Shiny App.

detectseparation v0.1: Provides pre-fit and post-fit methods for detecting separation and infinite maximum likelihood estimates in generalized linear models with categorical responses. The pre-fit methods solve the linear programming problems for the detection of separation developed in Konis (2007) The post-fit methods apply to models with categorical responses, including binomial-response generalized linear models and multinomial-response models. See the vignette for more information.

multilevelTools v0.1.1: Computes effect sizes, diagnostics and performance metrics for multilevel and mixed effects models including marginal and conditional R² estimates for linear mixed effects models based on Johnson (2014). See the vignette.

registr v1.0.0: Implements the functional data analysis method for registering curves (functional data), described in Wrobel et al. (2019) that are generated from exponential family distributions. The vignette provides an introduction.

specr v0.2.1: Provides utilities for conducting specification curve analyses, Simonsohn et al. (2015), or multiverse analyses ( Steegen et al. (2016)) including functions to setup, run, evaluate, and plot all specifications. There is an Introduction and vignettes on Customizing Plots, Identifying Variance Components, Investigating Selected Specifications, and Visualizing Progress.

spsurv v1.0.0: Provides routines to ease semiparametric survival regression modeling based on Bernstein polynomials, including proportional hazards, proportional odds and accelerated failure time frameworks for right-censored data. See Panaro (2020) for background and the vignette to get started.

terra v0.5-8: Provides classes and methods for spatial data as well as methods to allow for low-level data manipulation as well as high-level global, local, zonal, and focal computation. See the manual and tutorials to get started.

tramME v0.0.2: Implements likelihood-based estimation of mixed-effects transformation models using the Template Model Builder. See Hothorn et al. (2018) for the technical details and the vignette to get started.

Time Series

ForecastTB v1.0.1: Provides a test bench for comparing forecasting methods for uni-variate time series using different error metrics. See the vignette for examples.

fsMTS v0.1.5: Implements feature selection routines for multivariate time series including external structure, Pfeifer & Deutsch (1980); cross-correlation; graphical LASSO, Haworth & Cheng (2014), least angle regression, Gelper & Croux (2008); mutual information Liu et al. (2016), and partial spectral coherence Davis et al. (2016). There is a vignette using simulated data and another using real traffic data.

Utilities

brio v1.0.0: Provides functions to handle basic input output, these functions always read and write UTF-8 (8-bit Unicode Transformation Format) files and provide more explicit control over line endings.

deepdep v0.2.0: Provides the tools for exploring package dependencies for both CRAN and Bioconductor packages. There is an Introduction and vignette on Comparisons.

dm v0.1.1: Provides tools for working with multiple related tables, stored as data frames or in a relational database. There is a Getting Started Guide, and vignettes on Basic Operations, Filtering, Function Naming Logic, Relational Data Models, Joining, Low Level Operations. Data Prep, Databases, Visualizing dm Objects, and on Zooming and Manipulating tables.

emayili v0.3.9: Implements a tool for sending email with minimal dependencies. Look here to get started.

gt v0.2.0.5: Provides an API to create presentation-ready display tables. Look here for information to get started.

ipaddress v0.2.0: Provides classes and functions for working with IP (Internet Protocol) addresses and networks and offers full support for both IPv4 and IPv6 (Internet Protocol versions 4 and 6) address spaces. There is an Introduction.

scaffolder v0.0.1: Comprehensive set of tools for scaffolding R interfaces to modules, classes, functions, and documentations written in other programming languages, such as Python. See the vignette for examples.

colorBlindness v0.1.6: Provide the safe color sets for color blindness collected from: Wong, B. (2011), bcgs and University of Oregon and also simulators of protanopia, deuteranopia based on Vienot et al. (1999). See the vignette for examples.

ggcharts v0.1.0: Provides functions to streamline the process of creating some common ggplot2 plots. See the vignette for examples.

ggimg v0.1.0: Provides two new layer types for displaying image data as layers within the Grammar of Graphics framework. See the vignette for details.

simplevis v1.1.0: Provides ggplot2 and leaflet wrapper functions designed to simplify the creation of high quality graph and map visualizations. The vignette contains several examples.

Visualize.CRAN.Downloads v1.0: Provides functions to visualize trends and historical package download information using the RStudio CRAN mirror. The Vignette shows how.

The Case for tidymodels

Tue, 21 Apr 2020 00:00:00 +0000

If you are a data scientist with a built-out set of modeling tools that you know well, and which are almost always adequate for getting your work done, it is probably difficult for you to imagine what would induce you to give them up. Changing out what works is a task that rarely generates much enthusiasm. Nevertheless, in this post, I would like to point out a few features of tidymodels that could help even experienced data scientists make the case to give tidymodels a try.

So what are we talking about? tidymodels are an integrated, modular, extensible set of packages that implement a framework that facilitates creating predicative stochastic models. tidymodels are first class members of the tidyverse. They adhere to tidyverse syntax and design principles that promote consistency and well-designed human interfaces over speed of code execution. Nevertheless, they automatically build in parallel execution for tasks such as resampling, cross validation and parameter tuning. Moreover, they don’t just work through the steps of the basic modeling workflow, they implement conceptual structures that make complex iterative workflows possible and reproducible.

If you are an R user and you have building predictive models then there is a good chance that you are familiar with the caret package. One straightforward path to investigate tidymodels is to follow the thread that leads form caret to parsnip. caret, the result of a monumental fifteen year plus effort, incorporates two hundred thirty-eight predictive models into a common framework. For example, any one of the included models can be substituted for lm in the following expression.

lmFit <- train(Y ~ X1 + X2, data = training, 
                 method = "lm", 
                 trControl = fitControl)

By itself this is a pretty big deal. parsnip refines this idea by creating a specification structure that identifies a class of models that allows users to easily change algorithms and also permits the models to run on different “engines”.

spec_lin_reg <- linear_reg() %>%   # a linear model specification
                set_engine( "lm")  # set the model to use lm
# fit the model
lm_fit <- fit(spec_lin_reg, Y ~ X1 + X2, data = my_data)

This same specification can be modified to run a Bayesian model using Stan, or any number of other linear model backends such as glmnet, keras or spark.

spec_stan <- 
  spec_lin_reg %>%
  set_engine("stan", chains = 4, iter = 1000) # set engine specific arguments
fit_stan <- fit(spec_stan, Y ~ X1 + X2, data = my_data)

On its own, parnsnip provides a time saving framework for exploring multiple models. It is really nice not to have to worry about the idiosyncratic syntax developed for different model algorithms. But, the real power of tidymodels is baked into the recipes package. Recipes are structures that bind a sequence of preprocessing steps to a training data set. They define the roles that the variables are to play in the design matrix, specify what data cleaning needs to take place, and what feature engineering needs to happen.

To see how all of this comes together, lets look at recipe used in the tidymodels recipes tutorial that uses the New York City flights data set, nycflights13. We assume that all of the data wrangling code in the tutorial has been executed, and we pick up with the code to define the recipe:

flights_rec <- 
  recipe(arr_delay ~ ., data = train_data) %>% 
  update_role(flight, time_hour, new_role = "ID") %>% 
  step_date(date, features = c("dow", "month")) %>% 
  step_rm(date) %>% 
  step_dummy(all_nominal(), -all_outcomes())

The first line identifies the variable arr_delay as the variable to be predicted and the other variables in the data set train_data to be predictors. The second line amends that by updating the roles of the variables flight and time_hour to be identifiers and not predictors. The third and fourth lines continue with the feature engineering by creating a new date variable and removing the old one. The last line explicitly converts all categorical or factor variables into binary dummy variables.

The recipe is ready to be evaluated, but if a modeler thought that she might want to keep track of this workflow for the future, she might bind the recipe and model together in a workflow() that saves everything as a reproducible unit with a command something like this.

lr_mod <- logistic_reg() %>% set_engine("glm")

flights_wflow <- 
  workflow() %>% 
  add_model(lr_mod) %>% 
  add_recipe(flights_rec)

Then, fitting the model is just a matter calling fit with the workflow as a parameter.

flights_fit <- fit(flights_wflow, data = train_data)

At this point, everything is in place to complete a statistical analysis. A modeler can extract coefficients, p-values etc., calculate performance statistics, make statistical inferences and easily save the workflow in a reproducible markdown document. However the real gains from tidymodels become apparent when the modeler goes on to build predictive models.

The following diagram from Kuhn and Johnson (2019) illustrates a typical predictive modeling workflow.

It indicates that before going on to predict model performance on new data (the test set), a modeler will want to make use of cross validation or some other resampling technique to first evaluate the performance of multiple candidate models, and then tune the selected model. This is where the great power of the recipe() and workflow() constructs becomes apparent. In addition, to encouraging experiments with multiple models by rationalizing algorithm syntax, providing interchangeable model constructs, and enabling modelers to grow chains of recipe steps with the pipe operator; recipies helps to enforce good statistical practice.

For example, although it is common practice to split the available data between training and test sets before preprocessing the training data set, it is also very common to see pipelines where data preparation is applied to the entire training set at one go. It is not common to see data cleaning and preparation processes individually applied to each fold of a ten-fold cross validation effort. But, that is exactly the right thing to do to mitigate the deleterious effects of data imputation, centering and scaling and numerous other preparation steps that contribute to bias and limit the predictive value of a model. This is the whole point of resampling, but it is not easy to do in a way that saves necessary intermediate artifacts, and provides a reproducible set of instructions for others on the modeling team.

Because, recipes are not evaluated until the model is fit tidymodel workflows make an otherwise laborious and error prone process very straightforward. This is a game changer!

The next two lines of code set up and execute ten-fold cross-validation for our example.

set.seed(123)
folds <- vfold_cv(train_data, v = 10)
flights_fit_rs <- fit_resamples(flights_wflow, folds)

And then, another line of code collects the metrics over the folds and prints out the statistics for accuracy and area under the ROC curve.

collect_metrics(flights_fit_rs)

So, here we are with a mediocre model, and I’ll stop now having shown you only a small portion of what tidymodels can do, but enough, I hope to motivate you to take a closer look. tidymodels.org is a superbly crafted website with multiple layers of documentation. There are sections on packages, getting started guides, detailed tutorials, help pages and a section on making contributions.

Happy modeling!

Some Select COVID-19 Modeling Resources

Tue, 07 Apr 2020 00:00:00 +0000

There is an incredible amount of COVID-19 related material available online. While many dashboards, data sets, shiny apps and models represent significant contributions towards fighting the pandemic, we seem to have reached a point where we should be thinking about standards of quality, and should be exploring avenues for cooperation before launching more individual efforts. Below are a few examples of what I believe are exemplars of different types of COVID-19 related contributions. Please feel free to criticize my choices, or suggest other examples of quality contributions in the comments following this post.

A Shiny App

At the top of my list is the Shiny based model: Modeling COVID-19 Spread vs Healthcare Capacity developed by Alison Hill of Harvard’s Program for Evolutionary Dynamics with contributions from several researchers at the University of Pennsylvania, Harvard and Iowa State. Some things I really like about this app are: (1) that it provides enough background information to be self-documenting and self-contained. (2) The interactive visualizations are without unnecessary adornment and clearly labeled. (3) Default settings for sliders and other parameters are mostly based on documented data. (4) The differential equation theory underlying the model is clearly presented with the intention of helping model users understand the math. The content behind Model and Tutorial tabs sets a standard of excellence for concise explanations within the limits of the form factor. (5) The extensive documentation under the Sources tab inspires confidence and opens wide a window to this field of health care planning. (6) The code is straightforward and well organized. I believe that anyone developing a Shiny app for use by professionals would do well to try and meet or exceed the standard set by this app.

A Curious Number

To me, no single number in the catalog of COVID-19 models is more alluring, or cloaked in deeper mystery than R₀, the reproduction number. So I was delighted to discover James Holland Jones’ 2007 Notes on R₀, a very readable account of the assumptions and mathematics that underlie this parameter that provides links for those who want an even deeper understanding.

Some Prominent R Packages

The R packages from the R Epidemics Consortium are tools for professional epidemiologists. See the Tim Churches 3/19 post for an example of using EpiModel and his 3/5 post for and example using earlyR and EpiEstim.

A Site for Researchers

The comprehensive Coronavirus Disease (COVID-19) Statistics and Research site by Max Roser, Hannah Ritchie and Esteban Ortiz-Ospina offers a global view of the pandemic that is updated daily.

A Few Dashboards

The Johns Hopkins Data Center provides daily updates on New Cases, Mortality and Cumulative Cases.
The New York Times Coronavirus in the U.S.: Latest Map and Case Count page is an excellent way to drill down into U.S. State and County data.
The USAFACTS Coronavirus in the United States: Mapping the COVID-19 outbreak in the states and counties aggregates CDC data directly harvested from state and local agencies accessed through the Public Health Professionals Gateway

Two High Impact Blog Posts

Tomas Pueyo’s 3/10 post which raised the alarm in American Statistical Association circles is still worth reading. The alarming predictions of human suffering backed up by data visualizations make an emotional impact. This post also displays the the masterful graph from the by Wu and McGoogan’s JAMA paper showing the early history of the outbreak in Wuhan.
Terry Tao’s March 25th post which announces the Christopher Strohmeier COVID-19 Polymath Proposal initiated a valuable discussion on relevant data sets. See the Covid-19 dataset clearinghouse wiki.

Some Data

In addition to the data behind some of the above links, you may find the following data sources useful. The Johns Hopkins COVID-19 data seems to have become the defacto standard for COVID-19 modeling. You can directly download: time_series_covid19_confirmed_global.csv and time_series_covid19_deaths_global.csv. Additionally, both esri and data.world (See The Corononavirus (COVID-19) Data Resource Hub) provide a number of curated COVID-19 data sets.

Close Encounters of the R Kind

Tue, 31 Mar 2020 00:00:00 +0000

Affiliation

Harrison – Center for Strategic and Budgetary Analysis, Washington DC

Cara – Department of the Air Force (Studies, Analyses, and Assessments - AF/A9), Washington DC

Disclaimer

The views expressed in this article represent the personal views of the author and are not necessarily the views of the Department of Defense (DoD) or the Department of the Air Force.

This post is an effort to condense the ‘buzz’ surrounding the explosion of open source solutions in all facets of analysis – to include those done by Military Operations Research Society ( MORS) members and those they support – by describing our experiences with the R programming language.

The impact of R in the statistical world (and by extension, data science) has been big: worthy of an entire issue of SIGNIFICANCE Magazine (RSS). Surprisingly, R is not a new computing language; modeled on S and Scheme,- the technology at the core of R is over forty years old. This longevity is, in itself, noteworthy. Additionally, fee-based and for-profit companies have begun to incorporate R with their products. While statistics is the focus of R, with the right packages – and know-how – it can also be used for a much broader spectrum, to include machine learning, optimization, and interactive web tools.

In this post, we go back and forth discussing our individual experiences.

Getting Started with R

Harrison: I got started with R in earnest shortly before retiring from the U.S. Navy in 2016. I knew that I was going to need a programming language to take with me into my next career. The reason I chose R was not particularly analytical; the languages that I had done the most work in during grad school – MATLAB and Java – were not attractive in that the first required licensing fees and the second was – to me at the time – too ‘low level’ for the type of analysis I wanted to perform. I had used SPlus in my statistics track, but never really ‘took’ to it while in school. Several tools to ‘bridge’ the gap between Excel and R had been recommended to me by a friend, including RStudio and Rcommander.

Onboard ship many years ago, I learned to eat with chopsticks by requesting that the wardroom staff stop providing me with utensils, substituting a bag of disposable chopsticks I purchased in Singapore. Turns out when deprived of other options, you can learn very fast. Learning R basics was the same; instead of silverware, it was removing the shortcuts to my usual tools on my home laptop (Excel). I simply did every task that I could from the mundane to the elegant in R.

Cara: I started dabbling with R in 2017 when I had about a year and a half left in my PhD journey, after I decided to pursue a post-doctoral government career. Sitting comfortably in academia with abundant software licenses for almost a decade, I had no reason until that point to consider abandoning my SAS discipleship (other than abhorrent graphics capability, which I bolstered with use of SigmaPlot). My military background taught me not to expect access to expensive software in a government gig, and I had a number of friends and colleagues already using R and associated tools, so I installed it on my home computer. Other than being mildly intrigued by the software version naming convention, I stubbornly clung to SAS to finish my doctoral research, however.

It was not until I became a government OR analyst in late 2018 that I really started my R journey (mostly out of necessity at this point). R does not require a license, and furthermore, is approved for use on government computers. I found R to be an easier and more natural programming language to learn coming from a SAS background than Python, Javascript, or C++ would have been.

How has using R shaped your practice?

Harrison: There is a lot of talk about how various tools perform in the sense of runtime, precision, graphics, etc. These are considerations, but they are completely eclipsed by the following: We don’t talk as a community about how much the tools we use shape our thinking. I frequently tell my colleagues that the fundamental unit in Excel is called a cell because it is your mind prison. There’s actually some truth to that. R is vectorized, so for most functions, passing an array gives an appropriate array output. When you work day-in and day-out with vectors, you stop thinking about individual operations start to think in terms of sentences. The magrittr %>% operator, which takes the expression on the left as the first argument to the function on the right, makes this possible. Analysis begins to feel more like writing sentences – or even short poems – than writing computing code.

Early in my work with R, I was told by a colleague that “R might be good but the graphics are terrible”. This was a bit of a shock, as graphics has been one of the main selling points of the language, and I didn’t want to be making seedy graphs. From that point on, I made it a point to make the best graphics I possibly could, usually – but not always – using methods and extensions found in the ggplot2 package. It is no exaggeration to say that I spend roughly 20% of my analysis time picking colors and other aesthetics for plots. If you are willing to take the time, you can get the graphics to sing; there are color schemes based on The Simpsons and Futurama, and fonts based on xkcd comics.

Cara: When I began teaching myself R – and using it daily – I thought I was merely learning the syntax of a new programming language. With the analytic capability inherent with R and the flexibility of development environments, however, it is really more of a way of thinking. Fold in the powerful (and mostly free!) resources and passionate following of analysts and data scientists, and you get an R community that I truly enjoy being a part of.

The R environment, even as a novice user, can have positive impacts on your workflow. For example, beyond syntax, my earliest explorations in R taught me that if you are going to do something more than once, write a function. I had never truly internalized that idea, even after a decade of using SAS. Another thing I learned relatively early on – get to know the dplyr package, and use it! I had been coding in R for about 6 months before I was really introduced to functions like dplyr::filter(), dplyr::select(), and dplyr::mutate(); these are powerful functions that can save a ton of code. I’ve been analyzing data for over a decade and I’ve never come across a dataset that was already in the form I needed. Prior to using the dplyr package, however, I was spending a lot of time manipulating data using no functions and a lot of lines of code. Beyond time savings, dplyr helps you think about your data more creatively. As a very basic example, dplyr::summarise() is a more powerful option than mean() used alone, especially for multiple calculations in a single data table. And once you master the Wonder Twin-esque combination of using group_by() with summarise(), you’ll be amazed at what you can (quickly) reveal through exploratory analysis. Data wrangling is (and always will be) a fact of life. The more efficiently you manipulate data, however, the more time you have to spend on the seemingly more exciting aspects of any project.

Disadvantages of R

Harrison: This piece is not a ‘sales pitch’ for R; but rather a sober consideration of what the tradeoffs an organization needs to consider when choosing an analytic platform writ large:

Compatibility and Editing. Because R is a computing language, graphics built in R are not editable by non-R users, as opposed to Excel graphs. This can be a challenge in the frequent case where the reviewers are not the same people that created the plots. If you made the plot, you are going to have to be the one who does the editing, unless there is another R user who understands your particular technique in the office.
No license costs do not mean that it’s free: I frequently like to say that I haven’t spent a dime on analytics software since I retired from the Navy; this is strictly true, but also misleading. I have spent considerable time learning the best practices in R over the past 4 years. An organization that is looking to make this choice needs to realize upfront that the savings in fees will be largely eaten up by extra manpower to learn how to make it work. The reward for investing the time in increasing the ability of your people to code is twofold; first, it makes them closer in touch with the actual analysis, and secondly, it allows for bespoke applications.

Cara: I work in a pretty dynamic world as a government operations research analyst (ORSA); we don’t typically have dedicated statisticians, programmers, data scientists, modelers, or data viz specialists. Most of the time, we are all functioning in some or all of those capacities. As a former engineer with a dynamic background, this suits me well. However, it also means that things change from day to day, from project to project, and as the government analytic world changes (rapidly). I do not have the flexibility to use one software package exclusively. Further, I face challenges within DoD related to systems, software, classification, and computing infrastructure that most people in academia or industry do not. In my organization, there has been a relatively recent and rapid shift in the analytic environment. We formerly leaned heavily on Excel-based regressions and descriptive statistics, usually created by a single analysts that answer a single question, and in many cases these models were not particular dynamic or scalable. We now focus on using open-source tools in a team construct, sometimes with industry partners, to create robust models that are designed to answer multiple questions from a variety of perspectives; scale easily to mirror operational requirements; fit with other models; and transition well to high performance computing environments.

The two open-source tools we (i.e., my division) currently use most for programming are R and Python. We have had success combining data analysis, statistics, and graphical models to create robust tools coded as RShiny apps. Recently, we chose to code in Python for a project that involved machine learning and high performance computing. I do not propose to debate the strengths and weaknesses of either R or Python in this forum; rather, I challenge you to consider carefully the implications of programming language choice for any project with a cradle to grave perspective.

References

Getting started with R can be daunting. We recommend the following references. Stack Overflow. This invaluable resource is a bulletin board exchange of programming ideas and tips. The real skill required to use it effectively is knowing how to write an effective question. “I hate ggplot” or “My R code doesn’t work” are not useful; try “Could not subset closure” or “ggplot axis font size” instead.
Vignettes. Well-developed R packages have vignettes, which are very useful in seeing both an explanation of the code as well as an example. Two very good references are the ggplot2 gallery and the dplyr vignette Finally, the RViews blog is a great way to keep up-to-date with practice.
Books. Although I tend to acquire books with reckless abandon, the ones I actually keep and use have withstood careful consideration and have generally pegged the daily utility meter. Try R for Data Science by Wickham and Grolemund (O’Reilly Publishing 2017) and Elegant Graphics for Data Analysis by Wickham (Springer 2016); available both as print copies or electronic editions.
Podcasts. For those moments in your life when you need some data science-related enrichment, the producers of DataCamp host an excellent podcast called DataFramed. Fifty-nine episodes have been recorded so far; find them on soundcloud, Spotify, YouTube, or VFR direct from the creator’s listening notes.
RStudio Cheatsheets. Sometimes you need densely constructed (read: compact yet surprisingly in-depth), straightforward references. RStudio creates (and updates) these two-pagers for the most popular and versatile R packages to be great portable references for programmers – think of them as a combined dictionary and thesaurus for learning R. Fun fact: they can be downloaded in multiple languages.
Forums. (1) Data Science Center of Education (DSCOE) is a CAC-enabled collaboration site that hosts data science tutorials developed by Army analysts, mostly using R, and supports a week-long R immersion course offered at Center for Army Analysis (CAA) twice a year. The DSCOE forum is managed collaboratively by the CAA, U.S. Army Cyber Command (ARCYBER), Naval Postgraduate School (NPS), and the United Stated Military Academy (USMA). Contributions are both welcome and encouraged. (2) R-bloggers, created in 2015, is an R centric forum designed to foster connection, collaboration, and resource sharing within the R community. The utility of this forum lies in its array of technical resources that will benefit both new and practiced users. (3) Data Science DC, for those in the NCR, was formed via the concatenation of numerous meetup groups – including RMeetup DC – and a major proponent of a number of events, including hackathons and the DCR conference (held annually in the fall).

February 2020: "Top 40" New R Packages

Thu, 26 Mar 2020 00:00:00 +0000

One hundred sixty-four new packages made it to CRAN in February. Here are my “Top 40” picks in eleven categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Time Series, Utilities, and Visualizations.

Computational Methods

delayed v0.3.0: Implements mechanisms to parallelize dependent tasks in a manner that optimizes the computational resources. Functions produce “delayed computations” which may be parallelized using futures. See the vignette for details.

tergmLite v2.1.7: Provides functions to efficiently simulate dynamic networks estimated with the framework for temporal exponential random graph models implemented in the tergm package.

Data

crsmeta v0.2.0: Provides functions to obtain coordinate system metadata from various data formats including: CRS (Coordinate Reference System), EPSG (European Petroleum Survey Group), PROJ4 and WKT (Well-Known Text 2).

danstat v0.1.0: Implements an interface into the Statistics Denmark Databank API. The vignette provides an Introduction.

osfr v0.2.8: Implements an interface for interacting with OSF which enables users to access open research materials and data, or to create and manage private or public projects. There is a Getting Started Guide and a vignette on Authentication.

Genomics

selectSNPs v1.0.1: Provides a method using unified local functions to select low-density SNPs. See the Vignette for a tutorial.

varitas v0.0.1: Implements a multi-caller variant analysis pipeline for targeted analysis sequencing data. There is an Introduction and a vignette on Errors.

Machine Learning

autokeras v1.0.1: Implements an interface to AutoKeras, an open source software library for automated machine learning. See README for an example.

MTPS v0.1.9: Implements functions to predict simultaneous multiple outcomes based on revised stacking algorithms as described in Xing et al. (2019). See the vignette to get started.

quanteda.textmodels v0.9.1: Implements methods for scaling models and classifiers based on sparse matrix objects representing textual data. It includes implementations of the Laver et al. (2003) wordscores model, the Perry & Benoit’s (2017) class affinity scaling model, and the Slapin & Proksch (2008) wordfish model. See the vignette to get started.

SeqDetect v1.0.7: Implements the automaton model found in Krleža, Vrdoljak & Brčić (2019) to detect and process sequences. See the vignette for examples and theory.

studyStrap v1.0.0: Implements multi-Study Learning algorithms such as Merging, Study-Specific Ensembling (Trained-on-Observed-Studies Ensemble), the Study Strap, and the Covariate-Matched Study Strap. and offers over 20 similarity measures. See Kishida, et al. (2019) for background and the vignette for how to use the package.

Mathematics

PlaneGeometry v1.1.0: Provides R6 classes representing triangles, circles, circular arcs, ellipses, elliptical arcs and lines, plot methods, transformations and more. The vignette offers multiple examples.

Medicine

Science

baRUlho v1.0.1: Provides functions to facilitate acoustic analysis of (animal) sound transmission experiments including functions for data preparation, analysis and visualization. See Dabelsteen et al. (1993) for background and the vignette for an introduction.

CBSr v1.0.3: Uses monotonically constrained Cubic Bezier Splines to approximate latent utility functions in intertemporal choice and risky choice data. See the Lee et al. (2019) for the details.

Statistics

blockCV v2.1.1: Provides functions for creating spatially or environmentally separated folds for cross-validation in spatially structured environments and methods for visualizing the effective range of spatial autocorrelation to separate training and testing datasets as described in Valavi, R. et al. (2019). See the vignette for examples.

BGGM v1.0.0: Implements the methods for fitting Bayesian Gaussian graphical models recently introduced in Williams (2019), Williams & Mulder (2019) and Williams et al. (2019). There are vignettes on Credible Intervals, Plotting Network Structure, Comparing GGMs with the Posterior Predicive Distributions, and Predictability.

metagam v:0.1.0: Provides a method to perform the meta-analysis of generalized additive models and generalized additive mixed models, including functionality for removing individual participant data from models computed using the mgcv and gamm4 packages. A typical use case is a situation where data cannot be shared across locations, and an overall meta-analytic fit is sought. For the details see Sorensen et al. (2020), Zanobetti (2000), and Crippa et al. (2018). There is an Introduction and vignettes on Dominance, Heterogenity Plots, and Multivariate Smooth Terms.

MKpower v0.4: Provides functions for power analysis and sample size calculations for Welch and Hsu t-tests, Wilcoxon rank sum tests and diagnostic tests. See Flahault et al. (2005) and Dobbin & Simon (2007) for background, and the vignette for examples.

mvrsquared v0.0.3: Implements a method to compute the coefficient of determination for outcomes in n-dimensions. See Jones (2019) for the theory and the vignette to get started.

pdynmc v0.8.0: Provides functions to model linear dynamic panel data based on linear and nonlinear moment conditions as proposed by Holtz-Eakin et al.(1988), Ahn & Schmidt (1995), and Arellano & Bover (1995). See the vignette for the underlying theory and a sample session.

Superpower v0.0.3: Provides functions to simulate ANOVA designs of up to three factors, calculate the observed power and average observed effect size for all main effects and interactions. See Lakens & Caldwell (2019) for background, and the vignette for an introduction.

tune v0.0.1: Provides functions and classes for use in conjunction with other tidymodels packages for finding reasonable values of hyper-parameters in models, pre-processing methods, and post-processing steps. Look here for and example.

xrnet v0.1.7: Provides functions to fit hierarchical regularized regression models incorporating potentially informative external data as in Weaver & Lewinger (2019). See README for examples.

Time Series

seer v1.4.1: Implements a framework for selecting time series forecast models based on features calculated from the time series. For details see Talagala et al. (20180).

testcorr v0.1.2: Provides functions for computing test statistics for the significance of autocorrelation in univariate time series, cross-correlation in bivariate time series, Pearson correlations in multivariate series and test statistics for i.i.d. property of univariate series as described in Dalla et al. (2019). See the vignette for the math and examples.

Utility

bioC.logs v1.1: Fetches download statistics BioConductor.org. See the vignette.

matricks v0.8.2: Provides function to help with creation of complex matrices along with a plotting function. See the vignette for examples.

rco v1.0.1: Provides functions to automatically apply different strategies to optimize R code. These functions take R code as input, and returns R code as output. There are vignettes on: Contributing an optimizer, Docker files, Common Subexpression Elimination, Constant Folding, Constant Propagation, Dead Code Elimination, Dead Expression Elimination, Dead Store Elimination, and Loop-invariant Code Motion.

slider v0.1.2: Provides type-stable rolling window functions over any R data type and supports both cumulative and expanding windows. See the vignette for examples.

taxadb v0.1.0: Provides fast, consistent access to taxonomic data, and supports common tasks such as resolving taxonomic names to identifiers and looking up higher classification ranks of given species. There is an Introduction and a Schema.

tidyfst v0.8.8: Provides a toolkit of tidy data manipulation verbs with data.table as the backend, combining the merits of syntax elegance from dplyr and computing performance from data.table. There is a vignete written in Chinese, an English Language Introduction and vignettes on join, reshape, nest, fst and dt.

tidytable v0.3.2: Provides an rlang compatible interface to data.table. See README for examples.

Visualization

iNzightTools v1.8.3: Provides wrapper functions for common variable and dataset manipulation workflows primarily used by iNZight, a graphical user interface providing easy exploration and visualization of data for students. Many functions return the tidyverse code used to obtain the result in an effort to bridge the gap between GUI and coding.

IPV v0.1.1: Provides functions to generate item pool visualizations which are used to display the conceptual structure of a set of items. See Dantlgraber et al. (2019) for background and the vignette for examples.

spacey v0.1.1: Provides utilities to download USGS and ESRI geospatial data and produce high quality rayshader maps for locations in the United States. There is an Introduction

Tendril v2.0.4: Provides functions to compute and display tendril plots. See the vignnette for and introduction..

tidyHeatmap v0.99.9: Provides an implementation of the Bioconductor ComplexHeatmap package based on tidy data frames. See the vignette.

January 2020: "Top 40" New R Packages

Mon, 24 Feb 2020 00:00:00 +0000

One hundred forty-seven new packages made it to CRAN in January. Here are my “Top 40” picks in nine categories: Computational Methods, Genomics, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Utilities and Visualization.

Computational Methods

FSSF v0.1.1: Provides three methods proposed by Shang & Apley (2019) to generate fully-sequential space-filling designs inside a unit hypercube.

seagull v1.0.5: Implements a proximal gradient descent solver for the operators lasso, group lasso, and sparse-group lasso. There is an Introduction.

Genomics

babette v2.1.2: Provides access to the BEAST2 Bayesian phylogenetic tool, that uses DNA/RNA/protein data and many model priors to create a posterior of jointly estimated phylogenies and parameters. There is a Tutorial, a Basic Demo, a Step by Step Demo, a vignette on Nested Sampling, and another with Examples.

statgenGWAS v1.0.3: Provides fast single trait Genome Wide Association Studies (GWAS) following the method described in Kang et al. (2010). See the vignette for details.

Machine Learning

akc 0.9.4: Provides a tidy framework for automatic knowledge classification and visualization. There is a Tutorial and a vignette on classification based on keyword co-occurrence.

ced v1.0.1: Provides R bindings for the Google Compact Encoding Detection library which takes as input a source buffer of raw text bytes and probabilistically determines the most likely encoding for that text.

forestError v0.1.1: Provides functions to estimate the conditional error distributions of random forest predictions and common parameters of those distributions, including conditional mean squared prediction errors, conditional biases, and conditional quantiles as proposed by Lu & Hardin (2019).

ksharpv0.1.0.1: Provides functions to sharpen clusters by adjusting existing clusters to create contrast between groups. The vignette provides examples and references.

mosmafs v0.1.1: Provides functions for simultaneous hyperparameter tuning and feature selection through both single-objective and multi-objective optimization as described in Binder et al. (2019). There is an Introduction and another vignette on Multi-Fidelity.

themis v0.1.0: Provides recipes for dealing with unbalanced data sets including balancing by increasing the number of minority cases using SMOTE, Borderline-SMOTE and ADASYN; or by decreasing the number of majority cases using NearMiss or Tomek link removal. Look here examples.

Mathematics

caracas v0.1.0: Implements computer algebra by providing access to the Python SymPy library making it possible to solve equations symbolically, find symbolic integrals, symbolic sums and other important quantities. There is a vignette.

clifford v1.0-1: Provides a suite of routines for Clifford algebras. Special cases include Lorentz transforms, quaternion multiplication, and Grassman algebra. See the vignette for details.

Medicine

metan v1.3.0: Provides functions for the stability analysis of multi-environment trial data using parametric and non-parametric methods including additive main effects and multiplicative interaction analysis, Gauch (2013); genotype plus genotype-environment biplot analysis, Yan & Kang (2003); joint regression analysis, Eberhart & Russel (1966) and much more. See the vignette to get started.

nosoi v1.0.0: Implements a flexible agent-based stochastic transmission chain, epidemic simulator. There is a Getting Started Guide and vignettes on Homogenous Populations, Discrete Populations, Continuous Populations, and Visualization.

shinySIR v0.1.1: Provides interactive plotting for mathematical models of infectious disease spread. Users can choose from a variety of common built-in ordinary differential equation models or create their own. See Keeling & Rohani (2008) and Bjornstad (2018) for background and the vignette for details.

transplantr v0.1.0: Provides a set of vectorised functions to calculate medical equations used in transplantation, focused mainly on transplantation of abdominal organs. There are vignettes on Estimated GFR, HLA Mismatch Level, Kidney Risk Scores, and Liver Recipient Scoring.

Statistics

bggum v1.0.2: Provides a Metropolis-coupled Markov chain Monte Carlo sampler, post-processing, parameter estimation functions, and plotting utilities for the generalized graded unfolding model of Roberts et al.(2000). See the vignette for the math and examples.

cSEM v0.1.0: Provides functions to estimate, assess, test, and study linear, nonlinear, hierarchical and multigroup structural equation models using composite-based approaches and procedures, including estimation techniques such as partial least squares path modeling (PLS-PM) and its derivatives (PLSc, ordPLSc, robustPLSc), generalized structured component analysis (GSCA) and others. There is an Introduction and vignettes on Notation, Terminology, and Post Estimation.

mcp v0.2.0: Implements regression with multiple change points which can be for means, variances, autocorrelation structure, and any combination of these. It provides a generalization of the approach described in Carlin et al. (1992) and Stephens (1994). See README for examples.

metopolis v0.1.5: Provides functions for learning how the Metropolis algorithm works. The vignette includes examples of hand-coding a logistic model using several variants of the Metropolis algorithm.

miceRanger v1.3.1: Implements multiple imputation by chained equations with Random Forests. There are vignettes on the Mice Algorithm, Filling in Missing Data, and Diagnostics Plotting.

momentfit v0.1-0. Provides functions to perform method of moment fits including the Generalized method of moments, (Hansen (1982) and the Generalized Empirical Likelihood, (Smith (1997). There are vignettes on Generalized Empirical Likelihood and Generalized Method of Moments.

nlraa v0.53: Implements nonlinear regression functions using self-start algorithms including the Beta growth function proposed by Yin et al. (2003). There is an Introduction, a vignette with examples from Archontoulis & Miguez (2015) and another vignette with examples from Oddi et al. (2019).

PoissonBinomial v1.0.2: Implements multiple exact and approximate methods for computing the probability mass, cumulative distribution and quantile functions, as well as generating random numbers for the Poisson Binomial distribution as described in Hong (2013) and Biscarri et al. (2018). There are vignettes on Efficient Computation, Approximate Procedures, Exact Procedures, and Usage with Rcpp.

relgam v1.0: Implements a method for fitting the entire regularization path of the reluctant generalized additive model (RGAM) for linear regression, logistic, Poisson and Cox regression models. See Tay & Tibshirani (2019) for details.

s2net v1.0.1: Implements the generalized semi-supervised elastic-net, extending the supervised elastic-net to make it practical to perform feature selection in semi-supervised contexts. See Culp (2013) for references on the Joint Trained Elastic-Net and the vignette for an example.

signnet v0.5.1: Implements methods for the analysis of signed networks including several measures for structural balance as introduced by Cartwright & Harary (1956), blockmodeling algorithms from Doreian (2008), various centrality indices, and projections introduced by Schoch (2020). There are vignettes on Blockmodeling, Centrality, Complex Matrices, Signed Two-Mode Networks, Signed Networks, and Structural Balance.

survParamSim v0.1.0: Provides functions to perform survival simulation with parametric survival model generated from survreg function in survival package. See the vignette.

xnet v0.1.11: Provides functions to fit a two-step kernel ridge regression model for predicting edges in networks, and carry out cross-validation. See Stock et al. (2018) for background. There is an Introduction, and vignettes on Data Preparation, and the S4 Class Structure.

Time Series

pcts v0.14-4: Provides classes and methods for modeling and simulating periodically correlated and periodically integrated time series. For background see Boshnakov & Iqelan (2009) and Boshnakov (1996).

fdaACF v0.1.0: Provides functions to compute autocorrelation functions for functional time series. Look here for examples.

Utilities

dmdScheme v1.0.0: Provides a framework for developing domain specific metadata. There is an Introduction and vignettes on Creating a New Scheme and Minimum Requirements.

gridtext v0.1.0: Provides support for rendering of formatted text using grid graphics. Look here for examples.

netstat v0.1.1: Implements an interface to the netstat command line utility for retrieving and parsing network statistics from Transmission Control Protocol (TCP) ports. See The Linux System Administrator’s Manual, and the Microsoft website for basic information.

progressr v0.4.0: Provides a minimal, unifying API for scripts and packages to report progress updates including when using parallel processing. The vignette offers an introduction.

PROJ v0.1.0: Implements a wrapper around the generic coordinate transformation software, PROJ that transforms geospatial coordinates from one coordinate reference system to another, and cartographic projections as well as geodetic transformations. See the vignette.

round v0.12-1: Provides functions to explore differences between current and potential future versions of the base R round() function along with some partly related C99 math lib functions not in base R. See the vignette for the details.

warp v0.1.0: Implements tooling to group dates by a variety of periods including: yearly, monthly, by second, by week of the month, and more. See the vignette for examples.

Visualization

apyramid v0.1.0: Provides a quick method for visualizing non-aggregated line-list or aggregated census data stratified by age and one or two categorical variables (e.g. gender and health status) with any number of values. This package is part of the R4Epis Project. See the vignette for examples.

mlr3viz v0.1.1: Provides visualizations for mlr3 objects including barplots, boxplots, histograms, ROC curves, and Precision-Recall curves. README offers some examples.

R, Public Health and Politics

Wed, 19 Feb 2020 00:00:00 +0000

Last week, Lancet published the paper Improving the prognosis of health care in the USA by Alison P Galvani, Alyssa S Parpia, Eric M Foster, Burton H Singer, Meagan C Fitzpatrick of CIDMA, the Center for Infectious Disease Modeling and Analysis, Yale School of Public Health. The paper, which, provides a detailed analysis of the single-payer system introduced by Senator Sanders in the Medicare for All Act was published with a Shiny application that allows readers to test key assumptions regarding health care budgets, projected revenue, and the projected expansion of health care use.

While the authors are clearly arguing the case for the single-payer system, the publication in a prestigious peer-reviewed journal, the detailed, documented data presented, and the Shiny app for testing assumptions should make this paper the basis for all serious, rational discussion about the economic viability of the single-payer system.

This Shiny app is also a milestone for R, as it demonstrates the ability of R to help experts interactively engage with informed citizens to help them develop their own insights on complex matters.

Photo Mosaics in R

Thu, 13 Feb 2020 00:00:00 +0000

Harrison Schramm, CAP, PStat, is a Senior Fellow at the Center for Strategic and Budgetary Assessments.

In this short piece, I’m going to discuss a fun photography project I did over the winter using R. I’m also going to touch on some of the implications of the R license, which underlies our entire ecosystem, but we don’t usually think about that often.

I’ve been a dedicated useR for the past 4 years. I started by using R for all the things that I previously did with spreadsheets - a great way to learn your way around the magrittr and dplyr packages. From there, I Replaced my word processor and slide software with markdown. All the while, I’ve been increasing the amount of time I spend on graphics, particularly within the ggplot2 construct, as well as color scales provided by the ggsci package (there are Simpsons and Futurama color palettes)! At some point over the summer, my interest in developing graphics began to inspire my R-tistic side, which I wrote about in INFORMS/Analytics magazine this summer.

All of last year, I worked on a project titled ‘Mosaic’. When we finished our report, my coauthors asked for suggestions on cover art, I naturally suggested we create a photo mosaic. While there are both free and commercially available solutions, my first choice, of course, was to find an R-centric solution. The advantages to using R for this project (as well as other things) is that it allows for the creation of bespoke solutions; in other words, I don’t want just any photo mosaic, but rather one that has the attributes that I want.

After a quick stack overflow / CRAN search, I found the RsimMosaic package which gave me the tools I was looking for.

Making a photo mosaic

Basically, the tools in this package take a base image and replace each pixel with a tile. If properly chosen, a close-up view will focus on the tiles, but at a distance the base image will emerge. While this is simple enough, there are a few thing to consider.

First, the dimension of the resulting image will be (in pixels) approximately the base image expanded by the tiles; for instance, if the base is 150x150, and the tiles are 30x30, the resulting image will be 4500 x 4500. I (somewhat embarrassingly) discovered this by having my R instance return a cannot allocate vector of size 9GB error ¹.

Second, there is an R-tistic balance between the size of the base image and the size of the tiles; some experimentation is necessary. If you have an extensive library of tiles (I had over 600 in this instance), it is possible - but ill advised - to try to adjust their sizes manually. Fortunately, the package has a utility for doing this. However, there’s a catch; not all pictures have a base resolution that is amenable to being scaled down.

Building a photo mosaic is really an R-tistic thing to do. The key is to collect a library of tiles that will allow sufficient diversity so that the program can make good contrast choices. For example, pictures of ships and airplanes are heavy in blue tones, etc.

Making a tile library

Once you have found your tiles, you will want to resize them. For example, I have found in most images I’ve been working with that a 30 x 30 (pixel) tile is a good size, balancing resolution with ‘mosaic-ness’. However, this is not the size of most raw images, and while you can resize them in MS Paint, this is a painstaking process. Fortunately RsimMosaic provides a handy method: createTiles. It’s almost perfect for my application.

Here’s the bit about the license

Because R and it’s packages are distributed under the GPL license, you have the ability to adjust the functions that are in packages. This is straightforward; if you want to adjust a function foo, you can assign it a new name foo2 <- foo and then fix(foo2) (or use a script / markdown) to make changes. You can of course simply edit the original function, but I do not recommend this as it can cause confusion on subsequent sessions.

Remember in the preceding paragraph where I said it was almost perfect? Some tiles have base sizes that cannot be coerced to rectangular shapes. The method that comes with the package simply generates an error. What is useful when generating ~600 tiles is to have a list so that I know the one that threw an error ².

But because we can do whatever we want with existing functions, consider the following modification:

createTiles2 = createTiles

fix(createTiles2)

{add the following directly after line 15:}

print(filenameArray[i])

And voila! Fast and artistic! I used this list to find the tiles that could not be resized, and removed them from my tile library.

Making the Mosaic

Now, for the part where we make the mosaic. The command:

composeMosaicFromImageRandomOptim("RLogo_small.jpg", "RMosaic.jpg", "OutputPathGoesHere")

generates the following:

is it surprising that an accomplished professional would admit to such a silly mistake in public? I don’t think so! We would all be better off if we were as open with our success as our failures; first to prevent others from wasting precious time with similar mistakes, and second to show that if we are going to work at the cutting edge, we are all in a sense students.↩
other programmers might suggest a tryCatch() environment, and I did think of that. Here was a case where I wanted something that worked fast, vice something that worked well.↩

Some 2020 R Conferences

Wed, 05 Feb 2020 00:00:00 +0000

rstudio::conf kicked off the 2020 season for R conferences last week with record attendance somewhere north of twenty-one hundred. Session topics ranged from business to science, marketing to medicine and attracted R users with very varied backgrounds including DevOps professionals, data scientists, journalists, physicians, statisticians, R package developers, Shiny developers and more. Although it is true that the San Francisco Bay Area is home to a large R Community, and that a great deal of planning and promotion went into making rstudio::conf a success, I don’t think that the enthusiasm and energy that permeated the conference was a local phenomenon. I expect 2020 to be a good year for R conferences worldwide. Here is my short, somewhat eclectic, and by no means complete list of upcoming 2020 R events.

While not an R specific conference, the Conference on Statistical Practice (Sacramento, February 20 - 22) coming up soon will have some interesting R content. I am particularly looking forward to the talk by Songtao Wang on Thinking Statistically in Social Science and Humanities.
The NICAR Confererence produced by the nonprofit Investigative Reporters and Editors Inc. (New Orleans, March 5 - 8) is a gathering of R and Python savvy journalists using R as an every day tool. Last year, I found the workshops and R training sessions to be outstanding.
The PAA Annual Meeting (Washington D.C., April 22 - 25) is an opportunity to meet demographers, economists public health professionals and sociologists using R.
The R Conference New York (May, 7 - 9), the re-branded NY R Conference of previous years, promises to be the major East Coast R event of the year with a diverse and talented roster of speakers.
The European R Users meeting, eRUM (May, 27 - 30) will be the first major European R conference of the season. Note that Sharon Machlis, Director of Data & Analytics at IDG Communications will be one of the keynote speakers. 2020 may be the year that journalism captures the attention of the R Community.
The Symposium on Data Science & Statistics, SDSS, (Pittsburgh, June 3 - 6) has become one of my favorite statistics conferences. Smaller and more manageable than the JSM, talks are sure to be infused with R content. Note that there will be a session on Data Journalism and Visualization.
Now in its twelfth year, R / Finance (Chicago, June 5 - 6) has set the bar for small, single-session, collegial, technical R conferences. Focusing on Financial applications with a serious dose of advanced time series applications, R / Finance provides the opportunity to interact with professionals who put their money on R.
Always the pivotal event of the R Community, useR! 2020 (St. Louis, July 7 -10) is sure to be a great event and a good time. The final program has not been published (Note that the Call for papers is still open.), but the tutorial sessions alone indicate that useR! 2020 be an outstanding educational opportunity.
The useR! 2020 European Hub Conference (Munich, July 7 - 10) will feature live talks as well as video streaming sessions from useR! 2020 in St. Louis. This innovative format promises to be an important event in its own right and a satisfying community experience.
The JSM (Philadelphia, August 1 - 6), the mother of all statistical conferences, is expecting more than 6,500 attendees from 52 countries. This is an event you have to train for, but with some preparation and a little planning the JSM can be an opportunity to interact with statisticians who depend on the deep statistical knowledge embedded in R.
The Bioconductor Conference, BioC 2020, the event where Software and Biology Connect (Boston, July 29 - 30) is the premier conference for R and Genomics. If you have an interest in learning about cutting edge statistical applications using the big data of modern Biology, this the conference to attend.
The Why R? 2020 Conference (Warsaw, August 27 - 30) is not only positioned to be the third significant European R conference of the year, the organizers have developed an ambitious and innovative strategy of supporting a number of satellite pre-meetings that stretch from Warsaw to Limerick and South Africa. I think this is a fantastic community initiative that represents a commitment to build the R Community in under served areas.

There is no website yet, but it is breaking news that R / Medicine 2020 will be held in Philadelphia from August 27th to 29th. In its third year, R / Medicine is establishing itself as the R conference for physicians seeking to advance clinical practice with R fueled data science.
The LatinR 2020 Conference (Montevideo, October 7 - 9) will be a major South American event. The Call for papers is open. Look here for previous programs.
The BiocAsia 2020 Conference (Beijing, October 17 - 18), the yearly Asian Bioconductor event, has confirmed that Robert Gentleman will be speaking.

Other conferences on my radar are:
* The R Conference Dublin (June)
* The CascadiaRConf (Eugene, OR)
* R / Pharma which will most likely be held at Harvard University in August.
* BioCEurope which will likely be held in December.

Please let me know what upcoming conferences I may have missed by adding them to the comments section of this post.

December 2019: "Top 40" New R Packages

Mon, 20 Jan 2020 00:00:00 +0000

One hundred fifty-two packages made it to CRAN in December. Here are my “Top 40” picks in ten categories: Data, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Data

climate v0.3.0: Provides access to meteorological and hydrological data from OGIMET, University of Wyoming - atmospheric vertical profiling data, and Polish Institute of Meteorology and Water Management - National Research Institute. Look here for more information as well as the vignette.

CCAMLRGIS v3.0.1: Loads and creates spatial data, including layers and tools that are relevant to the activities of the Commission for the Conservation of Antarctic Marine Living Resources ( CCAMLR). Have a look at the vignette.

schrute v0.1.1: Contains the complete scripts from the American version of the Office television show in tibble format. Have a look at the vignette and practice NLP.

simfinR v0.1.0: Provides access to SimFin financial data including balance sheets, cash flow and income statements through the api. Look here for details.

statcanR v0.1.0: Provides access to Statistics Canada’s Web Data Service. See Warin & Le Duc (2019) and the vignette.

Genomics

ampir v0.1.0: Implements a toolkit to predict antimicrobial peptides from protein sequences on a genome-wide scale, including an SVM model trained on publicly available antimicrobial peptide data using calculated physico-chemical and compositional sequence properties described in Meher et al. (2017). There is a brief Introduction.

simplePHENOTYPES v1.0.5: Implements algorithms for simulating pleiotropy and Linkage Disequilibrium under additive, dominance and epistatic models. See Lipka et al. (2012) and Rice and Lipka (2019) for background and the vignette for an introduction.

TreeTools v0.1.3: Provides functions for the creation, modification and analysis of phylogenetic trees and the import and export of trees from Newick, Nexus (Maddison et al. 1997), and TNT formats. There are vignettes on Loading Data, Loading Trees, and Navigating the File System.

Machine Learning

AzureVision v1.0.0: Implements an interface to Azure Computer Vision and Azure Custom Vision which allow users to leverage the cloud to carry out visual recognition tasks using advanced image processing models. There is a vignette on Computer Vision and another on Custom Vision.

dann v0.1.0: Implements discriminant Adaptive Nearest Neighbor Classification, variation of k nearest neighbors where the neighborhood is elongated along class boundaries. See Hastie (1995) for details. There is an Introduction and a vignette on Sub-dann.

eventstream v0.1.0: Provides functions to extract and classify events in contiguous spatio-temporal data streams of 2 or 3 dimensions. For details see Kandanaarachchi et al. 2018. There is an example in README.

isotree v0.1.8: Provides multi-threaded implementations of isolation forest, extended isolation forest, SCiForest, and fair-cut forest for isolation-based outlier detection, clustered outlier detection, distance or similarity approximation, and imputation of missing values as described in Cortes (2019). Look here for an example.

mlr3proba v0.1.1: Extends mlr3 for probabilistic supervised learning that includes probabilistic and interval regression, survival modeling, and other specialized models. There is a vignette on Survival Analysis.

NLPclient v1.0: Implements an interface to the Stanford CoreNLP annotation client which includes a part-of-speech (POS) tagger, a named entity recognizer (NER), a parser, and a co-reference resolution system. See README for installation details.

stray v0.1.0: Modifies the HDoutliers package for outlier detection in high dimensional data to include the algorithm proposed in Talagala, Hyndman and Smith-Miles (2019).

tfhub v0.7.0: is a library for the publication, discovery, and consumption of reusable parts of machine learning models. Modules comprise self-contained parts of TensorFlow graphs along with weights and assets that can be reused across different tasks in a process known as transfer learning. There is an Overview and vignettes on Key Concepts and using TensorFlow with Keras.

Mathematics

dual v0.0.3: Implements automatic differentiation using dual numbers and returns the output value of a mathematical function along with its exact first derivative (or gradient). For more details see Baydin et al. (2018).

set6 v0.1.0: Provides an object-oriented interface for constructing and manipulating mathematical sets, including (countably finite) sets, tuples, intervals (countably infinite or uncountable), and fuzzy variants. using R6. See the vignette for an introduction.

Medicine

LARisk v0.1.0: Provides functions to compute lifetime attributable risk of radiation-induced cancer. See Gonzalez et al. (2012) for background and the vignette for an example.

SCtools v0.3.0: Provides extensions to the synthetic controls analyses performed by the package Synth as detailed in Abadie et al. (2011) that include generating and plotting placebos, post/pre-MSPE (mean squared prediction error) significance tests and plots, and calculating average treatment effects for multiple treated units. There is a vignette on replicating the Basque Study and another on Alcohol Consumption.

Science

chronosphere v0.2.0: Provides functions to facilitate the spatial analyses in (paleo)environmental/ecological research and serves as a gateway to plate tectonic reconstructions, deep time global climate model results as well as fossil occurrence datasets such as the Paleobiology Database and the PaleoReefs Database. See the vignette for an introduction.Chronosphere.png

OCNet v0.1.1: Provides functions to generate analyze Optimal Channel Networks (OCNs): oriented spanning trees reproducing all scaling features characteristic of real, natural river networks. See Rinaldo et al. (2014) for an overview on the OCN concept, Furrer and Sain (2010) for the construct used, and the vignette for examples.

Statistics

bnma v1.0.0: Provides functions for network meta-analyses using Bayesian framework of Dias et al. (2013). See the vignette.

npsurvSS v1.0.1: Provides sample size and power calculations for common non-parametric tests in survival analysis including the difference in (or ratio of) t-year survival, difference in (or ratio of) p-th percentile survival, difference in (or ratio of) restricted mean survival time, and the weighted log-rank test. There are vignettes on Basic Functions, Optimal Randomization Ratio, and Delayed Treatment Effect.

sail v0.1.0: Implements sparse additive interaction learning with the strong heredity property, i.e., an interaction is selected only if its corresponding main effects are also included. See Bhatnagar et al. (2019) for background. There is also an Introduction and a vignette on supplying a user-defined design matrix.

SequenceSpikeSlab v0.1.1: Implements the algorithms described in Van Erven & Szabo (2018) to calculate the exact Bayes posterior for the Sparse Normal Sequence Model. See the vignette.

tcensReg v0.1.5: Implements maximum likelihood estimation (MLE) assuming an underlying left truncated normal distribution with left censoring described in Williams et al. (2019). See the vignette.

univariateML v1.0.0: Looks back to the roots of maximum likelihood estimation (Fisher (1921) to provide functions for the ML estimation of uni variate densities. There is an Overview and vignettes on Copula Modeling and Distributions.

Time Series

imputeFin v0.1.0: Provides functions to impute the missing values based on modeling the time series with a random walk or an autoregressive (AR) model, convenient to model log-prices and log-volumes in financial data. See Liu et al. (2019) for background and the vignette for examples.

VLTimeCausality v0.1.0: Implements a framework to infer causality on a pair of time series of real numbers based on variable-lag Granger causality and transfer entropy. See Zheleva & Berger-Wolf (2019) for the details and the vignette for examples.

Utilities

asciicast v1.0.0: Implements tools to record screen casts from R scripts and convert them to animated SVG images for use in README files and blog posts. It includes asciinema-player as an HTML widget, and a knitr engine, to embed ascii screen casts in R Markdown documents. There is a vignette.

funneljoin v0.1.0: Implements a time-based joins to analyze sequence of events, both in memory and out of memory. See the vignette for details.

hardhat v0.1.1: Provides tools to reduce the burden around building new modeling packages by providing functionality for preprocessing, predicting, and validating input. There is an Introduction and vignettes on Forging Data and Molding Data.

proffer v0.0.2: Builds on pprof to provide profiling tools capable of detecting sources of slowness in R code. Look here for more information.

robenblas v0.2.0: Facilitates downloading, compiling and linking the OpenBLAS library for users of any GNU/Linux distribution. See README for help.

sortable v0.4.2: Provides functions to enables drag-and-drop behavior in Shiny apps, by exposing the functionality of the SortableJS JavaScript library as an htmlwidget. There is a live demo on Using Sortable and another on Using Sortable widgets, and a vignette on the Interface to SortableJS.

sparkhail v0.1.1: Implements a sparklyr interface to Hail, an open-source, general-purpose, Python based data analysis tool with additional data types and methods for working with genomic data, that has been built to scale and provide first-class support for multi-dimensional structured data which is typical of genome-wide association studies. See README for information on how to use the package.

trimmer v0.8.1: Implements a lightweight toolkit to reduce the size of a list object based on user input. See the vignette.

Visualization

gggibbous v0.1.0: Extends ggplot2 to offer moon charts, pie charts where the proportions are shown as crescent or gibbous portions of a circle, like the lit and unlit portions of the moon. It i all illuminated in the vignette.

patchwork v1.0.0: Extends the ggplot2 API to allow for arbitrarily complex plot compositions by providing mathematical operators for combining multiple plots. See the vignette for examples.

RStudio Blogs 2019

Tue, 31 Dec 2019 00:00:00 +0000

If you are lucky enough to have some extra time for discretionary reading during the holiday season, you may find it interesting (and rewarding) to sample some of the nearly two hundred posts written across the various RStudio blogs.

R Views

R Views, our blog devoted to the R Community and the R Language, published over sixty posts in 2019. Many of these were contributed by guest authors from the R Community who volunteered to share some outstanding work. Among my favorites are the multi-part posts that explored data science modeling issues in some detail. These include Roland Stevenson’s three-part series on Multiple Hypothesis Testing and A/B Testing, the four-part series on Analyzing the HIV pandemic by Andrie de Vries and Armand Bester, and Jonathan Reginstein’s two-part series on Tech Dividends.

RStudio Blog

The RStudio blog is the place to go for official information on RStudio. It includes posts on open-source and commercial products, events, and company news. Just scanning the summary paragraphs will give you a good overview of what went on at RStudio this past year. Among my favorite posts for the year is Lou Bajuk’s take on the complementary roles of R and Python: R vs. Python: What’s the best language for Data Science?.

TensorFlow for R Blog

The TensorFlow for R Blog provides “nuts and bolts” reading on building TensorFlow models that ought to be on the list of every data scientist working in R. The posts cover an amazingly wide range of cutting edge topics. For example, see Sigrid Keydana’s recent posts Differential Privacy with TensorFlow, and Getting started with Keras from R - the 2020 edition.

Tidyverse Blog

The Tidyverse Blog offers insight into Tidyverse packages and capabilities at all levels. Scan the summaries like you would a bookshelf in your favorite technical bookstore, and pick out something new like Davis Vaughan’s exposition of the new hardhat package which provides tools for developing new modeling packages, or take a deep dive into task queues with Gábor Csárdi’s Multi Process Task Queue in 100 Lines of R Code.

Ursa Labs Blog

Ursa Labs is a project devoted to open source data science and cross-language software sponsored by RStudio along with several other organizations for which we have great hope. Wes McKinney’s post Ursa Labs Team Report August to December 2019 provides an overview of the progress made in 2019.

Happy Reading!
and
Happy New Year!
from all of us at RStudio.

November 2019: "Top 40" New R Packages

Fri, 20 Dec 2019 00:00:00 +0000

One hundred forty-four new packages made it to CRAN in November. Here are my picks for the “Top 40” in eight categories: Computational Methods, Data, Genomics, Machine Learning, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

calculus v0.1.1: Provides C++ optimized functions for numerical and symbolic calculus including symbolic arithmetic, tensor calculus, Einstein summation convention, Taylor series expansion, multivariate Hermite polynomials and much more.

Jaya v0.1.9: Implements a gradient-free algorithm, without hyperparameters, for solving both constrained and unconstrained optimization problems. See Rao (2016) for details and the vignette for how to use the package.

treenomial v1.1.1: Provides functions for creating and comparing polynomials that uniquely describe trees as introduced in Liu (2019). See README for information.

Data

eudract v0.9.0: Provides access to the European Clinical Trials Data Base ( EudraCT), which summarizes of all registered clinical trial results. The intent is to prevent non-reporting of negative results and provide open-access to results to inform future research. There is a vignette.

ozmaps v0.2.0: Provides maps of Australian coastline and administrative regions as well as simple functions for country or state maps of Australia, and in-built data sets of administrative regions from the Australian Bureau of Statistics. See README for examples.

presentes v0.1.0: Provides a compilation and digitization of the official registry of victims of state terrorism in Argentina during the last military coup. The original data comes from RUVTE-ILID (2019) research and the List of Victims.

VancouvR : Provides a wrapper for the City of Vancouver Open Data API. There is an Introduction.

wiesbaden v1.2.0: Implements an interface to retrieve and import data from different databases of the Federal Statistical Office of Germany (DESTATIS). There is a vignette.

Genomics

biocompute v1.0.3: Provides tools to create, validate, and export BioCompute Objects as described in King et al. (2019). There is an Introduction and a vignette on Authoring Biocompute Objects.

diem v1.0: Implements a novel semi-supervised machine learning classier DIEM, Debris Identification using Expectation Maximization, to identify debris-containing droplets from a droplet-based single cell/nucleus RNA-seq. See the vignette for details.

Machine Learning

azuremlsdk v0.5.7: Implements an interface to the Azure Machine Learning Software Development Kit enabling data scientists to train, deploy, automate, and manage machine learning models on the Azure Machine Learning service. There are vignettes on Setup and Installation.

hereR v0.2.0: Implements an interface to the HERE REST APIs which provide information on geocoding, routing directions, traffic flow, and weather forecasts. There are vignettes on Authentication, the Geocoder API, the Routing API, the Traffic API, and the Weather API.

hilbertSimilarity v0.4.3: Uses Hilbert Curves to develop the notion of Hilbert Similarity to quantify the similarity between samples in high dimensional data. There are vignettes on Comparing Samples and Identifying Treatment Effects.

orf v0.1.2: Implements the Ordered Forest estimator as developed in Lechner & Okasa (2019) to estimate the conditional probabilities of models with ordered categorical outcome (ordered choice models). See the vignette for details.

RPEClust v0.1.0: Implements the random projection ensemble clustering algorithm described in Anderlucci et al.(2019) and Raftery and Dean (2006).

Statistics

DiffXTables v0.0.2: Provides functions for statistical hypothesis testing of pattern heterogeneity via differences in underlying distributions across two or more contingency tables. It includes the comparative chi-squared test, the Sharma-Song test, and the heterogeneity test. See the vignette for details.

effectsize v0.0.1: Provides functions to work with indices of effect size and standardized parameters for a wide variety of models (See Lüdecke, Waggoner & Makowski (2019).) There are vignettes on Bayesian Models, Converting Between Incices, Automated Interpretation of Indices, Data Standardization and Parameter Standardization.

exPrior v1.0.1: Provides practitioners of statistics in geology, hydrology, etc. with a tool for deriving prior distributions for Bayesian inference. See the vignette.

fitHeavyTail v0.1.1: Implements robust estimation methods for the mean vector and covariance matrix from data (possibly containing NAs) under multivariate heavy-tailed distributions such as angular Gaussian, Cauchy, and Student’s t. See Sun et al. (2014), Sun et al. (2015), Liu and Rubin (1995) and Zhou et al. (2015) for background, and the vignette for examples.

mixl v1.1: Provides functions for simulated maximum likelihood estimation of multinomial logit models, mixed models, random coefficients and hybrid choice models. See Molloy et al. (2019) for details and the User Guide.

MKdescr v0.5: Provides functions to compute a standardized interquartile range (IQR), a Huber-type skipped mean as described in Hampel (1985), a robust coefficient of variation as described in Arachchige et al. (2019), a robust signal to noise ratio (SNR), and more. See the vignette.

MNLpred v0.0.1: Provides functions to return simulated predicted probabilities and first differences for multinomial logit models. The methodological approach is based on the principles laid out by King, Tomz, and Wittenberg (2000). See the vignette for examples.

pdqr v0.2.0: Provides functions to create, transform, and summarize custom discrete and continuous random variables with distribution functions that are analogues of p*(), d*(), q*(), and r*(). There are vignettes on Creating, Converting, Transforming, and Summarizing pdqr functions.

tensorregression v1.0: Implements the generalized tensor regression in Xu, Hu and Wang (2019) to solve tensor-response regression given covariates on multiple modes with alternating updating algorithm.

tidydice: v0.0.4: Provides functions for basic statistical experiments, that can be used for teaching introductory statistics. See the vignette.

tvgeom v1.0.1: Implements the probability mass, distribution, quantile, and random number generating functions for the time-varying right-truncated geometric distribution. See the vignette for background.

Time Series

gravitas v0.1.0: Provides tools for systematically exploring large quantities of temporal data across different temporal granularities (deconstructions of time) by visualizing probability distributions. There are vignettes on exploring probability distributions for cricket and for bivariate temporal franularities.

smoots v1.0.1: Provides nonparametric estimates of trend and its derivatives in equidistant time series with short-memory stationary errors. See Feng and Gries (2017) for the methods employed, and see README for an example.

tsfgrnn v0.1.0: Implements a general regression neural network (GRNN), a variant of a radial basis function network, for forecasting time series. See Martinez et al. (2019) and Yan (2012) for background and the vignette for examples.

Utilities

dipsaus v0.0.3: Provides enhancement functions that fall into four categories: shiny input widgets; high-performance computing using RcppParallel and future; functions to modify R calls and convert numbers, strings, and other objects; and utility functions to get system information such as CPU chipset, memory limit, etc. See the vignettes: Asynchronous Evaluator, R Expression Add-ons, Shiny Customized Widgets, and Utility Functions.

extraoperators v0.1.1: Provides operator functions for common tasks such as logical or relational comparisons, finding indices and subsetting. See the vignette for details.

gluedown v1.0.1: Provides functions to transition between R vectors and markdown text. Users can create vectors in R, glue strings together with the markdown syntax, and print formatted vectors directly to the document. This package primarily uses GitHub Flavored Markdown. There is a vignette on GitHub Flavored Markdown and another on Printing Markdown.

googlesheets4 v0.1.0: Provides functions for interacting with Google Sheets through the Sheets API v4. See README for help.

hdd v0.1.0: Provides a data class for importing and manipulating out of memory data sets. See the Introduction.

RVerbalExpressions v0.1.0: Provides tools to build regular expressions using grammar and functionality inspired by VerbalExpressions. See the vignette for examples.

shinyMobile v0.1.0: Provides tools for building shiny apps for iOS, Android, and desktop computers as well as beautiful shiny gadgets. shinyMobile is built on top of the latest ‘Framework7’ template. There is a Getting Started Guide and vignettes on Dark-Theme, Gadgets, Single-Layout, Split-Layout, Tabs-Layout, and Tools.

tidycwl v1.0.4: Implements the Common Workflow Language for describing data analysis workflows. See the vignette for details.

Visualization

barplot3d v1.0.1: Provides functions for creating 3D plots including sequence context plots used in DNA sequencing analysis. See the vignette for examples.

fplot v0.2.0: Provides functions to plot regular/weighted/conditional distributions by using formulas. See the vignette for examples.

robvis: v0.3.0: Provides functions for visualizing risk-of-bias assessments performed as part of a systematic review, providing tools for randomized controlled trials ( Sterne et al. (2019)), non-randomized studies of interventions ( Sterne et al (2016)), and diagnostic accuracy studies ( Whiting et al (2011)). There is a vignette.

October 2019: "Top 40" New R Packages

Mon, 18 Nov 2019 00:00:00 +0000

Two Hundred twenty-three new packages made it to CRAN in October. Here are my “Top 40” picks in ten categories: Computational Methods, Data, Genomics, Machine Learning, Mathematics, Medicine, Pharmacology, Statistics, Utilities, and Visualization.

Computational Methods

admmDensestSubmatrix v0.1.0: Implements a method to identify the densest sub-matrix in a given or sampled binary matrix. See Bombina et al. (2019) for the technical details and the vignette for examples.

mbend v1.2.3: Provides functions to “bend”” non-positive-definite (symmetric) matrices to positive-definite matrices using weighted and unweighted methods. See Jorjani et al. (2003) and Schaeffer (2010) for background and the vignette for an Introduction.

Data

cqcr v0.1.2: Provides access to data from the Care Quality Commission, the health and adult social care regulator for England. Data available under the Open Government License include information on service providers, hospitals, care homes, and medical clinics locations, and ratings and inspection reports.

fpp3 v0.1: Contains all data sets required for the examples and exercises in the book Forecasting: principles and practice by Rob J Hyndman and George Athanasopoulos.

fsbrain v0.0.2: Provides high-level access to FreeSurfer neuroimaging data on the level of subjects and groups. There is a vignette.

opendatatoronto v0.1.0: Provides access to data from the City of Toronto Open Data Portal. There is an Introduction and vignettes on Geospatial Data, Zip Resources, Retrieving Multiple Resources, and Retrieving XLS/XLSX Resources.

povcalnetR v0.1.0: Provides an interface to Povcalnet, a computational tool that allows users to estimate poverty rates for regions, sets of countries or individual countries, over time, and at any poverty line that is managed by the World Bank’s development economics division. There is a Getting Started Guide, and vignettes on Examples and Advanced Usage.

Genomics

dynwrap v1.1.4: Provides functions to infer trajectories from single-cell data, represent them into a common format, and adapt them. See Saelens et al. (2019) for background. There are vignettes on Containers, Scripts, Adding Methods, and Wrapping Trajectories.

phyr v1.0.2: Provides a collection of functions to do model-based phylogenetic analysis, including functions to calculate community phylogenetic diversity, to estimate correlations among functional traits while accounting for phylogenetic relationships, and to fit phylogenetic generalized linear mixed models. The Bayesian phylogenetic generalized linear mixed models are fitted with the INLA package. There is a Performance Benchmark and vignettes on Usage and Plotting.

Machine Learning

cwbtools v0.1.0: Provides tools to create, modify, and manage Corpus Workbench (CWB) Corpora. See Evert and Hardie (2011) for background, and the vignettes Introducing cwbtools and Europal for information on the package.

discrim v0.0.1: Provides bindings for additional classification models for use with the parsnip package including linear discriminate (See Fisher (1936).), regularized discriminant analysis (See Friedman (1989).), and flexible discriminate analysis (See (Hastie et al. (1994).), as well as naive for Bayes classifiers Hand and Yu (2007).

forecastML v0.5.0: Provides functions for forecasting time series using machine learning models and an approach inspired by Bergmeir, Hyndman, and Koo’s (2018). There is an Overview, and vignettes on Customizing Wrapper Functions, Multiple Time Series, and Custom Feature Lags.

interpret v0.1.23: Implements the Explainable Boosting Machine (EBM) framework for machine learning interpretability. See Caruana et al. (2015), for details, and look here for help with the package.

mlr3pipelines v0.1.1: Implements a dataflow programming toolkit that enriches mlr3 with a diverse set of pipelining operators that can be composed into graphs. Operations exist for data preprocessing, model fitting, and ensemble learning. There is an Introduction and a vignette on Comparing Frameworks.

postDoubleR v1.4.12: Implements the double/debiased machine learning algorithm described in Chernozhukov et al. (2017).

SLEMI v1.0: Implements the method described in Jetka et al. (2019) for estimating mutual information and channel capacity from experimental data by classification procedures (logistic regression). The vignette describes how to use the package.

tfprobability v0.0.2: Provides an interface to TensorFlow Probability, a Python library built on TensorFlow that makes it easy to combine probabilistic models and deep learning on modern hardware including TPUs and GPUs. There are vignettes on Dynamic Linear Models, Multi-level Modeling with Hamiltonian Monte Carlo, and Uncertainty Estimates.

Mathematics

Ryacas0 v0.4.2: Provides and interface to the yacas computer algebra system. There is a Getting Started Guide and vignettes on Ryacas functionality, a Naive Bayes Model, a State Space Model and Matric and Vector Objects.

silicate v0.2.0: Provides functions to generate common forms for complex hierarchical and relational data structures inspired by the simplicial complex. There is a vignette.

Medicine

diyar v0.0.2: Implements multistage record linkage and case definition for epidemiological analyses. There are vignettes on Case Definitions and Multistage Deterministic Linkage.

ushr v0.1.0: Presents an analysis of longitudinal data of HIV decline in patients on antiretroviral therapy using the canonical biphasic exponential decay model described in Perelson et al. (1997) and Wu and Ding (1999), and includes options to calculate the time to viral suppression. The vignette walks through the analysis.

Pharmacology

chlorpromazineR v0.1.2: Provides functions to convert doses of antipsychotic medications to chlorpromazine-equivalent doses using conversion keys generated from Gardner et. al (2010) and Leucht et al. (201). See the vignette.

ubiquity v1.0.0: Implements a complete work flow for the analysis of pharmacokinetic pharmacodynamic (PKPD), physiologically-based pharmacokinetic (PBPK) and systems pharmacology models including: creation of ODE-based models, pooled parameter estimation, simulations for clinical trial design and modeling assays and deployment with Shiny and reporting with PowerPoint. There are vignettes on Deployment, Estimation, Language, NCA, Reporting, Simulation, and Titration.

Statistics

DPQ v0.3-5: Provides the computations for approximations and alternatives for the density, cumulative density and quantile functions for R’s probability distributions. This package from researchers working with R-core is intended primarily for researchers working to improve R’s beta, gamma and related distributions. See the vignettes Non-central Chi-Swuared Probabilities - Algorithms in R and Computing Beta for Large Arguments.

hypr v0.1.3: Provides functions to translate between experimental null hypotheses, hypothesis matrices, and contrast matrices as used in linear regression models based on the method described in Schad et al. (2019). There is an Introduction and vignettes on Contrasts and Linear Regression.

HTLR v0.4-1: Implements Bayesian multinomial logistic regression based on heavy-tailed (hyper-LASSO, non-convex) priors for high-dimensional feature selection. Li and Yao (2018) provides a detailed description of the method, and the vignette introduces the package.

meteorits v0.1.0: Provides a unified mixture-of-experts (ME) modeling and estimation framework to model, cluster and classify heterogeneous data in many complex situations where the data are distributed according to non-normal, possibly skewed distributions. See Chamroukhi et al. (2009), Chamroukhi (2010). Chamroukhi (2015), Chamroukhi (2016), and Chamroukhi (2017) for background, and the vignettes NMoE, SNMoE, StMoE and tMoE.

mHHMbayes v0.1.1: Implements multilevel (mixed or random effects) hidden Markov model using Bayesian estimation in R. For background see Rabiner (1989) and de Haan-Rietdijk et al. (2017). There is a Tutorial and a vignette on Estimation.

nhm v0.1.0: Provides functions to fit non-homogeneous Markov multistate models and misclassification-type hidden Markov models in continuous time to intermittently observed data. See Titman (2011) for background and the User Guide for package details.

mniw v1.0: Implements the Matrix-Normal Inverse-Wishart (MNIW) distribution, as well as the the Matrix-Normal, Matrix-T, Wishart, and Inverse-Wishart distributions. the vignette does the math.

PosteriorBootstrap v0.1.0: Implements a non-parametric statistical model using a parallelized Monte Carlo sampling scheme that allows non-parametric inference to be regularized for small sample sizes. The method is described in full in Lyddon et al. (2018). There is a vignette.

spBFA v1.0: Implements functions for spatial Bayesian non-parametric factor analysis model with inference. See Berchuck et al. (2019) for the technical background and the vignette for package details.

VARshrink v0.3.1: Provides functions that integrate shrinkage estimation with vector autoregressive models including nonparametric, parametric, and semiparametric methods such as the multivariate ridge regression (See Golub et al. (1979).), a James-Stein type nonparametric shrinkage method (See Opgen-Rhein and Strimmer (2007).), and Bayesian estimation methods as in Lee et al. (2016) and Ni and Sun (2005). There is a vignette.

Utilities

geospark v0.2.1: Provides simple features bindings to GeoSpark extending the sparklyr package to bring geocomputing to Spark distributed systems. See README for more information.

laelmachine v1.0.0: Provides functions to assign meaningful labels to data frame columns, and to manage label assignment rules in yaml files making it easy to use the same labels in multiple projects. There is a Getting Started Guide and vignettes on Altering lama-dictionaries, Creating lama-dictionaries, and Translating Variables.

renv v0.8-3: Implements a dependency management toolkit that enables creating and managing project-local R libraries, saving the state of these libraries and later restoring them. There is an Introduction and a series of vignettes: Continuous Integration, Collaborating with renv, Using renv with Docker, Frequently Asked Questions, Local Sources, Lockfiles, and Using Python with renv.

ymlthis v0.1.0: Provides functions to write YAML front matter for R Markdown and related documents. There is an Introduction to the package, a YAML Field Guide and an YAML Overview.

Visualization

geometr v0.1.1: Provides tools that generate and process tidy geometric shapes. There is a vignette.

ggVennDiagram v0.3: Provides functions to generae publication quality Venn diagrams using two to four sets. See README for more information.

rayrender v0.4.2: Provides functions to render scenes using path tracing including building 3D scenes out of geometrical shapes and 3D models in the Wavefront OBJ file format. Look here for more information.

sankeywheel v0.1.0: Implements bindings to the Highcharts library to provide a simple way to draw dependency wheels and sankey diagrams. There is a vignette.

A First Look at Confidence Distributions

Tue, 05 Nov 2019 00:00:00 +0000

Using a probability distribution to characterize uncertainty is at the core of statistical inference. So, it seems natural to try to summarize the information about the parameters in statistical models with probability distributions. R. A. Fisher thought so. In fact, he expended a great deal of effort over more than thirty years, and put his professional reputation on the line trying to do so, with only limited success. Fisher’s central difficulty was that, in the Frequentist tradition to which he was committed, parameters are not random variables. They are fixed and immutable constituents of the statistical models describing the behavior of populations, which we must estimate because we generally only have access to samples from populations, not to the full populations themselves. Now Bayesians, of course, characterize parameters with probability distributions from the get-go. Parameters are given prior distributions and combined with the likelihood function generated by the data to produce posterior distributions that characterize the parameters. Fisher wanted the posterior distributions without having to assume the priors. This was a key motivating idea for his work on Fiducial probability.

A few statisticians apparently quietly worked on this program throughout the twentieth century, even though Fisher’s Fiducial ideas were mostly forgotten and not part of the mainstream statistical current. D. R. Cox (Cox (1958)), for example, pioneered the idea of constructing confidence distributions from confidence intervals, and Bradley Efron (Efron (1998)) expressed great optimism that Fisher’s work in this area would become important in the twenty-first century. (Efron’s paper is a masterpiece that summarizes a good bit of twentieth-century statistical research.)

Recently, however, there seems to have been a resurgence of Fisher’s ideas among statisticians interested in chasing the idea of an orthodox Frequentist view of parameter distributions. The 2013 paper of Xie and Singh lays out the modern theory of confidence distributions as a fundamental idea that organizes a great deal of statistical practice. In the initial Summary, the authors write:

. . .the concept of a confidence distribution subsumes and unifies a wide range of examples, from regular parametric (fiducial distribution) examples to bootstrap distributions, significance (p-value) functions, normalized likelihood functions, and, in some cases, Bayesian priors and posteriors.

Later in the paper, they go on to define a confidence distribution as:

A function H_n(·) = H_n(x, ·) on X × $\Theta$ → [0, 1] is called a confidence distribution (CD) for a parameter $\theta$, if
* R1) For each given x ∈ X , H_n(·) is a cumulative distribution function on $\Theta$;
* R2) At the true parameter value $\theta$ = $\theta$₀, H_n($\theta$₀) ≡ H_n(x, $\theta$₀), as a function of the sample

The simplest example of a confidence distribution I could find that is adequate to illustrate some of the key concepts comes from the book Confidence, Likelihood, Probability: Statistical Inference with Confidence Distributions by Schweder and Hjort. On page 62, the authors point out that the distribution of the p-value for the parameter $\theta$ describing the probability of success for a binomial trial can be considered as an approximate confidence distribution for $\theta$. The distribution is approximate because the distribution is discrete and the “half-correction” is used to improve the approximation. Note that because the p-values follow uniform distributions under the null hypothesis, it should be clear that the assumptions of the definition above are satisfied.

Suppose Y ~ Bin(n,$\theta$), then

C($\theta$) = P(Y > y₀) + .5 * P(Y = y₀) is a confidence distribution for $\theta$.

To illustrate this, we consider the experiment of realizing 8 successes in 20 trials and write a short helper function.

CD <- function(theta,n=20,y0=8){
            1 - sum(dbinom(x = seq(from = 0, to = y0), size = n, prob = theta)) + 
           .5 * dbinom(x = y0, size = n, prob = theta)}

Here, we compute the CDF and plot it.

library(tidyverse)
library(highcharter)
conf_dist <-  
  tibble(theta = seq(0, 1, by = .01)) %>%  
  mutate(probability = map_dbl(theta, CD))

hchart(conf_dist, "line", hcaes(x = theta, y = probability)) %>%
hc_title(text = "Confidence Distribution for Binomial Model",
         margin = 20, align = "left",
         style = list(color = "black", useHTML = TRUE)) %>%
hc_tooltip(valueDecimals=4, valuePrefix="cum prob = ")

This is nice, but to really see how computing the confidence distribution might be useful, we compute and plot the confidence curve introduced by Birnbaum in his 1961 paper.

conf_curve <-  
  conf_dist %>%  
  mutate(confidence = 2 * abs(.5 - probability))

hchart(conf_curve, "line", hcaes(x = theta, y = confidence)) %>%
  hc_title(text = "Confidence Curve for Binomial Model",
           margin = 20, align = "left",
           style = list(color = "black", useHTML = TRUE)) %>%
  hc_tooltip(valueDecimals=4, valuePrefix="conf level = ")

Pick a point on the left branch of the curve. The y value gives you the level of confidence and the x value is the lower bound of the corresponding confidence interval. Move horizontally across to the right branch to read off the upper end of the confidence interval. So reading up and down the curve you can read off the confidence intervals for any value of confidence.

As a check, we compute the 95% confidence interval for $\theta$ using the normal approximation to the binomial.

ub <- 8 / 20 + (1.96 / 20) * sqrt(8 * 12 / 20)
lb <- 8 / 20 - (1.96 / 20) * sqrt(8 * 12 / 20)
cat("95% CI = [" , lb , "," , ub, "]")

## 95% CI = [ 0.1853 , 0.6147 ]

If you are a Bayesian, there is a really amusing side to confidence distributions. In order to be clear that what they are doing with confidence distributions is in fact different from what Bayesians do when they choose priors, the champions of confidence distributions appeal to epistomology, the study of knowledge and justified belief. On page (xiv) of their book, Schweder and Hjort write:

The concept of confidence distribution is rather basic, but has proved difficult for statisticians to accept. The main reason is perhaps that confidence distributions represent epistemic probability obtained from the aleatory probability of the statistical model (i.e. the chance variation in nature and society), and to face both types of probability at the same time might be challenging. The traditional Bayesian deals only with subjective probability, which is epistemic when based on knowledge, and the frequentist of the Neyman-Wald school deals only with sampling variability, that is, aleatory probability.

Please indulge me while I unpack this. Aleatory probabilities are what nature and the world give us: the decay times of alpha particles, the valuable behaviors of large populations, etc. Hard-core frequentists will only allow themselves to compute aleatory probabilities. As soon as you compute a confidence distribution or even a confidence interval, you are working with epistemic probabilities: what you believe would be true under repeated sampling that may be impossible to actually carry out. But, you are justified in doing this because these epistemic probabilities are anchored in aleatory probabilities. When Bayesians base their choice of priors on rational beliefs based on plausible evidence, they are on the same epistemic footing as frequentists computing confidence intervals. When Bayesians are capricious in choosing their priors, they are not. Asking most statisticians to think about these things gives them headaches. Thus, quietly, in work that builds bridges, ends the Bayesian vs. Frequentist controversy.

Notes:
1. Schweder and Hjolt’s book is really worth owning. Not only does it offer a comprehensive account of confidence distributions and how they may be useful in practice, but it is also a good general reference on statistical inference.

If you are interested in exploring confidence distributions further, have a look at the pvaluefunctions and gmeta that are both on CRAN.

Sept 2019: "Top 40" New R Packages

Tue, 29 Oct 2019 00:00:00 +0000

One hundred and thirteen new packages made it to CRAN in September. Here are my “Top 40” picks in eight categories: Computational Methods, Data, Economics, Machine Learning, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

eRTG3D v0.6.2: Provides functions to create realistic random trajectories in a 3-D space between two given fixed points (conditional empirical random walks), based on empirical distribution functions extracted from observed trajectories (training data), and thus reflect the geometrical movement characteristics of the mover. There are several small vignettes, including sample data sets, linkage to the sf package, and point cloud analysis.

freealg v1.0: Implements the free algebra in R: multivariate polynomials with non-commuting indeterminates. See the vignette for the math.

HypergeoMat v3.0.0: Implements Koev & Edelman’s algorithm (2006) to evaluate the hypergeometric functions of a matrix argument, which appear in random matrix theory. There is a vignette.

opart v2019.1.0: Provides a reference implementation of standard optimal partitioning algorithm in C using square-error loss and Poisson loss functions as described by Maidstone (2016), Hocking (2016), Rigaill (2016), and Fearnhead (2016) that scales quadratically with the number of data points in terms of time-complexity. There are vignettes for Gaussian and Poisson squared error loss.

Data

cde v0.4.1: Facilitates searching, download and plotting of Water Framework Directive (WFD) reporting data for all water bodies within the UK Environment Agency area. This package has been peer-reviewed by rOpenSci. There is a Getting Started Guide and a vignette on output reference.

eph v0.1.1: Provides tools to download and manipulate data from the Argentina Permanent Household Survey. The implemented methods are based on INDEC (2016).

leri v0.0.1: Fetches Landscape Evaporative Response Index (LERI) data using the raster package. The LERI product measures anomalies in actual evapotranspiration, to support drought monitoring and early warning systems. See the vignette for examples.

rwhatsapp v0.2.0: Provides functions to parse and digest history files from the popular messenger service WhatsApp. There is a vignette.

tidyUSDA v0.2.1: Provides a consistent API to pull United States Department of Agriculture census and survey data from the National Agricultural Statistics Service (NASS) QuickStats service. See the vignette.

Economics

bunching v0.8.4: Implements the bunching estimator from economic theory for kinks and knots. There is a vignette on Theory, and another with Examples.

fixest v0.1.2: Provides fast estimation of econometric models with multiple fixed-effects, including ordinary least squares (OLS), generalized linear models (GLM), and the negative binomial. The method to obtain the fixed-effects coefficients is based on Berge (2018). There is a vignette.

raceland v1.0.3: Implements a computational framework for a pattern-based, zoneless analysis, and visualization of (ethno)racial topography for analyzing residential segregation and racial diversity. There is a vignette describing the Computational Framework, one describing Patterns of Racial Landscapes, and a third on SocScape Grids.

Machine Learning

biclustermd v0.1.0: Implements biclustering, a statistical learning technique that simultaneously partitions, and clusters rows and columns of a data matrix in a manner that can deal with missing values. See the vignette for examples.

bbl v0.1.5: Implements supervised learning using Boltzmann Bayes model inference, enabling the classification of data into multiple response groups based on a large number of discrete predictors that can take factor values of heterogeneous levels. See Woo et al. (2016) for background, and the vignette for how to use the package.

corporaexplorer v0.6.3: Implements Shiny apps to dynamically explore collections of texts. Look here for more information.

fairness v1.0.1: Offers various metrics to assess and visualize the algorithmic fairness of predictive and classification models using methods described by Calders and Verwer (2010), Chouldechova (2017), Feldman et al. (2015), Friedler et al. (2018), and Zafar et al. (2017). There is a tutorial for the package.

imagefluency v0.2.1: Provides functions to collect image statistics based on processing fluency theory that include scores for several basic aesthetic principles that facilitate fluent cognitive processing of images: contrast, complexity / simplicity, self-similarity, symmetry, and typicality. See Mayer & Landwehr (2018) and Mayer & Landwehr (2018) for the theoretical background, and the vignette for an introduction.

ineqJD v1.0: Provides functions to compute and decompose Gini, Bonferroni, and Zenga 2007 point and synthetic concentration indexes. See Zenga M. (2015), Zenga & Valli (2017), and Zenga & Valli (2018) for more information.

lmds v0.1.0: Implements Landmark Multi-Dimensional Scaling (LMDS), a dimensionality reduction method scaleable to large numbers of samples, because rather than calculating a complete distance matrix between all pairs of samples, it only calculates the distances between a set of landmarks and the samples. See the README for an example.

modelStudio v0.1.7: Implements an interactive platform to help interpret machine learning models. There is a vignette, and look here for a demo of the interactive features.

nlpred v1.0: Provides methods for obtaining improved estimates of non-linear cross-validated risks obtained using targeted minimum loss-based estimation, estimating equations, and one-step estimation. Cross-validated area under the receiver operating characteristics curve ( LeDell sr al. (2015) ) and other metrics are included. There is a vignette on small sample estimates.

pyMTurkR v1.1: Provides access to the latest Amazon Mechanical Turk’ (‘MTurk’) Requester API (version ‘2017–01–17’), replacing the now deprecated MTurkR package.

stagedtrees v1.0.0: Creates and fits staged event tree probability models, probabilistic graphical models capable of representing asymmetric conditional independence statements among categorical variables. See Görgen et al. (2018), Thwaites & Smith (2017), Barclay et al. (2013), and Smith & Anderson](doi:10.1016/j.artint.2007.05.004) for background, and look here for and overview.

Statistics

confoundr v1.2: Implements three covariate-balance diagnostics for time-varying confounding and selection-bias in complex longitudinal data, as described in Jackson (2016) and Jackson (2019). There is a Demo vignette and another Describing Selection Bias from Dependent Censoring

distributions3 v0.1.1: Provides tools to create and manipulate probability distributions using S3. Generics random(), pdf(), cdf(), and quantile() provide replacements for base R’s r/d/p/q style functions. The documentation for each distribution contains detailed mathematical notes. There are several vignettes: Intro to hypothesis testing, One-sample sign tests, One-sample T confidence interval, One-sample T-tests, Z confidence interval for a mean, One-sample Z-tests for a proportion, One-sample Z-tests, Paired tests, and Two-sample Z-tests.

dobin v0.8.4: Implements a dimension reduction technique for outlier detection, which constructs a set of basis vectors for outlier detection that bring outliers to the forefront using fewer basis vectors. See Kandanaarachchi & Hyndman (2019) for background, and the vignette for a brief introduction.

glmpca v0.1.0: Implements a generalized version of principal components analysis (GLM-PCA) for dimension reduction of non-normally distributed data, such as counts or binary matrices. See Townes et al. (2019) and Townes (2019) for details, and the vignette for examples.

immuneSIM v0.8.7: Provides functions to simulate full B-cell and T-cell receptor repertoires using an in-silico recombination process that includes a wide variety of tunable parameters to introduce noise and biases. See Weber et al. (2019) for background, and look here for information about the package.

irrCAC v1.0: Provides functions to calculate various chance-corrected agreement coefficients (CAC) among two or more raters, including Cohen’s kappa, Conger’s kappa, Fleiss’ kappa, Brennan-Prediger coefficient, Gwet’s AC1/AC2 coefficients, and Krippendorff’s alpha. There are vignettes on benchmarking, Calculating Chance-corrected Agreement Coefficients, and Computing weighted agreement coefficients.

LPBlg v1.2: Provides functions that estimate a density and derive a deviance test to assess if the data distribution deviates significantly from the postulated model, given a postulated model and a set of data. See Algeri S. (2019) for details.

SynthTools v1.0.0: Provides functions to support experimentation with partially synthetic data sets. Confidence interval and standard error formulas have options for either synthetic data sets or multiple imputed data sets. For more information, see Reiter & Raghunathan (2007).

Time Series

fable v0.1.0: Provides a collection of commonly used univariate and multivariate time series forecasting models, including automatically selected exponential smoothing (ETS) and autoregressive integrated moving average (ARIMA) models. There is an Introduction and a vignette on transformations.

nsarfima v0.1.0.0: Provides routines for fitting and simulating data under autoregressive fractionally integrated moving average (ARFIMA) models, without the constraint of stationarity. Two fitting methods are implemented: a pseudo-maximum likelihood method and a minimum distance estimator. See Mayoral (2007) and Beran (1995) for reference.

Utilities

nc v2019.9.16: Provides functions for extracting a data table (row for each match, column for each group) from non-tabular text data using regular expressions. Patterns are defined using a readable syntax that makes it easy to build complex patterns in terms of simpler, re-usable sub-patterns. There is a vignette on capture first match and another on capture all match.

pins v0.2.0: Provides functions that “pin” remote resources into a local cache in order to work offline, improve speed, avoid recomputing, and discover and share resources in local folders, GitHub, Kaggle and RStudio Connect. There is a Getting Started Guide and vignettes on Extending Boards, Using GitHub Boards, Using Kaggle Boards, Using RStudio Connect Boards, Using Website Boards, Using Pins in RStudio, Understanding Boards, and Extending Pins.

queryparser v0.1.1: Provides functions to translate SQL SELECT statements into lists of R expressions.

rawr v0.1.0: Retrieves pure R code from popular R websites, including github, kaggle, datacamp, and R blogs made using blogdown.

Visualization

FunnelPlotR v0.2.1: Implements Spiegelhalter (2005) Funnel plots for reporting standardized ratios, with overdispersion adjustment. The vignette offers examples.

ggBubbles v0.1.4: Implements mini bubble plots to display more information for discrete data than traditional bubble plots do. The vignette provides examples.

gghalves v0.0.1: Implements a ggplot2 extension for easy plotting of half-half geom combinations: think half boxplot and half jitterplot, or half violinplot and half dotplot.

August 2019: "Top 40" R packages

Thu, 26 Sep 2019 00:00:00 +0000

Two hundred and twenty-seven new packages made it to CRAN in August. Quite a few were devoted to medical or genomic applications, and this is reflected in my “Top 40” selections, listed below in nine categories: Computational Methods, Data, Genomics, Machine Learning, Medicine and Pharma, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

fmcmc v0.2-0: Provides a flexible Markov Chain Monte Carlo (MCMC) framework for implementing Metropolis-Hastings algorithms. Thee is a vignette on user-defined kernels and another on workflows.

Mercator v0.9.5: Defines the classes used to explore, cluster, and visualize distance matrices, especially those arising from binary data. See the vignette.

tdigest v0.3.0: Implements the t-Digest construction algorithm by Dunning et al. (2019), which uses a variant of one-dimensional k-means clustering to produce a very compact data structure that allows accurate estimation of quantiles.

Data

arcos v0.8.2: Implements a wrapper for the ARCOS API that returns raw and summarized data frames from the Drug Enforcement Administration’s Automation of Reports and Consolidated Orders System, a database that monitors controlled substances transactions between manufacturers and distributors. There are vignettes on annual-maps, county-analysis, and per-capita-pharmacies.

censusxy v0.1.2: Provides access to the U.S. Census Bureau’s API for batch geocoding American street addresses. See the vignette.

hdfqlr v0.6-1: Implements an interface to HDFql along with helper functions for reading data from and writing data to HDF5 files. For more information, see the reference manual and the vignettes on Benchmarks, Low-level API, and Quick Start.

nhdplusTools v0.3.8: Implements tools documented by the US Environmental Protection Agency for traversing and working with [National Hydrography Dataset Plus](https://www.epa.gov/waterdata/nhdplus-national-hydrography-dataset-plus#targetText=National%20Hydrography%20Dataset%20Plus%20(NHDPlus,with%20the%20U.S.%20Geological%20Survey.) (NHDPlus) data. There is an Introduction and vignettes on plotting and point indexing.

Genomics

getspres v0.1.0: Implements the SPRE (standardized predicted random-effects) statistics to explore heterogeneity in genetic association meta-analyses, as described by Magosi et al. (2019). Look here for a very brief overview and see the vignette for a Tutorial.

simGWAS v0.2.0-2: Provides functions to simulate output from a case-control genome-wide association study (GWAS) with a given causal model. See Fortune and Wallace (2019) for the science, and the vignette for a simulation walk through.

viromeBrowser v1.0.0: Facilitates browsing virome sequencing using annotations in multiple fasta files, and allows users to select and export specific annotated sequences. The vignette shows how to use the package.

whoa v0.0.1: Provides functions to investigate the distribution of genotypes in genotype-by-sequencing (GBS) data where approximate Hardy-Weinberg equilibrium is expected, in order to assess rates of genotyping errors and the dependence of those rates on read depth. See Hendricks et al. (2018) for background and the vignette for a Tutorial.

Machine Learning

flashlight v0.2.0: Provides functions to examine black-box machine-learning models using permutation variable importance ( Fisher et al. (2018) ), ICE profiles, and partial dependence ( Friedman J. H. (2001) ). See the vignette.

imagefx v0.2.0: Provides functions to extract features from images for time-series analysis or machine-learning applications. There is a vignette for analysing video data and another for optical flow analysis.

rTorch v0.0.3: Provides an interface to the Python-based PyTorch machine-learning library. See the README for how to use the interface.

Medicine and Pharma

accept v0.7.0: Provides functions to allow clinicians to predict the rate and severity of future acute exacerbation in Chronic Obstructive Pulmonary Disease (COPD) patients, based on the clinical prediction model published in Adibi et al. (2019). The webapp shows the model.

DRAFT v0.3.0: Fits epidemic data to stochastic models with constant or time-dependent behavior. See Ben-Nun et al. (2019) for a case study, and the vignette for other examples.

idmodelr v0.3.1: Implements a framework that includes simulation and visualization tools for exploring a range of infectious disease models. It is primarily intended as an educational resource. There are vignettes for Model details, Parameter details, and Other resources.

OncoBayes2 v0.4-4: Implements a Bayesian logistic regression model with optional EXchangeability-NonEXchangeability parameter modelling, and includes a safety model that can guide dose-escalation decisions for adaptive oncology Phase I dose-escalation trials involving an arbitrary number of drugs. See Neuenschwander et al. (2008) and Neuenschwander et al. (2016), and the vignette for examples.

PML v1.1: Implements a penalized multi-band learning algorithm to analyze circadian rhythms from accelerometer data. See the vignette for an example.

xgxr v1.0.2: Provides functions to support a structured approach for exploring PKPD data. There is an Overview, a PKPD Single Ascending Dose example, and a vignette on PK Exploration with nlmixr dataset for theophylline.

visit v2.1: Implements a Bayesian Phase I cancer vaccine trial that allows for the simultaneous evaluation of safety and immunogenicity outcomes in the context of vaccine studies. See Wang (2019) for the details of the trial design, and the package overview.

Statistics

baggr v0.1.0: Provides function to fit and compare hierarchical Bayesian meta-analysis models with Stan. See the vignette.

BayesPostEst v0.0.1: Provides functions to generate and plot post-estimation quantities after estimating Bayesian regression models using Markov chain Monte Carlo (MCMC), including Precision-Recall curves (see Beger (2016)) and predicted probabilities, using the methods of Hanmer and Kalkan (2013) and King et al. (2000). The functions can be used with MCMC output generated by any Bayesian estimation tool, including JAGS, BUGS, MCMCpack, and Stan. See the Getting Started Guide.

cotram v0.1-0: Provides functions to implement count transformation models featuring parameters interpretable as discrete hazard ratios, odds ratios, reverse-time discrete hazard ratios, or transformed expectations. For the technical details, see Hothorn et al. (2018) and the vignette.

lax v1.0.0: Provides functions to adjust the standard errors for extreme-value models fitted with evd, evir, extRemes, fExtremes, ismev, POT, and texmex. See the vignette for an overview.

OwenQ v 1.0.2: Implements the Owen Q-function for integer value degrees of freedom, which is useful for calculating the power of equivalence tests. There is a vignette on the Owen Cumulative Function and another on Validation.

Time Series

avar v0.1.0: Implements the allan variance and allan variance linear regression estimator for latent time series models. For the theory, see Guerrier et al.(2016). The vignette contains examples.

feasts v0.1.1: Provides a collection of functions for producing decompositions, statistical summaries, and plots for analyzing tidy time series data. See the vignette for examples.

scorpeak v0.1.2: Provides functions for detecting peaks in time series based on the algorithms described in Girish Palshikar (2009). The vignette contains examples.

TSplotly v1.1.1: Provides functions to create interactive time series plots. See the vignette for an introduction to the package.

Utilities

arrow v0.14.1.1: Provides an interface to the Apache Arrow C++ library. Arrow is a cross-language development platform for in-memory data, which specifies a standardized language-independent columnar memory format.

butcher v0.1.0: Provides S3 generics to axe components of fitted-model objects to reduce the size of model objects saved to disk. There is an introduction and additional vignettes on adding-models-to-butcher and available-axe-methods.

mRpostman v0.2.0: Provides functions to make it easy to connect to your IMAP ( Internet Message Access Protocol ) server and execute commands such as list mailboxes, search for, and fetch messages in a tidy way. See the vignette for details.

pins v0.1.2: Provides a way to “pin”” remote resources into a local cache to work offline, improve speed, and avoid recomputing. Resources can be anything from CSV, JSON, or image files to arbitrary R objects. There is abundant documentation, including a Getting Started Guide and several vignettes: Extending Board, Using GitHub Boards, Using Kaggle Boards, Using RStudio Connect Boards, Understanding Boards, Using Website Boards, Extending Pins, and Using Pins in RStudio.

pmdplyr v0.3.0: Extends dplyr to provide a family of functions for manipulating panel data, including functions to manipulate data based on index variables. There are vignettes on dplyr Variants, Panel Tools, and Panel Maneuvers in dplyr.

tidycells v0.2.1: Provides utilities to read cells from complex tabular data and, using a heuristic method, assign those cells to a columnar or tidy format. See the vignette for an overview.

Visualization

ggpointdensity v0.1.1: Extends ggplot2 to provide a geom for point-density plots, which are a cross between 2D density plots and scatter plots. Look here for examples.

hpackedbubble v0.1.0: Provides a simple way to draw split-packed bubble charts based on Highcharts. See the vignette for examples.

SHAPforxgboost v0.0.2: Provides functions to aid in visual data investigations using SHAP (SHapley Additive exPlanation) visualization plots for XGBoost. Look here for examples.

July 2019 "Top 40" R Packages

Thu, 29 Aug 2019 00:00:00 +0000

One hundred seventy-six new packages made it to CRAN in July. Here are my “Top 40” picks organized into twelve categories: Data, Data Science, Finance, Genomics, Machine Learning, Mathematics, Medicine, Statistics, Time Series, Topological Data Analysis, Utilities and Visualization.

Data

eia v0.3.2: Provides API access to data from the US Energy Information Administration (EIA). Use of the API requires a free API key. See the Package Overview.

litteR v0.4.1: Implements a user interface to analyze litter data: beach litter, riverain litter, floating litter, seafloor litter, etc., in a consistent and reproducible manner. See the vignette for examples.

rSymbiota v1.0.0: Implements an interface to the Symbiota portals, allowing users to query taxon natural history collections that include plants, animals, and fungi. See the vignette to get started.

Data Science

bdpar v1.0.0: Provide a tool to easily build customized data flows to pre-process large volumes of information from different sources.

modeLLtest v1.0.0: Implements the cross-validated difference in means (CVDM) test by Desmarais and Harden (2014) and the cross-validated median fit (CVMF) test by Desmarais and Harden (2012). See the vignette to get started.

Finance

lazytrade v0.3.4: Provide sets of functions and methods to learn and practice data science using the idea of algorithmic trading. See the vignette.

RPEIF v1.0: Computes the influence functions time series of the returns for the risk and performance measures as mentioned in Zhang and Martin (2017) as well as Chen and Martin (2018). The vignette provides some theory and examples.

Genomics

MGDrivE v1.0.0: Implements a testbed for gene drive interventions for mosquito-borne diseases control. See Deredec et al. (2001) and Hancock & Godfray (2007) for the background. There are vignettes for examples, the math, and a complete run.

PACVr v0.8.1: Provides functions to visualize the coverage depth of a complete plastid genome, as well as the equality of its inverted repeat regions in relation to the circular, quadripartite genome structure and the location of individual genes. For details, see Gruenstaeudl and Jenke (2019), and for an example, see the vignette.

Machine Learning

forestRK v0.0-5: Provides functions that calculate common types of splitting criteria used in random forests for classification problems, as well as functions that make predictions based on a single tree or a Forest-R.K. The vignette offers an extended example.

greenclust v1.0.0: Implements a method of iteratively collapsing the rows of a contingency table, two at a time, by selecting the pair of categories whose combination yields a new table with the smallest loss of chi-squared, as described by Greenacre (1988). See the README for an introduction to the package.

imgrec v0.1.0: Implements an interface to Vision AI, the Google image recognition system. See the vignette to get started.

mlr3 v0.1.2: Provides R6 object-oriented programming building blocks for machine learning tasks. Look here for a brief overview.

Mathematics

odin v1.0.1: Provides functions to generate systems of ordinary differential equations (ODE) and integrate them, using a domain specific language (DSL). The DSL uses R’s syntax, but compiles to C to efficiently solve the system. There is a vignette on Discrete Models and another with several examples.

pCODE v0.9.2: Contains an implementation of the parameter cascade method from Ramsay, J. O., Hooker,G., Campbell, D., and Cao, J. (2007) for estimating ordinary differential equation models with missing or complete observations. The vignette shows the math.

Medicine

MBNMAdose v0.2.3: Provides functions to fit Bayesian dose-response, model-based network meta-analysis (MBNMA) that incorporate multiple doses within an agent by modelling different dose-response functions. See Mawdsley et al. (2016) for the theory and the vignette for examples.

qMRI v1.0.1: Implements methods to estimate quantitative maps described in Weiskopf et al. (2013). See the vignette for an example.

Statistics

borrowr v0.1.0: Provides functions to estimate population average treatment effects from a primary data source with borrowing from supplemental sources. Causal estimation is done with either a Bayesian linear model or with Bayesian additive regression trees (BART) to adjust for confounding. See Chipman et al.(2010) Borrowing is done with multisource exchangeability models (MEMs). See Kaizer et al. (2018). There is a vignette.

emax.glm v0.1.2: Implements Expectation Maximization (EM) regression for general linear models. See Hastie et al. (2009) and Zeileis et al. (2017) for background. There is an introduction and vignettes on the EM GLM algorithm, Residual Theory and on Warm up and Exposure in the GLM algorithm.

kosel v0.0.1: Implements functions to perform variable selection for many types of L1-regularised regressions using the revisited knockoffs procedure. See Gegout et al. (2019) for the details.

mipred v0.0.1: Calibrates generalized linear models and Cox regression models for prediction using multiple imputation to account for missing values in the predictors. See Mertens et al. (2018) for the method and the vignette for examples.

MixMatrix v0.2.2: Provides sampling and density functions for matrix variate normal, t, and inverted t distributions using the EM algorithm. See Thompson et al. (2019) for background. There are vignettes on Discriminant Analysis, matrix-t-estimation, the Matrix Normal Distributions, and Mixture Models.

sdcSpatial v0.1.1: Provides functions to create privacy protected raster maps can be created from spatial point data. Protection methods include smoothing of dichotomous variables by de Jonge and de Wolf (2016), continuous variables by de Wolf and de Jonge (2018), suppressing revealing values and a generalization of the quad tree method by Suñé et al. (2017).

Time Series

distanta v1.0.1: Provides tools to assess the dissimilarity between multivariate time-series. It is based on the psi measure described by Birks and Gordon (1985), which computes dissimilarity between irregular time-series constrained by sample order. There is a Package Summary with examples.

samurais v0.1.0: Provides a variety of statistical latent variable models and unsupervised learning algorithms to segment and represent both univariate and multivariate time-series data. There are vignettes for (1) Hidden Markov Model Regression (HMMR), (2) Model Selection for HMMR, (3) Multivariate HMMR, (4) Model Selection for MHMMR, (5) Regression with Hidden Logistic Process (RHLP), (6) [Model selection for RHLP, (7) Multivariate RHLP, (8) Model Selection for MRHLP, and (9) Piecewise Regression.

simts v0.1.1: Implements a system of tools to support time series analysis courses, including a technique called Generalized Method of Wavelet Moments (GMWM). See Guerrier et al. (2013) for the background and the vignette for examples.

Topological Data Analysis

BallMapper v0.2.0: Provides functions to compute a topologically accurate summary of data in the form of an abstract graph, using the algorithm described in Dlotko, (2019). Have a look at this video for an overview.

kernelTDA v0.1.1: Provides tools for exploiting topological information in standard statistical learning algorithms, implements kernels defined on the space of persistence diagrams, and provides a solver for kernel Support Vector Machines based on the C++ LIBSVM and functions to compute Wasserstein distance using the C++ HERA library. The vignette offers examples.

Utilities

babelwhale v1.0.0: Provides a unified interface to interact with docker and singularity containers, allowing users to execute a command inside a container, mount a volume, or copy a file.

fastmap v1.0.0: Provides a fast implementation of a key-value store that avoids common memory leakage issues by using data structures in C++. See the README for an overview.

modelsummary v0.1.0; Leverages the gt and broom packages to create customizable, publication-ready summary tables for statistical models. See the GitHup Repo for an overview of how to use the package.

readwritesqlite v0.0.2: Provides functions to read and write data frames to SQLite databases while preserving time zones (for POSIXct columns), projections (for sfc columns), units (for units columns), levels (for factors and ordered factors), and classes for logical, Date, and hms columns. The vignette provides usage information.

rolldown v0.1: Provides R Markdown output formats based on JavaScript libraries such as Scrollama for storytelling. There are vignettes on Basic Style and Sidebar Layout for Scrollama documents.

rray v0.1.0: Provides a toolkit for manipulating arrays in a consistent, powerful, and intuitive manner through the use of broadcasting and a new array class, rray. There are vignettes on Broadcasting, The rray, and Toolkit.

wyz.code.offensiveProgramming v1.1.8: Provides code to ease the transition from defensive to offensive programming as described in the Offensive Programming Book.

Visualization

altair v3.1.1: Implements an interface to Altair, which itself is a Python interface to Vega-Lite.

animint2 v2019.7.3: Provides functions for defining animated, interactive data visualizations in R code, and rendering on a web page. See the 2018 Journal of Computational and Graphical Statistics paper for a description of the underlying concepts, and the Github page for examples.

apexcharter v0.1.2: Provides an htmlwidgets interface to apexcharts.js, a modern JavaScript charting library to build interactive charts and visualizations with simple API. There is a Getting Started Guide and vignettes on labs and lines.

ggparty v1.0.0: Extends ggplot2 functionality to the partykit package, which provides the tools to create structured and highly customizable visualizations for tree-objects of the class party. There is a vignette on ggparty and another on edges.

metadynminer3d v0.0.1: Provides tools to read, analyze, and visualize Metadynamics 3D HILLS files from Plumed. See Tribello et al. (2014) for the background and look here for examples.

June 2019 "Top 40" R Packages

Wed, 24 Jul 2019 00:00:00 +0000

Approximately 136 new packages stuck to CRAN in June. (This number is difficult to nail down with certainty because packages may be removed from CRAN after sitting there for a few days.) Here are my picks for the June “Top 40” in ten categories: Computational Methods, Data, Finance, Genomics, Machine Learning, Science and Medicine, Statistics, Time Series, Utilities, and Visualization.

Computational Methods

cppRouting v1.1: Provides functions to calculate distances, shortest paths and isochrones on weighted graphs using several variants of Dijkstra algorithm. Algorithms include unidirectional Dijkstra Dijkstra (1959), bidirectional Dijkstra Goldberg et al. (2005), A* search Hart et al. (1968), and new bidirectional A* Pijls & Post (2009). See the vignette and website for how to use the package.

GuessCompx v1.0.3: Provides functions to test multiple increasing random samples of a data set, and tries to fit various complexity functions o(n), o(n2), o(log(n)), etc. to make an empirical guess about the time and memory complexities of an algorithm or a function. See the vignette for details.

SimJoint v0.2.1: Provides functions to simulate multivariate correlated data given non-parametric marginal distributions and their covariance structure, characterized by a correlation matrix from a purely computational perspective. See Iman and Conover (1982), Ruscio- and Kaczetow (2008), and the vignette for details.

Data

pinochet v0.1.0: Contains data about the victims of the Pinochet regime as compiled by the Chilean National Commission for Truth and Reconciliation Report (1991, ISBN:9780268016463). See the vignette.

usdarnass v:0.1.0: Offers an alternative for downloading various United States Department of Agriculture (USDA) data through R. There is a Getting Started Guide and a vignette on usdarnass output.

Finance

ceRtainty v1.0.0: Provides functions to compute the certainty equivalents and premium risks as tools for risk-efficiency analysis. For more technical information, see Hardaker et al. (2004), and Richardson and Outlaw (2008). The vignette contains examples.

portfolioBacktest v0.1.1: Implements automated backtesting of multiple portfolios over multiple data sets of stock prices in a rolling-window fashion. Intended for researchers, practitioners, and Finance instructors. See the vignette to get started.

Genomics

jackalope v0.1.2: Provides functions to simulate variants from reference genomes and read from both Illumina and Pacific Biosciences (PacBio) platforms. Simulating Illumina sequencing is based on ART by Huang et al. (2012). PacBio sequencing simulation is based on SimLoRD by Stöcker et al. (2016). There is an Introduction and a vignette on Models of nucleotide substitution.

Patterns v1.0: Implements tools for deciphering biological networks with patterned heterogeneous measurements that enables joint modeling of genes and proteins. There is an Introduction and a vignette on Network Inference.

subgxe v0.9.0: Implements functions to combine multiple GWAS using gene environment interactions and p-value assisted subset testing (PASTA), as described in Yu et al. (2019). Get started with the vignette.

Marketing

mmetrics v0.2.0: Provides a mechanism for easy computation of marketing metrics. Default metrics include Click Through Rate, Conversion Rate, and Cost Per Click, but you can define your own metrics easily. See the vignette.

promotionImpact v0.1.2: Provides functions to analyze and measure the promotion effectiveness on a given target variable (e.g., daily sales). Effects of these variables controlled for trend/periodicity/structural change using prophet Taylor and Letham (2017). See the GitHub page for information.

uplifteval Provides a variety of plots and metrics to evaluate uplift models including the R uplift package’s Qini metric and Qini plot. For background see Radcliffe (2007). There are vignettes on the plot_uplift_guelman(), plot_uplift(), and pl_plot() functions.

Machine Learning

archetypal v1.0.0: Provides functions to perform archetypal analysis by using a convex hull approximation. See Morup and Hansen (2012), Hochbaum and Shmoys (1985), Eddy (1977), Barber et al. (1996), and Christopoulos (2016) for background information, and the vignette for an introduction.

googleCloudVisionR v0.1.0: Provides access to the Google Cloud Vision API in R. It is part of the cloudyr project.

modelDown v1.0.1: Implements a website generator with HTML summaries for predictive models. This package uses DALEX explainers to describe global model behavior. See the GitHub page for getting started information and examples.

Science and Medicine

iCARH v2.0.0: Implements the integrative conditional autoregressive horseshoe model discussed in Jendoubi and Ebbels (2018). The vignette provides an example.

justifier v0.1.0: Implements a YAML-based standard for documenting justifications, such as for decisions taken during the planning, execution, and analysis of a study or during the development of a behavior change intervention. See Marques and Peters (2019) for background. There is an Introduction and vignettes on using justifier in behavior change intervention and in study design.

replicateBE v1.0.9: Implements comparative bioavailability-calculations for the EMA’s Average Bioequivalence with Expanding Limits (ABEL), and includes Method A and Method B detection of outliers. See the vignette.

StratefiedMedicine v0.1.0: Provides analytic and visualization tools to aid in stratified and personalized medicine. Stratified medicine aims to find subgroups of patients with similar treatment effects, while personalized medicine aims to understand treatment effects at the individual level. There is an Introduction.

Statistics

cpsurvsim v1.1.0: Provides functions to simulate time-to-event data with type I right censoring using two methods: the inverse CDF method and a proposed memoryless method. See Rainer Walke (2010) for background and the vignette for the math.

durmod v1.1: Provides functions to estimate piecewise constant mixed proportional hazard competing risk models as described in Gaure et al. (2007), Heckman and Singer (1984), and Lindsay (1983). The vignette provides examples.

kernelPSI v1.0.0: Implements post-selection inference strategies for kernel selection, as described in Slim et al. (2019). See the vignette to get started.

missSBM v0.2.0: Provides methods for handling missing data in stochastic block models. See Tabouy et al. (2019) for background and the vignette for a case study.

RandomCoefficients v0.0.2: Implement adaptive estimation of the joint density linear model where the coefficients, intercept, and slopes are random and independent from regressors. See Gaillac and Gautier (2019) for background information and the vignette for the theory and examples.

spatialfusion v0.6: Provides functions for multivariate modelling of geostatistical (point), lattice (areal), and point pattern data in a unifying spatial fusion framework. Details are given in Wang and Furrer (2019). Model inference is done using either Stan or INLA. See the vignette for examples.

ui v0.1.0: Implements functions to derive uncertainty intervals for (i) regression (linear and probit) parameters when outcomes are not missing at random (see Genbaeck et al. (2015) and Genbaeck et al. (2018)), and (ii) double robust and outcome regression estimators of average causal effects with possibly unobserved confounding (see Genbaeck and de Luna (2018)).

varclust v0.9.4: Provides functions to cluster quantitative variables, assuming that clusters lie in low-dimensional subspaces. Segmentation of variables, number of clusters, and their dimensions are selected based on BIC. There is a brief tutorial.

Time Series

bvartools v0.0.1: Implements some common functions used for Bayesian inference for mulitvariate time series models. There is an Introduction and vignettes on Bayesian Structural Vector Autoregression, Bayesian Error Correlation, and Stochastic Search Variable Selection.

wwwntests v1.0.0: Provides an array of white noise hypothesis tests for functional data and related visualizations. Methods are described in Kokoszka et al. (2017), Characiejus and Rice (2019), and Gabrys and Kokoszka (2007). See the vignette for details.

Utilities

gargle v0.3.0: Provides utilities for working with Google APIs, including functions and classes for handling common credential types and for preparing, executing, and processing HTTP requests. There are vignettes on authentication, API Credentials, tokens, and helper functions.

git2rdata v0.1: Provides functions to store and retrieve data frames in a Git repository, making versioning of data.frames easy and efficient using git repositories. There is a Getting Started Guide and vignettes on Efficiency, Optimizing Storage, and a Suggested Workflow.

metapost v1.0-6: Provides an interface to the MetaPost programming language. There are functions to generate an R description of a MetaPost curve, functions to generate MetaPost code from an R description, functions to process MetaPost code, and functions to read solved MetaPost paths back into R. Look here for different approaches for communicating between R and MetaPost.

rless Provides CSS preprocessor features using the LESS language, a CSS extension giving options to use variables, functions, or using operators while creating styles. See the vignette for examples.

rock v0.0.1: Implements the Reproducible Open Coding Kit, which was developed to facilitate reproducible and open coding, specifically geared towards qualitative research methods. See the Introduction for details.

tidyrules v0.1.0: Provides functions to convert a text-based summary of rule-based models to a tidy data frame (where each row represents a rule), with related metrics such as support, confidence, and lift. See the vignette for examples.

tsibbledata v0.1.0: Provides diverse data sets in the tsibble data structure, which are useful for learning and demonstrating how tidy temporal data can tidied, visualized, and forecasted. See README for an example.

websocket v1.0.0: Implements a WebSocket client interface for R. WebSocket is a protocol for low-overhead real-time communication. There is an Overview vignette.

Visualization

basetheme v0.1.1: Provides functions to create and select graphical themes for the base plotting system. See README for the themes.

condvis2 v0.1.0: Extends the condvis package and Shiny app with interactive displays for conditional visualization of models, data, and density functions. See O’Connell et al. (2017) for the background. There is an Introduction and vignettes on Exploring keras models and Exploring model-based clustering.

nomnoml v0.1.0: Implements a tool for drawing UML diagrams based on a simple syntax. See the README for examples, and look here for a demo.

ormPlot v0.3.2: Extends the rms Regression Modeling Strategies package that facilitates plotting ordinal regression model predictions together with confidence intervals for each dependent variable level. The vignette provides examples.

sugarbag v0.1.0: Provides functions to create a hexagon tilegram from spatial polygons. Developed to aid visualization and analysis of spatial distributions across Australia, which can be challenging due to the concentration of the population on the coast and the wide open interior of the country. There is a vignette pointing to ABS Data and another developing an example for Tasmania.

May 2019: "Top 40" New CRAN Packages

Tue, 25 Jun 2019 00:00:00 +0000

Two hundred twenty-two new packages made it to CRAN in May, and it was more of an effort than usual to select the “Top 40”. Nevertheless, here they are in nine categories, Computational Methods, Data, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities and Visualization.

Computational Methods

dde v1.0.0: Implements a Dormand-Prince algorithm for solving “non-stiff” differential equations. See the vignette for examples.

rTRNG v4.20-1: Implements an interface to Tina’s Random Number Generator C++ Library, and provides examples of how to use parallel RNG with RcppParallel. See Bauke (2018) for implementation details and the vignette for background, and useR!2017 Presentation for examples.

Data

ramlegacy v0.2.0: Provides functions to download, cache and read an Excel version of the RAM Legacy Stock Assessment Data Base, an online compilation of stock assessment results for commercially exploited marine populations from around the world. See the vignette for an introduction.

rnassqs v0.4.0: Implements an interface to the United States Department of Agricultre’s National Agricultural Statistical Service (NASS) Quick Stats API. There is a vignette showing how to use the package.

Machine Learning

EIX v1.0 Provides functions to examine the structure and explain interactions in XGBoost and LightGBM models including functions to visualize tree-based ensembles models, identify interactions and measure variable importance. EIX is a part of the DrWhy.AI universe. There is a vignette on Explaining Interactions and another analyzing the Titanic Data.

OTclust v1.0.2: Implements a mean partition for ensemble clustering by optimal transport alignment. The vignette

paws v0.1.1: An ensemble of packages including paws.compute, paws.storage, paws.database, paws.networking, paws.management, paws.machine.learning, paws.analytics, paws.security.identity, paws.application.integration, paws.cost.management, and paws.customer.engagement that provides a comprehensive interface to Amazon Web Services.

peRspective v0.1.0: Implements an interface to the Perspective API which uses machine learning models to score the perceived impact a comment might have on a conversation (i.e. TOXICITY, INFLAMMATORY, etc.). See README for an example.

SelectBoost v1.4.0: Implements the SelectBosot algorithm to enhance the performance of variable selection methods in correlated data sets. There are vignettes on Benchmarking, Confidence Estimates, Simulation Tools that are provided with the package.

spectralGraphTopology v0.1.1: Implements block coordinate descent estimators to learn k-component, bipartite, and k-component bipartite graphs from data by imposing spectral constraints on the eigenvalues and eigenvectors of the Laplacian and adjacency matrices. This package is based on the paper by Kumar et al. (2019). There is a vignette.

Mathematics

gridBezier v1.1-1: Provides functions for rendering both quadratic and cubic Bezier curves in grid. Look here for a tutorial on variable-width Bezier Splines in R.

wedge v1.0-1: Provides functions for working with differentials, k-forms, wedge products, Stokes’s theorem, and related concepts from the exterior calculus.

Medicine

adept v1.1.2: Provides functions for analyzing high-density data from walking strides collected from a wearable accelerometer worn during continuous walking activity. There is an Introduction and a vignette on walking stride segmentation.

basket v0.9.2: Implementation of multisource exchangeability models for Bayesian analyses of prespecified subgroups arising in the context of basket trial design and monitoring. See Hyman et al. (20185, Hobbs & Landin (2018), Hobbs et al. (2018) and Kaizer et al. (2017) for background, and the vignette for how to use the package.

BayesianFROC v0.1.3: Implements methods for Free-response Receiver Operating Characteristic (FROC) analysis to compare performance metrics such as area under the curve (AUC) for the purpose of finding lesions in radiographs of different modalities: Resonance Imaging (MRI), Computed Tomography (CT), Positron Emission Tomography (PET), …, etc. See Chakraborty (1981) for background and the vignettes Theory of Bayesian FROC and Single Reader and Single Modality for how to use the package.

getsummary v0.1.0: Provides functions to create presentation-ready tables summarizing data sets, regression models, and more. Function defaults follow reporting guidelines outlined in Assel et al. (2019). There are tutorials on using the fmt-regression() and fmt_table1() functions.

Science

beastier v2.01.15: Implements an API to access the BEAST2 tool for Bayesian phylogenetic analysis. The vignette shows how to use the package.

gdalcubes v0.1.0: Provides functions to process collections of Earth observation images as on-demand multispectral, multitemporal data cubes. Users define cubes by spatiotemporal extent, resolution, and spatial reference system and let ‘gdalcubes’ automatically apply cropping, reprojection, and resampling using the Geospatial Data Abstraction Library (GDAL). There is a Getting Starting Guide.

Statistics

explore v0.4.3O Provides interactive functions to facilitate exploratory data analysis. See the vignette.

Monte.Carlo.se v0.1.0: Provides functions to compute Monte Carlo standard errors for summaries of Monte Carlo output. See Boos and Osborne (2015) for background. Additionally, there is an Overview and vignettes on Creating Tables, Summary Statistics, and Pairwise Comparisons.

multinomialeq v0.2.1: Implements Gibbs sampling and Bayes factors for multinomial models with linear inequality constraints on the vector of probability parameters. See Heck and Davis-Stober (2019) for background and the vignette for an example.

PLNmodels v0.9.2: Implements the Poisson-lognormal model which is applicable to a variety of multivariate problems when count data are at play, including principal component analysis for count data (Chiquet et al. (2018)](doi:10.¹²¹⁴⁄₁₈-AOAS1177), and network inference Chiquet et al. (2018). There are vignettes for importing data, analyzing count data, classification, sparse structure estimation, PCA, and a description of the Trichoptera data set.

pvaluefunctions v1.2.0: Provides a function to compute and plot confidence distributions, confidence densities, p-value functions and s-value (surprisal) functions for several commonly used estimates. See Poole (1987), Schweder and Hjort (202) and Singh et al. (2007) for background, and the vignette for details of the package.

SGB v1.0: Implements multivariate regression using a generalization of the Dirichlet distribution, the Simplicial Generalized Beta distribution over the simplex space of compositions or positive vectors with sum of components equal to 1. See the vignette for details.

suddengains v0.2.1: Provides functions to identify sudden gains in longitudinal data based on the criteria outlined in Tang and DeRubeis (1999). There is a Tutorial.

Utilities

distill v0.7: Enables R Markdown formatting for web-based scientific and technical articles. Look here for documentation.

packageRank v0.1.0: Allows users to compute and visualize the cross-sectional and longitudinal number and rank percentile of package downloads from RStudio’s CRAN mirror. The documentation is on GitHub.

table.express v0.1.1: Offers versions of dplyr data manipulation verbs that parse and build expressions which are ultimately evaluated by data.table. See the vignette.

test2speech v0.2.4: Unifies different text to speech engines from Google, Microsoft, and Amazon by enabling users to switch between different services by setting a function argument. There is a vignette Listing out voices.

tidycode v0.1.0: Provides functions to analyze lines of R code using tidy principles allowing users to input lines of R code and output a data frame with one row per function. See the vignette for examples.

matahari v0.1.0: Allows users to spy on their R sessions by logging everything they type into the R console. See the vignette.

vroom v1.0.1: Uses lazy evaluation to quickly read data from csv, tsv and fwf files. There is a Getting Started Guide and a vignette with Benchmarks.

Visualization

ggPMX v0.9.4: Implements a toolbox of diagnostic functions and plots for non-linear mixed effects models that produces publication ready plots. See the User Guide for details.

ggResidpanel v0.3.0: Provides functions to create interactive diagnostic plots and panels of diagnostic plots for residuals. See the vignette.

mapcan v0.0.1: Provides functions to standard choropleth maps as well as choropleth alternatives for ggplot2 that address the peculiarities of plotting statistics for Canadian data at the riding level. There is an Introduction and a vignette on the riding_binplot() function.

oceanis v0.8.5: Provides functions to create interactive maps for statistical analysis that may include proportional circles, chroropleths, typology and flows. The vignette is in French.

parcoords v1.0.1: Implements an htmlwidget for the d3.js parallel coordinates function parcoords-es. See the vignette

plotdap v0.0.2: Provides functions to visualize and animate data from ERDDAP servers via the rerddap package using base graphics or ggplot2. There is a vignette.

prettyB v0.2.1: Provides drop-in replacements for some standard, base R graphics functions.

radarBoxplot v`.0.0: Creates radar-boxplots for visualizing multivariate data. See README for documentation.

Equal Size kmeans

Thu, 13 Jun 2019 00:00:00 +0000

We were recently presented with a problem where the decision maker wanted to understand how their data would naturally group together. The classic technique of k-means clustering was a natural choice; it’s well known, computationally efficient, and implemented in base R via the kmeans() function.

Our problem has a slight wrinkle: the decision maker wished to see the data grouped with (nearly) equal sizes. Now, a ‘true’ statistician would tell the client that the right thing to do from a theoretical perspective was to use native k-means results because some centers can simply have more nearby points than other centers. However, we are practitioners, and if the visualization provides additional information useful to the way people make decisions, we are not going to tell them they are wrong!

Approach

This is very similar to a mathematical optimization problem commonly faced by organizations like fire and police departments; specifically, ‘where trucks/patrol cars should be stationed to minimize response time’.

The general strategy is to decompose the hard problem into two easier sub-problems, to wit:

If we knew where the centroids were, determining group membership would be easy.
If we knew group membership, determining centroids would be trivial.

The key insight (and it really is all downhill from here) is to simply pretend that we have the solution to issue 1, and iterate between these two tasks until convergence is reached, that is - make a guess at where the centroids are, pick group members, then adjust the centroid based on group membership. This has the same ‘feel’ as mathematical induction, and we’ll name the steps accordingly.

Our example is based on mtcars a built-in R dataset, with three clusters of equal size.

First, some libraries:

library(magrittr); library(dplyr); library(ggplot2)

Basis Step

We have to start somewhere, and in this example, we will use an initial solution coming from the basic kmeans algorithm. Another approach would be to pick initial centroids at the ‘corners’ of the space, or to simply pick a few random data points as centroids:

data(mtcars)
k = 3
kdat = mtcars %>% select(c(mpg, wt))
kdat %>% kmeans(k) -> kclust

So far, so good. Now we’ll compute the distance matrix between each point and each centroid; this begins the

Assignment step

kdist = function(x1, y1, x2, y2){
  sqrt((x1-x2)^2 + (y1-y2)^2)
}
centers = kclust$centers

kdat %<>% 
  mutate(D1 = kdist(mpg, wt, centers[1,1], centers[1,2]))
kdat %<>% 
  mutate(D2 = kdist(mpg, wt, centers[2,1], centers[2,2]))
kdat %<>% 
  mutate(D3 = kdist(mpg, wt, centers[3,1], centers[3,2]))

From here, we assign clusters, which we do greedily, using a technique we like to call ‘little kids soccer’ - because this is the way kids generally pick teams - by going in order and picking the ‘best’ option available to them at the time. The algorithm interrogates each cluster in turn and picks the ‘closest’ unassigned member until each cluster is filled. There’s one minor wrinkle that needed to be worked out: the final round consists of the ones that are the ‘worst fits’ across all k clusters; in this case, the points choose the clusters.

kdat$assigned = 0
kdat$index = 1:nrow(kdat)
working = kdat
FirstRound = nrow(kdat) - (nrow(kdat) %% k)

for(i in 1:FirstRound){ 
  #cluster counts can be off by 1 due to uneven multiples of k. 
  j = if(i %% k == 0) k else (i %% k)
  itemloc = 
    working$index[which(working[,(paste0("D", j))] ==
    min(working[,(paste0("D",j))]))[1]]
  kdat$assigned[kdat$index == itemloc] = j
  working %<>% filter(!index == itemloc)
##The sorting hat says... GRYFFINDOR!!! 
}
for(i in 1:nrow(working)){
  #these leftover points get assigned to whoever's closest, without regard to k
  kdat$assigned[kdat$index ==
                  working$index[i]] = 
    which(working[i,3:5] == min(working[i, 3:5])) 
}

Next, we recalculate the centroids. It’s kind of smooth to simply use k-means with k = 1.

NewCenters <- kdat %>% filter(assigned == 1) %>% 
                        select(mpg, wt) %>%
                        kmeans(1) %$% centers

NewCenters %<>% rbind(kdat %>% 
                        filter(assigned == 2) %>%
                        select(mpg, wt) %>%
                        kmeans(1) %$% centers)

NewCenters %<>% rbind(kdat %>%
                        filter(assigned == 3) %>%
                        select(mpg, wt) %>%
                        kmeans(1) %$% centers)

NewCenters %<>% as.data.frame()

The result a single round is presented here:

kdat$assigned %<>% as.factor()
kdat %>% ggplot(aes(x = mpg, y = wt, color = assigned)) +
  theme_minimal() + geom_point() + 
  geom_point(data = NewCenters, aes(x = mpg, y = wt),
             color = "black", size = 4) + 
  geom_point(data = as.data.frame(centers), 
             aes(x = mpg, y = wt), color = "grey", size = 4)

Figure 1: Iterated k-means with one step

You will notice there is a single point assigned to Group 1 that is on the ‘frontier’ between Groups 2 and 3. This point appears to be misclassified, and the way to resolve this is to iterate the algorithm (see below).

You can see how coercing the size made the cluster centroids migrate - significantly in the case of the higher mpg cluster. Grey dots are the original centroid, black are the updated (equal size) centroid.

Functionalized and iterated

It is straightforward to ‘wrap’ the above code into a function (truncated here for brevity), which we call kMeanAdj. It and takes the incumbent centers, data, number of iterations, and $k$ as arguments. We may plot the result as follows:

x = kMeanAdj(NewCenters, kdat, iter = 3, k) 

x$Data$assigned %<>% as.factor()

x$Data %>% ggplot(aes(x = mpg, y = wt, color = assigned)) +
  theme_minimal() +  geom_point() +
  geom_point(data = x$centers, aes(x = mpg, y=wt),
             color = "black", size = 4)

Figure 2: Equal Size Clusters with 3 iterations

Iterating the algorithm over several steps ‘stabilizes’ both the groups and centers, yielding the desired characteristics.

Summary

Programming projects like this can sometimes feel like traveling by hot air balloon, in the sense that you don’t know which way you will be headed until you begin to travel. In this case, we did not initially anticipate the poor performance of our initial method in the case where k does not divide n. The only way to discover issues like this, of course, is to frequently prototype and test code. Overcoming this challenge added both to the fun and the reward of this exercise. Additionally, it showcases how the (robust) existing routines in the R language and popular packages may be rapidly combined with new ideas. This flexibility is what makes R a natural choice for both practitioners and theorists in Statistics and Operations Research.

Harrison Schramm, CAP, PStat, is a Senior Fellow at the Center for Strategic and Budgetary Assessments.

Carol DeZwarte, CAP, PMP, whose passion for advanced analytics predates it becoming a buzzword, is in Supply Chain Analytics at Wayfair.

reticulate, virtualenv, and Python in Linux

Mon, 10 Jun 2019 00:00:00 +0000

Roland Stevenson is a data scientist and consultant who may be reached on Linkedin.

reticulate is an R package that allows us to use Python modules from within RStudio. I recently found this functionality useful while trying to compare the results of different uplift models. Though I did have R’s uplift package producing Qini charts and metrics, I also wanted to see how things looked with Wayfair’s promising pylift package. Since pylift is only available in python, reticulate made it easy for me to quickly use pylift from within RStudio.

In the article below, I’ll show how I worked through the following circumstances:

Since pylift has only been tested on Python >= 3.6, and my system version of Python was 2.7, I needed to build and install Python 3.6 for myself, preferably within a self-contained virtual environment.
I wanted to install pylift in the virtual environment and set up reticulate in my R Project to work within that environment.
Finally, I needed to access pylift from an R Markdown document via the reticulate interface.

Setting up Python, virtualenv, and RStudio

Note: for consistency, I always use an instance created via r-studio-instance and a base project from r-studio-project.

Python 2.7 is the default on the systems I use (CentOS 6/7). Since I did not want to modify the system-level Python version, I installed Python 3.6.x at the user level in $HOME/opt and created a virtual environment using Python 3. I then activated the Python 3 environment and installed pylift. Finally, I ensured RStudio-Server 1.2 was installed, as it has advanced reticulate support like plotting graphs in line in R Markdown documents.

Below is a brief script that accomplishes the tasks in bash on CentOS 7:

cd ~
mkdir tmp
cd tmp
wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz
tar -xzvf Python-3.6.2.tgz
cd Python-3.6.2
./configure --prefix=$HOME/opt/python-3.6.2 --enable-shared
make
make install
cd ~
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/opt/python-3.6.2/lib
virtualenv -p $HOME/opt/python-3.6.2/bin/python3 pylift
source pylift/bin/activate
cd pylift
git clone https://github.com/wayfair/pylift
cd pylift
pip install .
pip install -r requirements.txt
cd
wget https://s3.amazonaws.com/rstudio-ide-build/server/centos6/x86_64/rstudio-server-rhel-1.2.1335-x86_64.rpm
sudo yum install -y --nogpgcheck rstudio-server-rhel-1.2.1335-x86_64.rpm
sudo rstudio-server start

Some notes:

the --enable-shared option is required when building Python in order for reticulate to work
the LD_LIBRARY_PATH library also needs to be set prior to creating the virtual environment
we use virtualenv to create a virtual environment called “pylift” and then ensure that all Python packages are installed to that environment only (so as not to pollute any other environments we are working with)
we then clone the pylift source and install pylift along with all of its requirements via pip install -r requirements.txt
finally, we install the RStudio Server 1.2 Preview version in order to leverage its advanced reticulate features

Using Python from within RStudio via reticulate

Switching from bash to RStudio, we load reticulate and set it up to use the virtual environment we just created. Finally, and specific to pylift, we set matplotlib parameters so that we can plot directly in R.

library(reticulate)

Sys.setenv(LD_LIBRARY_PATH = paste0(Sys.getenv("HOME"),"/opt/python-3.6.2/lib"))
Sys.getenv("LD_LIBRARY_PATH")
use_virtualenv("/home/rstevenson/pylift", required=TRUE)
py_config()

# Currently this must be run in order for R-markdown plotting to work
matplotlib <- import("matplotlib")
matplotlib$use("Agg", force = TRUE)

Test that it works

The following replicates the first part of pylift tutorial: simulated data

import matplotlib.pyplot as plt
import numpy as np
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t,s)

When run, the above code chunk should display a sinusoidal graph below it.

import numpy as np, matplotlib as mpl, matplotlib.pyplot as plt, pandas as pd
from pylift import TransformedOutcome
from pylift.generate_data import dgp
# Generate some data.
df = dgp(N=10000, discrete_outcome=True)

# Specify your dataframe, treatment column, and outcome column.
up = TransformedOutcome(df, col_treatment='Treatment', col_outcome='Outcome', stratify=df['Treatment'])

# This function randomly shuffles your training data set and calculates net information value.
up.NIV()

The above Python chunk uses reticulate from within RStudio to interact with pylift in the context of a custom virtual environment, using a custom version of Python. This degree of customization and functionality should be useful to users who:

want to use a different Python version than they typically use while not affecting their typical setup by way of a virtual environment
want to install a Python module like pylift within a virtual environment so as not to affect any of their user- or system-level Python module installations
want to use reticulate from RStudio to access a custom virtual environment, Python version, and Python modules
wants to be able to delete the virtual environment and R-Project and have everything go back to the way it was
wants to be able to reproduce or share the environment exactly so that the workflow can be shared with others

April 2019: "Top 40" New CRAN Packages

Thu, 30 May 2019 00:00:00 +0000

One hundred eighty-seven new packages made it to CRAN in April. Here are my picks for the “Top 40”, organized into ten categories: Biotechnology, Data, Econometrics, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities, and Visualization.

Biotechnology

genpwr v1.00: Provides functions for power and sample size calculations for genetic association studies allowing for mis-specification of the model of genetic susceptibility. The methods employed are extensions of Gauderman (2002) and Gauderman (2002). See the vignette for details.

rabhit v0.1.1: Implements an adaptive Bayesian framework to infer V-D-J haplotypes and gene deletions from AIRR-seq data. See Gidoni et al. (2019) for background and the vignette for an introduction to the package.

Data

compstatr v0.1.1: Provides a set of tools for creating yearly data sets of St. Louis Metropolitan Police Department (SLMPD) crime data, which are available from January 2008 onward as monthly CSV releases. See the vignette.

DataSpaceR v0.6.3: Provides a convenient API interface to access immunological data within the CAVD DataSpace, a data sharing and discovery tool that facilitates exploration of HIV immunological data from pre-clinical and clinical HIV vaccine studies. There is an Introduction.

ebirdst v0.1.0: Provides tools to download, map, plot and analyze eBird, a global database of bird observations collected by citizen scientists, Status and Trends data. There is an Introduction and vignettes on Generating maps and stats, Data Structure, and Predictor Performance, Directionaly and Performance Metrics

PropublicaR v0.9.2: Provides wrapper functions to access the ProPublica’s Congress and Campaign Finance APIs.

tradestatistics v0.2: Provides access to the Open Trade Statistics API from R to download international trade data. There is an Introduction and a vignette on data sets.

ukpolice v0.1.2: Provides access to UK Police public data, including data on police forces and police force areas, crime reports, and the use of stop-and-search powers. See the vignette for details.

Econometrics

modelplotr v1.0.0: Provides plots to assess the quality of predictive models from a business perspective, which can show how implementing the model will impact business targets like response on a campaign or return on investment. See the vignette for examples.

SortedEffects v1.0.0: Implements the estimation and inference methods for sorted causal effects and classification analysis as described in Chernozhukov et al. (2018). See the vignette for an introduction and example.

Machine Learning

iBreakDown v0.9.6: Implements Break Down Tables and Plots, which are model-agnostic tools for the decomposition of predictions from black boxes. The methodology behind it is described in Gosiewska and Biecek (2019). There are vignettes for classification models, regression, and an Example.

localModel v0.3.11: Provides local explanations of machine learning models, and describes how features contributed to a single prediction. See Ribeiro & Singh (2016) for details. There is an Introduction and vignettes on Methodology and Explaining Classification Models.

polyreg v0.6.4: Automates the formation and evaluation of polynomial regression models, and provides support for cross-validating categorical variables. See Cheng et al., and look here for examples.

rfVarImpOOB v1.0: Estimates variable importance for random forests by computing impurity reduction importance scores for out-of-bag (OOB) data, complementing the existing in-bag Gini importance. See Strobl et al (2007), Strobl et al (2007), and Breiman (2001). The vignette contains a small example.

rsparse v0.3.3.1: Implements several algorithms for statistical learning on sparse matrices, including matrix factorizations, matrix completion, elastic net regressions, and factorization machines. Look here for an example.

Medicine

blockRAR v1.0: Provides functions to compute power for response-adaptive randomization clinical trial with a block design that captures both the time and treatment effect. See [Chandereng & Chappell (2019)](arXiv:1904.07758 for details and the vignette for Bayesian and Frequentist examples.

gestate v1.3.2: Provides tools to assist planning and monitoring of time-to-event trials under complicated censoring assumptions and/or non-proportional hazards. There are vignette on Predicting Events and Planning Trials.

Science

streamDepletr v0.1.0: Implements analytical models for estimating streamflow depletion due to groundwater pumping, and other related tools. See Zipper et al. (2018) for more information on depletion apportionment equations, and Zipper et al. (2019) for more information on analytical depletion functions. There is a vignette.

RobustSingleCell v0.1.1: Implements functions for the robust single cell clustering and comparison of population compositions across tissues and experimental models via similarity analysis. See Magen (2019) for details and the vignette for examples.

Statistics

BayesSenMC v0.1.1: Provides functions to generate different posterior distributions of adjusted odds ratio under different priors of sensitivity and specificity, and plots the models for comparison. See Chu et al. (2006) and Chu et al. (2010) for background.

bayestestR 0.1.0: Provides utilities to describe posterior distributions and Bayesian models, and the vignette for details and examples. There is a Getting Started Guide, and vignettes on Comparison of Point Estimates, Comparison of Indices, and Examples.

CoSMos v1.0.1: Implements a framework unifying, extending, and improving a general-purpose modelling strategy, based on the assumption that any process can emerge by transforming a specific ‘parent’ Gaussian process. See Papalexiou (2018) and the vignette.

fic v1.0.0: Provides functions to determine how well different models fitted by maximum likelihood estimate a quantity of interest, including generalized linear models and parametric survival models. See Claeskens and Hjort (2003) for details. There is an Introduction along with vignettes for Linear regression, loss functions, multi-state models, skew normal models, and survival models.

foieGras v0.2.1: Provides functions to fit continuous-time state-space models for filtering Argos satellite (and other) telemetry data. The vignette provides an overview.

glmaag v0.0.6: Implements efficient procedures for adaptive LASSO and network regularized for Gaussian, logistic, and Cox models. See Ucar, et. al (2007) and Meinshausen and Buhlmann (2006) for a discussion of network estimation procedures. The vignette provides an example.

Irescale v0.2.6: Provides a scaling method to obtain a standardized Moran’s I measure for the spatial autocorrelation of a data set, which gives a measure of similarity between data and its surrounding. See Chen (2009) for the method of calculation and the vignette for an example.

mvgraphnorm v1.81: Provides a function to compute a constrained covariance matrix for a given graph to generate samples from a Gaussian graphical model, using different algorithms for the analysis of complex network structure. See Kim et al. (2008) and Speed et al. (1986) for algorithms, and the vignette for an example.

ptsuite v1.0.0: Implements several methods for tail index estimation for power law distributions, including maximum likelihood Newman (2005), Hill’s estimator Hill 1975, least squares Zaher et al. (2014), method of moments Rytgaard (1990), and percentiles Bhatti et al. (2018). There is a vignette.

spatialreg v1.1-3: Provides a collection of functions for fitting functions for spatial cross-sectional models on lattice/areal data using spatial weights matrices. There is a vignette on Moran Eigenvectors and another on the North Carolina SIDS data set.

Time Series

CSTools v1.0.0: Implements process-based methods for assessing climate forecasts, including forecast calibration, bias correction, statistical and stochastic down-scaling, optimal forecast combination, and multivariate verification. See Doblas-Reyes et al. (2005), Mishra et al. (2018), Terzago et al. (2018), Torralba et al. (2017), and D’Onofrio et al. (2014) for details. There are vignettes on Multi Model Skill Assessment, Multivariate RMSE, and the RainFarm Model.

DChaos v0.1-1: Implements several algorithms for detecting chaotic signals inside univariate time series using methods derived from chaos theory, which estimate the complexity of a data set through exploring the structure of the attractor. See Ruelle and Takens (1971) for some deep background.

otsad v0.1.0: Implements a set of online fault detectors for time-series, including PEWMA Carter et al. (2012), SD-EWMA and TSSD-EWMA H. Raza et al. (2015), KNN-CAD Burnaev et al. (2016), KNN-LDCD Ishimtsev et al. (2017), and CAD-OSE M. Smirnov (2018). There is a vignette.

Utilities

inspectdf v0.0.2: Provides a collection of utilities for column-wise summary, comparison, and visualization of data frames. Functions report missingness, categorical levels, numeric distribution, correlation, column types, and memory usage. See README for examples.

suppdata v1.1-1: Provides functions for downloading data supplementary materials from manuscripts, using papers’ DOIs as references. Facilitates open, reproducible research workflows: scientists re-analyzing published data sets can work with them as easily as if they were stored on their own computer. There is a brief Introduction.

tidync v0.2.1: Provides tidy tools for working with NetCDF data sources. The vignette provides background and describes data extraction and exploration.

tinytest v0.9.3: Provides a lightweight (zero-dependency) and easy to use unit testing framework. Main feature: install tests with the package. The vignette shows how to use the package.

Visualization

frequentdirections v0.1.0: Implements frequent-directions algorithm for efficient matrix sketching. See Edo Liberty (2013) for details and the README for examples.

ggdemetra v0.1.0: Provides ggplot2 functions to return the results of seasonal and trading-day adjustment made by RJDemetra, the seasonal adjustment software officially recommended to the members of the European Statistical System and the European System of Central Banks. The vignette provides examples.

graphlayouts v0.1.0: Implements several new layout algorithms to visualize networks. Most are based on the concept of stress majorization by Gansner et al. (2004). The vignette shows several examples.

tidymv v2.1.0: Provides functions for visualizing generalized additive models and getting predicted values using tidy tools from the tidyverse packages. There is a vignette for Plotting Model Predictions and another for Plotting Smoothing Curves.

March 2019: "Top 40" New CRAN Packages

Fri, 26 Apr 2019 00:00:00 +0000

By my count, two hundred and thirty-three packages stuck to CRAN last month. I have tried to capture something of the diversity of the offerings by selecting packages in ten categories: Computational Methods, Data, Machine Learning, Medicine, Science, Shiny, Statistics, Time Series, Utilities, and Visualization. The Shiny category contains packages that expand on Shiny capabilities, not just packages that implement a Shiny application. It is not clear whether this is going to be a new cottage industry or not.

Computational Methods

DistributionOptimization v1.2.1: Fits Gaussian mixtures by applying Genetic algorithms from the GA package using Gaussian Mixture Logic stems from AdaptGauss.

latte v0.2.1: Implements connections to LattE for counting lattice points and integration inside convex polytopes, and 4ti2 for algebraic, geometric, and combinatorial problems on linear spaces and front-end tools facilitating their use in the ‘R’ ecosystem. Look here for an example.

nlrx v0.2.0: Provides tools to set up, run, and analyze NetLogo model simulations in R. There is a Getting Started Guide, vignettes for Advanced Configuration, and Examples.

nvctr v0.1.1: Implements the n-vector approach to calculating geographical positions using an ellipsoidal model of the Earth. This package is a translation of the FFi Matlab library from FFI described in Gade (2010). The vignette provides examples.

Data

EHRtemporalVariability v1.0: Provides functions to delineate reference changes over time in Electronic Health Records through the projection and visualization of dissimilarities among data temporal batches, and explore results through data temporal heat maps, information geometric temporal (IGT) plots, and a Shiny app. The vignette shows how to use the package.

kayadata v0.4.0: Provides data for Kaya identity variables (population, gross domestic product, primary energy consumption, and energy-related CO2 emissions), and includes utility functions for exploring and plotting fuel mix for a given country or region. See the vignette for examples.

newsanchor v0.1.0: Implements an interface to gather news from the News API. A personal API key is required. The vignette shows how to scrape New York Times online articles.

raustats v0.1.0: Provides functions for downloading Australian economic statistics from the Australian Bureau of Statistics and Reserve Bank of Australia websites. The vignette shows how to use the package.

Machine Learning

akmedoids v0.1.2: Advances a set of R-functions for longitudinal clustering of long-term trajectories, and determines the optimal solution based on the Caliński-Harabasz criterion ( Caliński and Harabasz (1974) ). The vignette works through an extended example.

shapper v0.1.0: Implements a wrapper for the Python shap library that provides SHapley Additive exPlanations (SHAP) for the variables that influence particular observations in machine learning models. There are vignettes for classification and regression.

sparkxgb v0.1.0: Implements a sparklyr extension that provides an interface for XGBoost on Apache Spark. See the README for a brief overview.

xgb2sql v0.1.2: Enables in-database scoring of XGBoost models built in R, by translating trained model objects into SQL query. See Chen & Guestrin (2016) for details on XGBoost, and the vignette for an overview of the package.

Medicine

ctrdata v0.18: Provides functions for querying, retrieving, and analyzing protocol- and results-related information on clinical trials from two public registers, the European Union Clinical Trials Register and ClinicalTrials.gov. There is a Getting Started Guide and a vignette with examples.

pubtatordb v0.1.3: Provides functions to download PubTator (National Center for Biotechnology Information) annotations, and then create and query a local version of the database. There is a vignette.

tacmagic v0.2.1: Provides functions to facilitate the analysis of positron emission tomography (PET) time activity curve (TAC) data. See Logan et al. (1996) and Aizenstein et al. (2008) for use cases, and the vignette for a detailed overview.

Science

bulletcp v1.0.0: Provides functions to automatically detect groove locations via a Bayesian changepoint detection method, to be used in the data pre-processing step of forensic bullet matching algorithms. See Stephens (1994) for reference, the vignette for the theory, and Mejia et al. in the most recent issue of Significance for the big picture.

earthtide v0.0.5: Ports the Fortran ETERNA 3.4 program by H.G. Wenzel for calculating synthetic Earth tides using the Hartmann and Wenzel (1994) or Kudryavtsev (2004) tidal catalogs. See the vignette for an introduction.

steps v0.2.1: Implements functions to simulate population dynamics across space and time. The Eastern Grey Kangeroo vignette offers an extended example.

Shiny

periscope v0.4.1: Implements an enterprise-targeted, scalable and UI-standardized shiny framework. There are vignettes for a downloadFile module, downloadablePlot module, downloadableTable module, and the creation of a framework-based application.

reactlog v1.0.0: Provides visual insight into that black box of shiny reactivity by constructing a directed dependency graph of the application’s reactive state at any point in a reactive recording. See the vignette for an introduction.

reactlog highlight filter from Carson Sievert on Vimeo.

shinyhttr v1.0.0: Modifies the progress() function from the httr package to let it send output to progressBar() function from the shinyWidgets package.

Statistics

CoopGame v0.2.1: Provides a comprehensive set of tools for cooperative game theory with transferable utility, enabling users to create special families of cooperative games, such as bankruptcy games, cost-sharing games, and weighted-voting games. The vignette offers theory and examples.

discfrail v0.1: Provides functions for fitting Cox proportional hazards models for grouped time-to-event data, where the shared group-specific frailties have a discrete non-parametric distribution. See Gasperoni et. al (2018). The vignette shows the math.

fastglm v0.1.1: Provides functions to fit generalized linear models efficiently using RcppEigen. The iteratively reweighted least squares implementation utilizes the step-halving approach of Marschner (2011). There is a vignette.

hettx v0.1.1: Implements methods developed by Ding, Feller, and Miratrix (2016), and Ding, Feller, and Miratrix (2018) for testing whether there is unexplained variation in treatment effects across observations, and for characterizing the extent of the explained and unexplained variation in treatment effects. There are vignettes on heterogeneous treatment effects and systematic fariation estimation.

mcmcabn v0.1: Implements a structural MCMC sampler for Directed Acyclic Graphs (DAGs). It supports the new edge reversal move from Grzegorczyk and Husmeier (2008) and the Markov blanket resampling from Su and Borsuk (2016), and three priors: a prior controlling for structure complexity from Koivisto and Sood (2004), an uninformative prior, and a user-defined prior. The vignette provides an overview of the package.

networkABC v0.5-3: Implements a new multi-level approximation Bayesian computation (ABC) algorithm to decipher network data and assess the strength of the inferred links between network’s actors. The vignette provides an example.

retrodesign v0.1.0: Provides tools for working with Type S (Sign) and Type M (Magnitude) errors, as proposed in Gelman and Tuerlinckx (2000) and Gelman & Carlin (2014), using the closed forms solutions for the probability of a Type S/M error from Lu, Qiu, and Deng (2018). The vignette shows how to use Type S and M errors in hypothesis testing.

senssobol v0.1.1: Enables users to compute, bootstrap, and plot up to third-order Sobol indices using the estimators by Saltelli et al. (2010) and Jansen (1999), and calculate the approximation error in the computation of Sobol first and total indices using the algorithm of Khorashadi Zadeh et al. (2017). The vignette provides an overview.

Time Series

DTSg v:0.1.2: Provides a class for working with time series data based on data.table and R6 with reference semantics. There are vignettes for Basic and Advanced usage.

RJDemetra v0.1.2: Implements an interface to JDemetra+, the seasonal adjustment software officially recommended to the members of the European Statistical System (ESS) and the European System of Central Banks.

runstats v1.0.1: Provides methods for quickly computing time series sample statistics, including: (1) mean, (2) standard deviation, and (3) variance over a fixed-length window of time-series, (4) correlation, (5) covariance, and (6) Euclidean distance (L2 norm) between short-time pattern and time-series. See the vignette for examples.

Utilities

aweek v0.2.0: Converts dates to arbitrary week definitions. The vignette provides examples.

credentials v1.1: Provides tools for managing SSH and git credentials. See the vignette for details.

cyphr v1.0.1: Implements wrappers using low-level support from sodium and OpenSSL to facilitate using encryption for data analysis. There is an Introduction and a vignette on Data Encryption.

encryptr v0.1.2: Provides functions to encrypt data frame or tibble columns using strong RSA encryption. See README for examples.

lenses v0.0.3: Provides tools for creating and using lenses to simplify data manipulation. Lenses are composable getter/setter pairs for working with data in a purely functional way, which were inspired by the Haskell library lens ( Kmett (2012) ). For a comprehensive history of lenses, see the lens wiki and look here for examples.

yum v0.0.1: Provides functions to facilitate extracting information in YAML fragments from one or multiple files, optionally structuring the information in a data.tree. See the README file.

Visualization

ggasym v0.1.1: Provides functions for asymmetric matrix plotting with ggplot2. See the vignette for examples.

predict3d v:0.1.0: Provides functions for 2- and 3-dimensional plots for multiple regression models using packages ggplot2 and rgl. It supports linear models (lm), generalized linear models (glm), and local polynomial regression fittings (loess). There is a vignette.

A Few Old Books

Thu, 25 Apr 2019 00:00:00 +0000

Greg Wilson is a data scientist and professional educator at RStudio.

My previous column looked at a few new books about R. In this one, I’d like to explore a few books about programming that people coming from data science backgrounds may not have stumbled upon.

The first is Michael Nygard’s Release It!, which more than lives up to its subtitle, “Design and Deploy Production-Ready Software”. Most of us can write programs that work for us on our machines; this book explores what it takes to create software that will work reliably for other people, on machines you’ve never met, long after you’ve moved on to your next project. It focuses on software that’s deployed for general use rather than installed on individuals’ machines, and covers stability patterns and anti-patterns, designing software to meet production needs, security, and a range of other pragmatic issues. You might not need to take care of these things yourself, but whoever has to get your software running on the departmental cluster will be grateful that you thought about it, and can have a sensible conversation about trade-offs.

The second book is Andreas Zeller’s Why Programs Fail, which bills itself as “a guide to pragmatic debugging”, and has been turned into a Udacity course. Programmers spend anywhere from a quarter to three quarters of their time debugging, but most only get an in-passing overview of how to do this well, and are never shown tools more advanced than print statements and break-point debuggers. Zeller starts with that, but goes much further to look at automatic and semi-automatic ways of simplifying programs to localize problems, isolating values’ origins, program slicing, anomaly detection, and much more. Some of the methods he describes will seem very familiar to data scientists, though the domain is new; others will take readers without a computer-science background into new territory in the same way that Advanced R does.

Our third entry is Michael Feathers’ Working Effectively with Legacy Code. Feathers defines legacy code as software that we’re reluctant to modify because we don’t understand how it works and are afraid of breaking. Having a comprehensive test suite allays this fear, but how can we construct one after the fact for a tangled mess of code? The bulk of the book explores answers to this question, including how to identify seams where code can be split, how to break dependencies so that parts can be improved incrementally, and so on. Some of the examples may seem a little out of date (the book is almost 15 years old), but they all apply directly to the unholy mixture of Perl, shell scripts, hundred-line SQL statements, and ten-page R scripts that you were just handed.

Number four is Jeff Johnson’s GUI Bloopers. I was in two startups in the 1990s, and in both of them, I was told after a few weeks that I was never allowed to work on the user interface again. It was the right decision, but this book might have made it unnecessary. Rather than trying to explain the rules for designing a good user interface, Johnson gives example after example of how to fix bad ones. The companion book, Web Bloopers, is less useful today because web interfaces have evolved so rapidly, but either will help you make an interface that is at least not bad.

The last entry for this post is Ashley Davis’s Data Wrangling with JavaScript. As its title suggests, it doesn’t spend very much time on statistical theory; instead, it covers the “other 90%” of squeezing answers out of data, from establishing your data pipeline and getting started with Node (a widely-used command-line version of JavaScript) to cleaning, analyzing, and visualizing data. There are lots of code samples and plenty of diagrams, and you can download both the data sets the author uses in examples and his Data-Forge library. I suspect readers will need some prior familiarity with JavaScript to dive into this, but Davis shows just how far you can go with what’s available today, and that the journey is a lot smoother than people might think.

On Meeting Data Journalists

Mon, 08 Apr 2019 00:00:00 +0000

“I’d rather do data than date”. I overheard this while eavesdropping on a conversation among three female data journalists while waiting for an elevator at the IRE-CAR (Investigative Reporters and Editors - Computer-Assisted Reporting) conference last month. I would like to think the remark was overloaded with hyperbole, but maybe not. Most of the attendees as this conference were motivated, tenacious, and highly skilled data hounds, the kind of investigative journalists who pry information from government databases through persistent requests, legal leverage, and SQL expertise.

This was my first CAR conference, and I was very impressed by the mission-driven enthusiasm with which the speakers, panelists, and attendees focused on data as an essential tool for the pursuit of the truth. I was impressed, but not surprised, to find this passion for data. Journalists have been sifting through data to find the truth since at least the early twentieth century when social work, academic social science, and journalism were all in the same primeval soup¹. The modern tradition of computer-assisted data journalism dates at least as far back as Philip Meyer’s coverage of the Detroit Riots, his 1973 book Precision Journalism, and subsequent collaboration with Donald Bartlet and James Steele examining patterns in 1970’s Philadelphia criminal conviction sentences². I mention all this to emphasize that data journalism is not just a trendy offshoot of data science. In fact, it might be the other way around! Data scientists probably owe as much to data journalists as they do to statisticians.

The number of packed workshops and talks on data wrangling and visualization far exceeded my expectations. I did expect some R content. (I saw the recent BBC post, and New York Times R-based visualizations are a daily part of my news consumption.) But, there were over 15 R-related sessions on the schedule, along with at least as many sessions devoted to Python, SQL, JavaScript, D3, and other programming tools. Moreover, the fact that several featured technical workshops were repeated over multiple days indicated that the conference organizers expected the data journalists to want to dig into the details of all these technologies.

You can get a flavor for the technology presentations by looking into the tip sheets that are available for many of these sessions. There is a wealth of information buried here, well worth a couple of hours of exploration. For example, see Peter Aldhous’ talk Text mining in R with tidytext,

Andrew Ba Tran’s Mapping with R, and the panel discussion How and why to make your data analysis reproducible, and Sharon Machlis’ video series: Do More with R.

A session on statistical inference that I very much enjoyed was Jevin West’s talk Calling bullshit: Data reasoning in a digital world. There is no tip sheet for the talk, but the website for the course that he teaches at the University of Washington with his colleague Carl Bergstrom contains voluminous material. Something like this course ought to be included in every statistics and data science syllabus. In addition to discussing the standard topics, such as attributing cause to correlation and deconstructing misleading visualizations, it also presents several up-to-date cautionary tales: for example, have a look at the case study Criminal Machine Learning.

Some additional tip sheets that I found illuminating for what they reveal about the types of data sources that data journalists seek out are: Tracking dark money tips, 2020 Census Reporting Mistakes, Tips on Finding Nonprofit Data, and Before you ever begin your analysis.

If you are a data journalist, or a data journalist in training, or really anyone new to R, and are looking for a fast on-ramp to becoming productive at data wrangling and creating visualizations, I highly recommend Sharon Machilis book Practical R for Mass Communication and Journalism and Andrew Ba Tran’s tutorial, R For Journalists. Both of these resources are unusual in that they provide up-to-date, tidyverse-based, GitHub-aware introductions to R, stressing data acquisition, manipulation, reporting, and graphing without the burden of having to simultaneously take an introductory course in statistics.

Finally, thanks to Andrew, here is a list of R-fluent journalists whom you may want to follow on Twitter:

¹C.W. Anderson. Apostoles of Certainty: Oxford University Press 2018. Chapter 2

² Data Journalism, Wikipedia

February 2019: “Top 40” New CRAN Packages

Tue, 26 Mar 2019 00:00:00 +0000

One hundred and fifty-one new packages arrived at CRAN in February. Here are my “Top 40” picks organized into eight categories: Bioinformatics, Data, Machine Learning, Medicine, Statistics, Time Series, Utilities and Visualization.

Bioinfomatics

Cascade v1.7: Implements a modeling tool allowing gene selection, reverse engineering, and prediction in cascade networks. See Jung et al. (2014) for details, along with a Package Introduction and a vignette on re-analysis.

Result of reverse engineering a TH1 network

countfitteR v1.0: Implements functions and a Shiny app for the automatized evaluation of distribution models for count data with an eye towards use in DNA analyses. The vignette provides an overview.

Data

noaaoceans v0.1.0: Provides tools to access the National Oceanic and Atmospheric Administration (NOAA) API. See the vignette for details.

guardianapi v0.1.0: Provides functions to access to The Guardian’s open API, containing all articles published in ‘The Guardian’ from 1999 to the present. The vignette shows how to use the package.

RobinHood v:1.0.1: Implements an interface to the RobinHood investing platform, including the ability to access account data, retrieve investment statistics and quotes, place and cancel orders, and more.

stlcsb v0.1.2: Provides functions working with data from The Citizens’ Service Bureau of the City of St. Louis including downloading data, categorizing problem requests, cleaning and subsetting CSB data, and projecting the data using the x and y coordinates. See the vignette.

Machine Learning

bigMap v2.1.0: Implements an unsupervised clustering protocol for large scale structured data, based on a low dimensional representation of the data. See Garriga and Bartumeus (2018) and the vignette for details.

fastNaiveBayes v1.0.1: Provides an extremely fast implementation of a Naive Bayes classifier that is largely based on the paper Schneider (2003). See the vignette for an introduction.

gama v1.0.3: Implements a genetic, evolutionary approach to performing hard partitional clustering. For details see Scrucca (2013), Charrad et al. (2014), and Tsagris and Papadakis (2018). The vignette shows how to use the package.

leiden v0.2.3: Uses reticulate to implement the Python leidenalg clustering algorithm for partitioning graphs in to communities in R. See the Python repository and Traag et al (2018) for details. There is also a vignette.

r.blip v1.1: Provides functions to learn Bayesian networks from datasets containing thousands of variables, and includes algorithms for (1) parent set identification (Scanagatta (2015)), (2) general structure optimization (Scanagatta (2018)), (3) bounded tree width structure optimization (Scanagatta (2016)), and (4) structure learning on incomplete data sets (Scanagatta (2018)).

RTML v0.9: Implements efficient solvers for 10 regularized multi-task learning algorithms applicable for regression, classification, joint feature selection, task clustering, low-rank learning, sparse learning and network incorporation. The details are described Cao and Schwarz (2018). There is a Tutorial.

Spectrum v0.4: Implements a fast, adaptive spectral clustering algorithm for single and multi-view data. The vignette provides an introduction.

SAR v1.0.0: Provides both a stand-alone and Azure Cloud implementation of the Smart Adaptive Recommendations (SAR) algorithm for personalized recommendations. Look here for a description of the SAR algorithm.

tfdeploy v0.6.0: Provides tools to deploy TensorFlow models across several services. There is a vignette of Deploying TensorFlow Models and another for using Saved Models.

tfio v0.4.0: Provides an interface to TensorFlow IO. There is a brief Introduction.

stabm v1.0.0: Implements several measures for the assessment of the stability of feature selection. See Bommert et al. (2017).

tidystopwords 0.9.0: Provides functions to generate stopword lists in 53 languages, in a way consistent across all the languages supported. There is a vignette.

Medicine

ClinReport v0.9.1.11: Provides functions to create formatted statistical tables in Microsoft Word documents that meet clinical standards. There is a vignette for Getting Started, a vignette for Modifying Outputs, and another for Graphic Outputs.

safetyGraphics v0.7.3: Implements a framework for evaluation of clinical trial safety through a Shiny application or standalone htmlwidget charts. See the User Guide.

Statistics

dosearch v1.0.2: Implements a method to identify causal effects from arbitrary observational and experimental probability distributions via do-calculus and standard probability manipulations, using a search-based algorithm that handles selection bias (Bareinboim and Tian (2015)), transportability (Bareinboim and Pearl (2014)), missing data (Mohan et al. (2013)), and arbitrary combinations of these. There is an informative Introduction

geosample v0.2.1: Provides functions for constructing sampling designs. For details, see Chipeta et al. (2016) and the vignette.

interactions v1.0.0: Provides a suite of functions for conducting and interpreting the analysis of statistical interaction in regression models, and includes visualization of two- and three-way interactions. There is a vignette for Exploring Interactions and another for Plotting Interactions.

IrregLong v0.1.1: Provides functions to analyze longitudinal data for which the times of observation are random variables that are potentially associated with the outcome process, and includes inverse-intensity weighting methods (Lin et al. (2004)) and multiple outputation (Pullenayegum (2016)). Look here for an overview.

missCompare v1.0.1: Implements a pipeline to test and compare various missing data imputation algorithms on simulated and real data. There is a Tutorial.

OutlierDetection v0.1.0: Implements various methods to detect outliers including: model-based (Barnett (1978)), distance-based (Hautamaki et al. (2004)), dispersion-based (Jin et al. (2001)), depth-based (Johnson et al. (1998)), and density-based (Ester et al. (1996)).

plsr v0.0.1: Provides functions for the partial least squares analysis of the relation between two high-dimensional data sets. See McIntosh & Lobaugh (2004) and the vignette for more information.

pliable v1.1: Fits a pliable lasso model. For details see Tibshirani and Friedman (2018) and the package vignette.

PointFore v0.2.0: Provides functions to estimate specification models for the state-dependent level of an optimal quantile/expectile forecast along with Wald Tests and a test of overidentifying restrictions. There is a Tutorial and vignettes on the GDP Greenbook and Preciptation examples.

segmenTier v0.1.2: Implements a dynamic programming solution to segmentation based on maximization of arbitrary similarity measures within segments, based on the theory described in Machne et al. (2017). The vignette provides an Introduction.

TextForecast v0.1.1: Provides functions for regression analysis and forecasting using textual data, which are based on Lima (2018). The vignette shows how to use the package.

Time Series

Rlgt v0.1-2: Provides functions to use rstan to fit several Global Trend models for time series forecasting that are Bayesian generalizations and extensions of some Exponential Smoothing models. There is an Intorduction to global trend time series forecasting and an Introduction to the package.

tsfeatures v1.0.0: Implements methods for extracting various features from time series data as described in Hyndman et al. (2013) , Kang et al.(2017) and Fulcher et al. (2013). The vignette contains examples.

Utilities

pak v0.1.2: Streamlines and improves package installation. See README.

qs v0.14.1: Provides functions for quickly writing and reading any R object to and from disk. See the vignette for use and timings.

ropendata v0.1.0: Provides functions to collect cyber-security data and make it available via the Open Data portal. Look at README for information on using the package.

rosr v0.0.5: Provides methods to create reproducible academic projects with integrated academic elements, including datasets, references, codes, images, manuscripts, dissertations, slides and so on.

ShinyEventLogger v0.1.1: Implements a logging framework for complex Shiny apps. The vignette shows how to start logging.

Visualization

gratia v0.2-8: Provides graceful ggplot-based graphics and utility functions for working with generalized additive models (GAMs) fitted using the mgcv package. Look here for examples.

jskm v0.3.1: Provides the function jskm() to create publication quality Kaplan-Meier plots with at-risk tables below, and svyjskm() to plot a weighted Kaplan-Meier estimator.

Paid in Books: An Interview with Christian Westergaard

Thu, 07 Mar 2019 00:00:00 +0000

R is greatly benefiting from new users coming from disciplines that traditionally did not provoke much serious computation. Journalists¹ and humanist scholars², for example, are embracing R. But, does the avenue from the Humanities go both ways? In a recent conversation with Christian Westergaard, proprietor of Sophia Rare Books in Copenhagen, I was delighted to learn that it does.

JBR: I was very pleased to learn when I spoke with you recently at the California Antiquarian Book Fair that you were an S and S+ user in graduate school. What were you studying and how how was S and S+ helpful?

CW: I did a Master’s in mathematics and a Bachelor’s in statistics at the University of Copenhagen in Denmark, graduating in 2005. During the first year of my courses in statistics, we were quickly introduced to S+ in order to do monthly assignments. In these assignments, we were to apply the theory we had learned in the lectures on some concrete data. I still remember how difficult I initially found applying the right statistical tools to real-world problems, rather than just understanding the math in theoretical statistics. I developed a deep respect for applied statistics. Our minds can easily be deceived and we need proper statistics to make the right decisions.

JBR: How did you move from technical studies to dealing in rare scientific books and manuscripts? On the surface it seems that these might be two completely unrelated activities. How did you find a path between these two worlds?

CW: It was a gradual shift. When I first began studying, I went into an old antiquarian book shop to acquire some second-hand math and statistic books to supplement my ordinary course texts. One of them was Statistical Methods by Anders Hald. Hald was no longer working at the university, but his text had become a classic. I was fascinated by this book shop. The owner was an old, grey-haired man sitting behind a huge stack of books smoking a pipe, and writing his sales catalogues on an old IBM typing machine. He allowed me to go down into his cellar where there books everywhere from floor to ceiling. There were many books which I wanted to acquire down there, but I hardly had any money. It was a mess in the cellar and I offered to tidy up if he could pay me in the books I wanted, and he agreed. I loved coming to work there, and I continued to do so my entire studies. My boss gave me more and more responsibility and put me in charge of the mathematics, physics, statistics and science books in general. When I finished my masters, I was considering doing a PhD. I loved mathematics and still do until this day. But I also found that when I woke up in the morning, I was thinking of antiquarian books and in the evening I couldn’t get to bed because I was thinking of books. It gave me energy and happiness. So I thought, why not try and be a rare book dealer for a year or two and see how it works out? It’s been 14 years since I made that decision, and I have really enjoyed it. In 2009, I decided to start my own company and specialize in important books and manuscripts in science.

JBR: What is it like to be immersed in these rare artifacts that were so important for the transmission of scientific knowledge? What kinds of scholars do you consult to establish the authenticity of works like Euler’s Opuscula Varii Argumenti or Cauchy’s Leçons sur le calcul diffrentiel?

CW: I feel privileged to handle some of these objects on a daily basis. One day I am sitting with an original autograph manuscript by Einstein doing research on relativity, and the next day I have a presentation copy of Darwin’s Origin of Species in my hands. These are objects which have changed the world and the way we think about ourselves. In addition to the books and manuscripts, I find the people who I meet extremely interesting. A few years before Anders Hald (whose book had originally brought me into my old boss’ shop) passed away, I went to buy his books. He was 92 and completely fresh in his mind. We spoke about the history of statistics – a subject about which he authored several books.

JBR: I have noticed that collectors seem to be very interested in the works of twentieth-century mathematicians and physicists. You have works by Alonzo Church, Kurt Gödel, Richard Feynman, and others in your catalogue. But your roster of statisticians seems to focus on the old masters such as Laplace and de Moivre. Are collectors also interested in Karl Pearson, Udny Yule, and R. A. Fisher?

CW: Certainly. Maybe so much so that every time I get one of Pearson or Fisher’s main papers they sell immediately. That’s why you don’t see them in my stock at the moment.

JBR: I noticed that you had two works by Sofya Vasilyevna Kovalevskaya on display in California. Do you see a renewed interest in the works of women scientists and mathematicians, or is this remarkable and brilliant woman an exception?

CW: There has definitely been a renewed interest in exceptional woman scientists. A few years ago the New York-based Grolier Club hosted an exhibition called ‘Extraordinary Women in Science and Medicine’, and several institutions are focusing on the subject. These woman who broke through the social constraints against them are exceptional and fascinating people.

JBR: Although there are notable exceptions (Donald Knuth’s typesetting comes to mind), I think most data scientists, computer scientists, and statisticians work in a digital world of ebooks and poorly printed texts. Do you think that the technical book as a collectable artifact will survive the twenty-first century? What advice would you give to working data scientists and statisticians who are interested in collecting?

CW: Good question. Many important papers nowadays are not even printed, and the only physical material a researcher might have left from some landmark work might be some scribbles he or she did on a piece of paper. There are examples of people who collect digital art. They use various ways of signing or otherwise authenticating the artists work even if it’s on a USB stick. Maybe that’s how some research papers might be collected in the future?

My advice for anyone wanting to start collecting would be to first focus on some of the classics in their field or some other field that fascinates them. The classics will have been collected by many others in the past and there will be good descriptions, bibliographies, and catalogues describing them and why they are collectible. That way one will gradually get a feeling about which mechanisms are important when collecting and what to focus on, e.g., condition, provenance, etc. And then I’d say it’s important to build a good relationship with at least one dealer with a good reputation in the trade. Any great collection is built on a collaboration were collectors and dealers work together.

JBR: Excellent advice! Thank you Christian.

¹ For example, have a look at some of the R training at this year’s IRE-CAR Conference.

² See, for example, these University of Washington resources for the digital humanities.

Sophia Rare Books (Copenhagen), specializes in rare and important books and manuscripts in the History of Science and Medicine fields.

January 2019: “Top 40” New CRAN Packages

Mon, 25 Feb 2019 00:00:00 +0000

One hundred and fifty-three new packages made it to CRAN in January. Here are my “Top 40” picks in eight categories: Computational Methods, Data, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

cPCG v1.0: Provides a function to solve systems of linear equations using a (preconditioned) conjugate gradient algorithm. The vignette shows how to use the package.

RcppDynProg v0.1.1: Implements dynamic programming using Rcpp. Look here for examples.

Data

cimir v0.1-0: Provides functions to connect to the California Irrigation Management Information System (CIMIS) Web API. See the Quick Start for details.

ecmwfr v1.1.0: Provides a programmatic interface to the European Centre for Medium-Range Weather Forecasts’ public and restricted dataset web services ECMWF, as well as Copernicus’s Climate Data Store CDS, allowing users to download weather forecasts and climate data. There are vignettes for both CDS and ECMWFR.

germanpolls v0.2: Provides functions to extract data from Wahlen.de.

nhdR v0.5.1: Provides tools for working with the National Hydrography Dataset, with functions for querying, downloading, and networking both the NHD and NHDPlus datasets. There are vignettes for Creating Simple Maps and Quering Flow Information, as well as an example.

snotelr v1.0.1: Provides a programmatic interface to the SNOTEL snow data. See the vignette for information.

wdpar v0.0.2: Provides an interface to the World Database on Protected Areas (WDPA). Data is obtained from Protected Planet. See the README for information.

Machine Learning

analysisPipelines v1.0.0: Implements an R interface that enables data scientists to compose inter-operable pipelines between R, Spark, and Python for data manipulation, exploratory analysis, modeling, and reporting. There are vignettes for Python Functions, R data frames, Spark data frames, Interoperable Pipelines, Meta-pipelines, Streaming Analysis Pipelines, and Using Pipelines with Spark.

bender v0.1.1: Implements an R client for Bender Hyperparameters optimizer.

FiRE v1.0: Implements an algorithm to find outliers and rare entities in voluminous datasets. Look here for information.

foto v1.0.0: Implements the Fourier Transform Textural Ordination method, which uses a principal component analysis on radially averaged, two-dimensional Fourier spectra to characterize image texture. See the vignette for details.

RcppHNSW v0.1.0: Provides bindings to the Hnswlib C++ library for Approximate Nearest Neighbors.

ruimtehol v0.1.2: Wraps the StarSpace library, allowing users to calculate word, sentence, article, document, webpage, link, and entity embeddings. The techniques are explained in detail in Wu et al. (2017). See the vignette for more information.

zoomgrid v1.0.0: Implements a grid search algorithm with zoom to help solve difficult optimization problems where there are many local optima inside the domain of the target function. Look here for information.

Medicine

bayesCT v0.99.0: Provides functions to simulate and analyze Bayesian adaptive clinical trials, incorporating historical data and allowing for early stopping. There is an Introduction, and vignettes for Binomial and Normal outcomes.

BioMedR v1.1.1: Provides tools for calculating 293 chemical descriptors and 14 kinds of chemical fingerprints, 9920 protein descriptors based on protein sequences, more than 6000 DNA/RNA descriptors from nucleotide sequences, and six types of interaction descriptors. There is a very informative vignette.

dr4pl v1.1.8: Models the relationship between dose levels and responses in a pharmacological experiment using the 4 Parameter Logistic model, and provides bounds that prevent parameter estimates from diverging. See Gadagkar and Call (2015) and Ritz et al. (2015) for background information, and the vignette for examples.

GMMAT v1.0.3: Provides functions to perform association tests using generalized linear mixed models (GLMMs) in genome-wide association studies (GWAS) and sequencing association studies. See Chen et al. (2016) and Chen et al. (2019) for background information, and the vignette for an introduction to the package.

Science

ethnobotanyR v0.1.4: Implements functions to calculate common quantitative ethnobotany indices to assess the cultural significance. See Tardio and Pardo-de-Santayana (2008) for background information, and the vignette for information on the package.

wsyn v1.0.0: Implements tools for a wavelet-based approach to analyzing spatial synchrony, principally in ecological data. The vignette gives the details.

Statistics

apcf v0.1.2: Implements the adapted pair correlation function, which transfers the concept of the pair correlation function from point patterns to patterns of objects of finite size and irregular shape. This is a re-implementation of the method suggested by Nuske et al. (2009). See the vignette for details.

concurve v1.0.1: Provides functions to compute confidence (compatibility/consonance) intervals for various statistical tests, along with their corresponding P-values and S-values. Consonance functions allow modelers to determine what effect sizes are compatible with the test model at various compatibility levels. For details, see Poole (1987), Schweder and Hjort (2002), Singh, Xie, and Strawderman (2007), and Amrhein, Trafimow and Greenland (2018). See the vignette for examples.

IMaGES v0.1: Provides functions to implement Independent Multiple-sample Greedy Equivalence Search (IMaGES), a causal inference algorithm for creating aggregate graphs and structural equation modeling data for one or more datasets. See Ramsey et. al (2010) for background information. There is a vignette.

metamer v0.1.0: Provides functions to create data with identical statistics (metamers) using an iterative algorithm proposed by Matejka & Fitzmaurice (2017). See README for help with the package.

mimi v0.1.0: Implements functions to estimate main effects and interactions in mixed data sets with missing values. Estimation is done through a convex program where main effects are assumed sparse and the interactions low-rank. See Geneviève et al. (2018).

pcLasso v1.1: Implements a method for fitting the entire regularization path of the principal components lasso for linear and logistic regression models. See Tay, Friedman, and Tibshirani (2014) for details and the vignette for an Introduction.

qrandom v1.1: Implements an API to the ANU Quantum Random Number Generator, provided by the Australian National University, that generates true random numbers in real-time by measuring the quantum fluctuations of the vacuum. The quantum Random Number Generator is based on the papers by Symul et al. (2011) and Haw, et al. (2015). Look here for live random numbers.

rstap v1.0.3: Provides tools for estimating spatial temporal aggregated predictor models with stan. See the vignette for an introduction.

ROCit v1.1.1: Provides functions to calculate and visualize performance measures for binary classifiers. The vignette describes the details.

surveysd v1.0.0: Provides functions to calculate point estimates and their standard errors in complex household surveys using bootstrap replicates. A comprehensive description of the methodology can be found here.

Utilities

askpass v1.1: Provides safe password entry for R, Git, and SSH. Look here for help.

logger v0.1: Provides a flexible and extensible way of formatting and delivering log messages with low overhead. There is an Introduction and vignettes on The Anatomy of a Log Request, Format Customization, Migration, Benchmarks, Logging, and Extensions.

pagedown v0.1: Implements tools to use the paged media properties in CSS and the JavaScript library paged.js to split the content of an HTML document into discrete pages. See the README for details.

rmd v0.1.4: Provides functions to manage multiple R Markdown packages. Look here for information.

tor v1.1.1: Provides functions to enable users to import multiple files at the same time. See the README for details.

vitae v0.1.0: Provides templates and functions to simplify the production and maintenance of curricula vitae. There is an Introduction and a vignette for creating templates.

Visualization

gganimate v1.0.1: Implements a ggplot2-compatible grammar for creating animations. The vignette will get you started.

RIdeogram v0.1.1: Implement tools to draw SVG (Scalable Vector Graphics) graphics to visualize and map genome-wide data in ideograms. See the vignette for information.

voronniTreeMap v0.2.0: Provides functions to create Voronni tree maps using the d3.js framework. Look here for examples.

A Few New R Books

Wed, 20 Feb 2019 00:00:00 +0000

Greg Wilson is a data scientist and professional educator at RStudio.

As a newcomer to R who prefers to read paper rather than pixels, I’ve been working my way through a more-or-less random selection of relevant books over the past few months. Some have discussed topics that I’m already familiar with in the context of R, while others have introduced me to entirely new subjects. This post describes four of them in brief; I hope to follow up with a second post in a few months as I work through the backlog on my desk.

First up is Sharon Machlis’ Practical R for Mass Communcation and Journalism, which is based on the author’s workshops for journalists. This book dives straight into doing the kinds of things a busy reporter or news analyst needs to do to meet a 5:00 pm deadline: data cleaning, presentation-quality graphics, and maps take precedence over control flow or the niceties of variable scope. I particularly enjoyed the way each chapter starts with a realistic project and works through what’s needed to build it. People who’ve never programmed before will be a little intimidated by how many packages they need to download if they try to work through the material on their own, but the instructions are clear, and the author’s enthusiasm for her material shines through in every example. (If anyone is working on a similar tutorial for sports data, please let me know - I have more than a few friends it would make very happy.)

In contrast, Chris Beeley and Shitalkumar Sukhdeve’s Web Application Development with R Using Shiny focuses on a particular tool rather than a industry vertical. It covers exactly what its title promises, step by step from the basics through custom JavaScript functions and animations through persistent storage. Every example I ran was cleanly written and clearly explained, and it’s clear that the authors have tested their material with real audiences. I particularly appreciated the chapter on code patterns - while I’m still not sure I fully understand when and how to use isolate() and req(), I’m much less confused than I was.

Functional programming has been the next big thing in computing since I was a graduate student in the 1980s. It does finally seem to be getting some traction outside the craft-beer-and-Emacs community, and Functional Programming in R by Thomas Mailund looks at how these ideas can be used in R. Mailund writes clearly, and readers who don’t have a background in computer science may find this a gentle way into a complex subject. However, despite the subtitle “Advanced Statistical Programming for Data Science, Analysis and Finance”, there’s nothing particularly statistical or financial about the book’s content. Some parts felt rushed, such as the lightning coverage of point-free programming (which should have had either a detailed exposition or no mention at all), but my biggest complaint about the book is its price: I think $34 for 100 pages is more than most people will want to pay.

Finally, we have Stefano Allesina and Madlen Wilmes’ Computing Skills for Biologists. As the subtitle says, this book presents a toolbox that includes Python, Git, LaTeX, and SQL as well as R, and is aimed at graduate students in biology who have just realized that a few hundred megabytes of messy data are standing between them and their thesis. The authors present the basics of each subject clearly and concisely using real-world data analysis examples at every turn. They freely admit in the introduction that coverage will be broad and shallow, but that’s exactly what books like this should aim for, and they hit a bulls eye. The book’s only weakness - unfortunately, a significant one - is an almost complete lack of diagrams. There are only six figures in its 400 pages, and none in the material on visualization. I realize that readers who are coding along with the examples will be able to view some plots and charts as they go, but I would urge the authors to include these in a second edition.

R is growing by leaps and bounds, and so is the literature about it. If you have written or read a book on R recently that you think others would be interested in, please let us know - we’d enjoy checking it out.

Stefano Allesina and Madlen Wilmes: Computing Skills for Biologists: A Toolbox. Princeton University Press, 978-0691182759.

Chris Beeley and Shitalkumar Sukhdeve: Web Application Development with R Using Shiny (3rd ed.). Packt, 2018, 978-1788993128.

Sharon Machlis: Practical R for Mass Communcation and Journalism. Chapman & Hall/CRC, 2018, 978-1138726918.

Thomas Mailund: Functional Programming in R: Advanced Statistical Programming for Data Science, Analysis and Finance. Apress, 2017, 978-1484227459.

December 2018: “Top 40” New CRAN Packages

Wed, 30 Jan 2019 00:00:00 +0000

By my count, 157 new packages stuck to CRAN in December. Below are my “Top 40” picks in ten categories: Computational Methods, Data, Finance, Machine Learning, Medicine, Science, Statistics, Time Series, Utilities and Visualization. This is the first time I have used the Medicine category. I am pleased that a few packages that appear to have clinical use made the cut. Also noteworthy in this month’s selection are the inclusion of four packages from the Microsoft Azure team (stuffing 41 packages into the “Top 40”), and some eclectic, but fascinating packages in the Science section.

Computational Methods

ar.matrix v0.1.0: Provides functions that use precision matrices and Choleski factorization to simulates auto-regressive data. The README offers examples.

mvp v1.0-2: Provides functions for the fast symbolic manipulation polynomials. See the vignette and this R Journal paper for details on how to create this image of the Rosenbrock function.

pomdp v0.9.1: Provides an interface to pomdp-solve, a solver for Partially Observable Markov Decision Processes (POMDP). See the vignette for examples.

Data

dbparser v1.0.0: Provides a tool for parsing the DrugBank XML database. The vignette shows how to get started.

rdhs v0.6.1: Implements a client querying the DHS API to download and manipulate survey datasets and metadata. There are introductions to using rdhs and the rdhs client, an extended example about Anemia prevalence, and vignettes on Country Codes, Interacting with the geojson API results, and Testing.

Finance

optionstrat v1.0.0: Implements the Black-Scholes-Merton option pricing model to calculate key option analytics and graphical analysis of various option strategies. See the vignette.

riskParityPortfolio v0.1.1: Provides functions to design risk parity portfolios for financial investment. In addition to the vanilla formulation, where the risk contributions are perfectly equalized, many other formulations are considered that allow for box constraints and short selling. The package is based on the papers: Feng and Palomar (2015), Spinu (2013), and Griveau-Billion et al.(2013). See the vignette for an example.

Machine Learning

BTM v0.2: Provides functions to find Biterm topics in collections of short texts. In contrast to topic models, which analyze word-document co-occurrence, biterms consist of two words co-occurring in the same short text window.

ParBayesianOptimization v0.0.1: Provides a framework for optimizing Bayesian hyperparameters according to the methods described in Snoek et al. (2012). There are vignettes on standard and advanced features.

Medicine

LUCIDus v0.9.0: Implements the LUCID method to jointly estimate latent unknown clusters/subgroups with integrated data. See the vignette for details.

metaRMST v1.0.0: Provides functions that use individual patient-level data to produce a multivariate meta-analysis of randomized controlled trials with the difference in restricted mean survival times ( RMSTD ).

webddx v0.1.0: Implements a differential-diagnosis generating tool. Given a list of symptoms, the function query_fz queries the FindZebra website and returns a differential-diagnosis list.

Science

bioRad v0.4.0: Provides functions to extract, visualize, and summarize aerial movements of birds and insects from weather radar data. There is an Introduction and a vignette on Exercises.

pmd v0.1.1: Implements the paired mass distance analysis proposed in Yu, Olkowicz and Pawliszyn (2018) for gas/liquid chromatography–mass spectrometry. See the vignette for an introduction.

tabula v1.0.0: Provides functions to examine archaeological count data and includes several measures of diversity. There are vignettes on Diversity Measures, Matrix Classes, and Matrix Seriation. This last vignette includes an example reproducing the results of Peeples and Schachner (2012).

traitdataform v0.5.2: Provides functions to assist with handling ecological trait data and applying the Ecological Trait-Data Standard terminology described in Schneider et al. (2018).

waterquality v0.2.2: Implements over 45 algorithms to develop water quality indices from satellite reflectance imagery. The vignette introduces the package.

Statistics

areal v0.1.2: Implements areal weighted interpolation with support for multiple variables in a workflow that is compatible with the tidyverse and sf frameworks. There are vignettes on Areal Interpolation, Wieghted Areal Interpoaltion, and Preparing Data for Interpolation.

FLAME v1.0.0: Implements the Fast Large-scale Almost Matching Exactly algorithm of Roy et al. (2017) for causal inference. Look at the README to get started.

mistr v0.0.1: Offers a computational framework for mixture distributions with a focus on composite models. There is an Introduction and a vignette on Extensions.

mlergm v0.1: Provides functions to estimate exponential-family random graph models for multilevel network data, assuming the multilevel structure is observed. There is a Tutorial.

MTLR v0.1.0: Implements the Multi-Task Logistic Regression (MTLR) proposed by Yu et al. (2011). See the vignette.

mulitRDPG v1.0.1: Provides functions to fit the Multiple Random Dot Product Graph Model and performs a test for whether two networks come from the same distribution. See Nielsen and Witten (2018) for details.

ocp v0.1.0: Implements the Bayesian online changepoint detection method of Adams and MacKay (2007) for univariate or multivariate data. Gaussian and Poisson probability models are implemented. The vignette provides an introduction.

probably v0.0.1: Provides tools for post-processing class probability estimates. See the vignettes Where does probability fit in? and Equivocal Zones.

smurf v1.0.0: Implements the SMuRF algorithm of Devriendt et al. (2018) to fit generalized linear models (GLMs) with multiple types of predictors via regularized maximum likelihood. See the package Introduction.

subtee v0.3-4: Provides functions for naive and adjusted treatment effect estimation for subgroups. Proposes model averaging Bornkamp et al. (2016) and bagging Rosenkranz (2016) to address the problem of selection bias in treatment effect estimation for subgroups. There is a Introduction and vignettes for the plot and subbuild functions.

xspliner v0.0.2: Provides functions to assist model building using surrogate black-box models to train interpretable spline based, additive models. There are vignettes on Basic Theory and Usage, Automation, Classification, Use Cases, Graphics, Extra Information, and the xspliner Environment.

Time Series

mfbvar v0.4.0: Provides functions for estimating mixed-frequency Bayesian vector autoregressive (VAR) models with Minnesota or steady-state priors as those used by Schorfheide and Song (2015), or by Ankargren, Unosson and Yang (2018). Look at the GitHub page for an example.

NTS v1.0.0: Provides functions to simulate, estimate, predict, and identify models for nonlinear time series.

Utilities

AzureContainers v1.0.0: Implements an interface to container functionality in Microsoft’s Azure cloud that enables users to manage the the Azure Container Instance, Azure Container Registry, and Azure Kubernetes Service. There are vignettes on Plumber model deployment and Machine Learning server model deployment.

AzureRMR v1.0.0: Implements lightweight interface to the Azure Resource Manager REST API. The package exposes classes and methods for OAuth authentication and working with subscriptions and resource group. There is an Introduction and a vignette on Extending AzureRMR.

AzureStor v1.0.0: Provides tools to manage storage in Microsoft’s Azure cloud. See the Introduction.

AzureVM v1.0.0: Implements tools for working with virtual machines and clusters of virtual machines in Microsoft’s Azure cloud. See the Introduction.

cliapp v0.1.0: Provides functions that facilitate creating rich command line applications with colors, headings, lists, alerts, progress bars, and custom CSS-based themes. See the README for examples.

projects v0.1.0: Provides a project infrastructure with a focus on manuscript creation. See the README for the conceptual framework and an introduction to the package.

remedy v0.1.0: Implements an RStudio Addin offering shortcuts for writing in Markdown.

solartime v0.0.1: Provides functions for computing sun position and times of sunrise and sunset. The vignette offers an overview.

Visualization

easyalluvial v0.1.8: Provides functions to simplify Alluvial plots for visualizing categorical data over multiple dimensions as flows. See Rosvall and Bergstrom (2010). See the README for details.

spatialwidget v0.2: Provides functions for converting R objects, such as simple features, into structures suitable for use in htmlwidgets mapping libraries. See the vignette for details.

transformr v0.1.1: Provides an extensive framework for manipulating the shapes of polygons and paths and can be seen as the spatial brother to the tweenr package. See the README for details.

2018 R Views Review and Highlights

Wed, 02 Jan 2019 00:00:00 +0000

2018 was a good year for R Views. With a total of sixty-three posts for the year, we exceeded the pace of at least one post per week. But, it wasn’t quantity we were shooting for. Our main goal was, and continues to be, featuring thoughtful commentary on topics of interest to the R Community and in-depth technical elaboration of R language applications.

Before highlighting a few of my favorite posts for 2018, I would like to express my profound gratitude to our guest bloggers (R Community members who are not employed at RStudio), our regular RStudio contributors who sparkled with creativity while meeting committed deadlines, and you, our readers, who made it all worthwhile.

At the top of my list for 2018 posts is the interview with mathematician Noah Giansiracusa: A Mathematician’s Pespective on Topological Analysis and R. Although my name is on the byline, this is really a guest post. Professor Giansiracusa’s openness, enthusiasm, lucidity and excitement about doing mathematics and finding genuine value in using R makes this post outstanding.

The three part series of posts: Statistics in Glaucoma (Part I, Part II and Part III) by statisticians Sam Berchuck and Joshua Warren establishes a new standard for posts introducing academic research. It is a masterful exposition of their quest to find a predictive model of disease progression. More than tutorial on the underlying R packages, the series prepares interested readers for reading the authors’ academic research papers.

Other noteworthy guest posts included:
* Eric Anderson: Alternative Design for Shiny
* Anqi Fu, Balasubramanian Narasimhan, Stephen Boyd: CVXR: A Direct Standardization Example
* David Kane: Player Data for the 2018 FIFA World Cup
* Roland Stevenson: In-database xgboost predictions with R
* Sebastian Wolf: How to Build a Shiny “Truck”!

Regular R Views readers will know that RStudio’s Jonathan Regenstein’s posts in his series Reproducible Finance with R are always dependable “good reads”. In addition to Jonathan’s financial analyses, his posts feature Shiny applications and elegant code applicable to many every-day programming tasks. If you have any interest at all in doing Finance with R, I highly recommend Jonathan’s new book Reproducible Finance with R: Code Flows and Shiny Apps for Portfolio Analysis which is based on his R Views posts.

The 2018 R for the Enterprise series featured posts by RStudio solution engineers Cole Arendt, James Blair, Kelly O’Briant, Edgar Ruiz, Nathan Stephens, and Andrie de Vries that provided sophisticated, in-depth coverage of a variety of enterprise-level topics such as Slack and Plumber Part One and Part Two and Enterprise Dashboards with R Markdown.

Finally, if in an idle moment you would like to review some of the more interesting new packages that made it to CRAN in 2018, you can peruse my monthly Top 40 picks.

All of us at R Views wish you a Happy and Prosperous New Year!

November 2018: “Top 40” New Packages

Fri, 21 Dec 2018 00:00:00 +0000

Having absorbed an average of 181 new packages each month over the last 28 months, CRAN is still growing at a pretty amazing rate. The following plot shows the number of new packages since I started keeping track in August 2016.

This November, 171 new packages stuck to CRAN. Here is my selection for the “Top 40” organized into the categories: Computational Methods, Data, Finance, Machine Learning, Marketing Analytics, Science, Statistics, Utilities and Visualization.

Computational Methods

mixsqp v0.1-79: Provides optimization algorithms (Kim et al. (2012) based on sequential quadratic programming (SQP) for maximum likelihood estimation of the mixture proportions in a finite mixture model where the component densities are known. The vignette shows how to use the package.

polylabelr v0.1.0: Implements a wrapper around the C++ library polylabel from Mapbox, providing an efficient routine for finding the approximate pole of inaccessibility of a polygon. See README.

Riembase v0.2.1: Implements a number of algorithms to estimate fundamental statistics including Fréchet mean and geometric median for manifold-valued data. See Bhattacharya and Bhattacharya (2012) if you are interested in statistics on manifolds, and Absil et al (2007) for information on the computational aspects of optimization on matrix manifolds.

SolveRationalMatrixEquation v0.1.0: Provides functions to find the symmetric positive definite solution X such that X = Q + L (X inv) L^T given a symmetric positive definite matrix Q and a non-singular matrix L. See Benner et al. (2007) for the details and the vignette for an example.

Data

metsyn v0.1.2: Provides an interface to the Meteo France Synop data API. This is meteorological data recorded every 3 on 62 French meteorological stations.

neonUtilities v1.0.1: Provides an interface to the National Ecological Observatory NEON API. For more information, see the README file.

phenocamapi v0.1.2: Allows users to obtain phenological time-series and site metadata from the PhenoCam network. There is a Getting Started Guide and a vignette with examples.

rdbnomics v0.4.3: Provides access to hundreds of millions data series from DBnomics API. See the vignette for examples.

restez v1.0.0: Allows users to download large sections of GenBank and generate a local SQL-based database. A user can then query this database using restez functions.

Finance

crseEventStudy v1.0: Implements the Dutta et al. (2018) standardized test for abnormal returns in long-horizon event studies to improve the power and robustness of the tests described in Kothari/Warner (2007).

psymonitor v0.0.1: Provides functions to apply the real-time monitoring strategy proposed by Phillips, Shi and Yu (2015) (and here) to test for “bubbles”. There is a vignette on detecting crises and another on monitoring bubbles.

Machine Learning

pivmet v0.1.0: Provides a collection of pivotal algorithms for relabeling the MCMC chains in order to cope with the label switching problem in Bayesian mixture models. Functions also initialize the centers of the classical k-means algorithm in order to obtain a better clustering solution. There is a vignette on K-means clustering and another on Label Swithching.

RDFTensor v1.0: Implements tensor factorization techniques suitable for sparse, binary and three-mode RDF tensors. See Nickel et al. (2012), Lee and Seung, Papalexakis et al. and Chi and T. G. Kolda (2012) for details.

rfviz v1.0.0: Provides an interactive data visualization and exploration toolkit that implements Breiman and Cutler’s original Java based, random forest visualization tools. It includes both supervised and unsupervised classification and regression algorithms. The Berkekey website describes the original implementation.

rJST v1.0: Provides functions to stimulate the Joint Sentiment Topic model as described by Lin and He (2009) and Lin et al. (2012). See the Introduction for details.

Marketing Analytics

Marketmatching v1.1.1: Enables users to find the best control markets using time series matching and analyze the impact of an intervention. Uses the dtw package to do the matching and the CausalImpact package to analyze the causal impact. See the vignette for an example.

Science

EpiSignalDetection v0.1.1: Provides functions to detect possible outbreaks using infectious disease surveillance data at the European Union / European Economic Area or country level. See Salmon et al. (2016) for a description of the automatic detection tools and the vignette for an overview of the package.

memnet v0.1.0: Implements network science tools to facilitate research into human (semantic) memory including several methods to infer networks from verbal fluency data, various network growth models, diverse random walk processes, and tools to analyze and visualize networks. See Wulff et al. (2018) and the vignette for an introduction.

phylocomr v0.1.2: Implements an interface to Phylocom, a library for analysis of phylogenetic community structure and character evolution. See the vignette for and introduction to the package.

plinkQC v0.2.0: Facilitates genotype quality control for genetic association studies as described by Anderson et al. (2010). There are vignettes on Ancestry Estimation, Processing 1000 Genomes, HapMap III Data and Genotype Quality Control.

Statistics

BivRec v1.0.0: Implements a collection of non-parametric and semiparametric methods to analyze alternating recurrent event data. See README for examples.

cusum v0.1.0: Provides functions for constructing and evaluating CUSUM charts and RA-CUSUM charts with focus on false signal probability. The vignette offers an example.

dabestr v0.1.0: Offers an alternative to significance testing using bootstrap methods and estimation plots. See Ho et al (2018). There is a vignette on Bootstrap Confidence Intervals, another on Statistical Visualizations, and a third on creating Estimation Plots.

deckgl v0.1.8: Implements an interface to deck.gl, a WebGL-powered open-source JavaScript framework for visual exploratory data analysis of large data sets and supports basemaps from mapbox. There are fourteen brief vignettes, each devoted to a different plot layer, but look here for a brief overview.

LindleyPowerSeries v0.1.0: Provides functions to compute the probability density function, the cumulative distribution function, the hazard rate function, the quantile function and random generation for Lindley Power Series distributions. See Nadarajah and Si (2018) for details.

modi v0.1.0: Implements algorithms that take sample designs into account to detect multivariate outliers. See Bill and Hulliger (2016) for details and the vignette for an introduction.

MPTmultiverse v0.1: Provides a function to examine the multiverse of possible modeling choices. See the paper by Steegen et al. (2016) and the vignette for an overview of the package.

pterrace v1.0: Provides functions to plot the persistence terrace, a summary graphic for topological data analysis that helps to determine the number of significant topological features. See Moon et al. (2018).

randcorr v1.0: Implements the algorithm by Pourahmadi and Wang (2015) for generating a random p x p correlation matrix by representing the correlation matrix using Cholesky factorization and hyperspherical coordinates. See Makalic and Schmidt (2018) for the sampling process used.

SMFilter v1.0.3: Provides filtering algorithms for the state space models on the Stiefel manifold as well as the corresponding sampling algorithms for uniform, vector Langevin-Bingham and matrix Langevin-Bingham distributions on the Stiefel manifold. See the vignette.

Utilities

IRkernel v0.8.14: Implements a native R kernel for Jupyter Notebook. See README for information on how to use the package.

lobstr v1.0.0: Provides set of tools for inspecting and understanding R data structures inspired by str(). See README for information on the included functions.

parsnip v0.0.1: Implements a common interface allowing users to specify a model without having to remember the different argument names across different functions or computational engines. The vignette goes over the basics.

pkgsearch v2.0.1: Allows users to search CRAN R packages using the METACRAN search server.

stevedore v0.9.0: Implements an interface to the Docker API. There is an Introduction and a vignette with Examples.

vctrs v0.1.0: Defines new notions of prototype and size that are used to provide tools for consistent and well-founded type-coercion and size-recycling. There are vignettes on S3 vectors, Type and Size Stability and Prototypes.

vtree v0.1.4: Provides a function for drawing drawing variable trees plots that display information about hierarchical subsets of a data frame defined by values of categorical variables. The vignette offers an introduction.

zipR v0.1.0: Implements the Python zip() function in R. See the vignette.

Visualization

countcolors v0.9.0: Contains functions to count colors within color range(s) in images, and provides a masked version of the image with targeted pixels changed to a different color. Output includes the locations of the pixels in the images, and the proportion of the image within the target color range with optional background masking. There is an Introduction and an Example.

coveffectsplot v0.0.1: Provides forest plots to visualize covariate effects from either the command line or an interactive Shiny application. There is an Introduction.

October 2018: “Top 40” New Packages

Thu, 29 Nov 2018 00:00:00 +0000

One hundred eighty-five new packages made it to CRAN in October. Here are my picks for the “Top 40” in eight categories: Computational Methods, Data, Machine Learning, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methods

compboost v0.1.0: Provides a C++ implementation of component-wise boosting written to obtain high run-time performance and full memory control. The vignette shows how to use the package.

RcppEnsmallen v0.1.10.0.1: Implements an interface to the C++ based Ensmallen mathematical optimization library that provides a simple set of abstractions for writing an objective function to optimize. Optimizers include full-batch gradient descent techniques, small-batch techniques, gradient-free optimizers, and constrained optimization.

SAMpack v0.1.1: Implements Stochastic Approximation Monte Carlo (SAMC) samplers capable of sampling from multimodal or doubly intractable distributions. See Liang et al (2010) for a complete introduction to the method, and the vignette for an introduction to the package.

Data

crimedata v0.1.0: Provides access to publicly available, police-recorded open crime data from large cities in the United States that are included in the Crime Open Database.

nasapower v1.02: Implements an interface to POWER (Prediction Of Worldwide Energy Resource), NASA’s global meteorology, surface solar energy, and climatology data API. Look here for a quick start.

wikisourcer v0.1.1: Provides access to public domain works from Wikisource, a free library from the Wikimedia Foundation project. See the vignette for a package tutorial.

Machine Learning

gcForest v0.2.7: Provides an API interface to the Python implementation of Deep Forest, an alternative to Deep Learning. The algorithm is described in Zhou and Feng (2017), and there is a brief package tutorial.

galgo v1.4: Allows users to build multivariate predictive models from large data sets having a far larger number of features than samples, such as in functional genomics data sets. See Trevino and Falciani (2006) for details.

MachineShop v0.2.0: Provides a common interface for machine learning model fitting, prediction, performance assessment, and presentation of results. There is an Introduction and a note on Implementation Conventions.

mlflow v0.8.0: Provides an interface to MLflow, an open-source platform for the complete machine learning life cycle that supports installation, tracking experiments, running projects, and saving models.

sboost v0.1.0: Provides a fast, C++-based implementation of Freund and Schapire’s Adaptive Boosting (AdaBoost) algorithm, and includes methods for classifier assessment, predictions, and cross-validation.

Medicine

CoRpower v1.0.0: Provides functions to calculate power for assessment of intermediate biomarker responses as correlates of risk in the active treatment group in clinical efficacy trials, as described in Gilbert et al. (2016). The vignette demonstrates the math.

radtools v1.0.0: Provides a collection of utilities for navigating medical image data in DICOM and NIfTI formats. An emphasis on metadata allows simple conversion of image metadata to familiar R data structures, such as lists and data frames. The vignette shows how to use the package.

rpact v1.0.0: Provides functions for designing and analyzing confirmatory adaptive clinical trials with continuous, binary, and survival endpoints according to the methods described in the monograph by Wassmer and Brannath (2016). Look here for an overview.

Science

ClimProjDiags v0.0.1: Provides functions for computing metrics and indices for climate analysis, comparing models, and combining them into ensembles. There are vignettes on anomaly agreement, diurnal temperatures, extreme indices, and heat and cold wave duration.

DEVis v1.0.0: Provides a comprehensive tool set for data aggregation, visual analytics, exploratory analysis, and project management that builds upon the Bioconductor DESeq2 differential expression package. The vignette offers a comprehensive introduction.

epimdr v0.6-1: Provides functions for studying epidemics, including the S(E)IR model, time-series SIR and chain-binomial stochastic models, catalytic disease models, and coupled map lattice models. It is a companion to the book Epidemics: Models and Data in R and the Coursera course Epidemics Massive Online Open Course.

firebehavior v0.1.1: Implements fire behavior prediction models, including those documented in Scott & Reinhardt (2001) and Alexander et al. (2006). The vignette is informative.

lorentz v1.0.0: Provides the functionality to work with Lorentz transforms and the gyrogroup structure in Special Relativity.

pubchunks v0.1.0: Provides functions for extracting chunks of XML from scholarly articles without having to know how to work with XML. See README to get going.

Statistics

BayesMallows v0.1.1: Implements the Bayesian version of the Mallows rank model (Vitelli et al. (2018)(http://jmlr.org/papers/v18/15-481.html). The vignette provides the details.

contextual v0.9.1: Facilitates the simulation and evaluation of context-free and contextual multi-Armed Bandit policies or algorithms to ease the implementation, evaluation, and dissemination of both existing and new bandit algorithms and policies. See the Getting Started Guide and this list of posts for more information.

coxrt v1.0.0: Implements Cox Proportional Hazards regression for right-truncated data. The vignette gives the details.

crossrun v0.1.0: Estimates the joint distribution of number of crossings and the longest run in a series of independent Bernoulli trials. There is a vignette.

logisticRR v0.2.0: Asserting that relative risk is often of interest in public health, this package provides functions to return adjusted relative risks from logistic regression model under potential confounders. The vignette does the math.

lognorm v0.1.3: Estimates the distribution parameters and computes moments and other basic statistics of the lognormal distribution Limpert al. (2001), and also provides an approximation to the distribution of the sum of several correlated lognormally distributed variables Lo (2013). There is a vignette on Aggregating Correlated Random Variables and another on Approximating Sums.

lolog v1.1: Provides functions to estimate Latent Order Logistic (LOLOG) Models for Networks, and also visual diagnostics and goodness of fit metrics are provided. See Fellows (2018) for a detailed description of the methods. One vignette works through an example, and another introduces lolog models.

matrixNormal v0.0.0: Provides the functions to compute densities, probabilities, and random deviates of the Matrix Normal distribution. See Iranmanesh et.al. (2010)

outcomerate v1.0.1: Implements standardized survey outcome rate functions, including the response rate, contact rate, cooperation rate, and refusal rate that allow researchers to measure the quality of survey data using standards published by the American Association of Public Opinion Research. For details, see AAPOR (2016). The vignette provides an Introduction.

parmsurvfit v0.0.1: Fits right-censored data to a given parametric distribution, and produces summary statistics, hazard, cumulative hazard and probability plots, and the Anderson-Darling test statistic. There is a vignette.

ppgmmga v1.0.1: Implements a Projection Pursuit algorithm for dimension reduction based on Gaussian Mixture Models. The vignette provides a quick tour of the package.

RcppDist v0.1.1: Provides additional statistical distributions that can be called from C++ when writing code using Rcpp or RcppArmadillo. See the vignette for a list of the distributions supported.

simstandard v0.2.0: Enables the creation of simulated data from structural equation models with standardized loading. The vignette shows how to use the package.

Utilities

carrier v0.1.0: Enables users to create functions that are isolated from their environment. These isolated functions, also called crates, print at the console with their total size and can be easily tested locally before being sent to a remote.

carbonate v0.1.0: Implements an interface to carbon.js, which allows developers to create images of source code. There is a vignette on Tests and Coverage.

generics v0.0.1: In order to reduce potential package dependencies and conflicts, generics provides a number of commonly used S3 generics.

REPLesentR v0.3.0: Allows users to create presentations and display them inside the R REPL (console). Supports RMarkdown and other text format.

stationery v0.98.5.5: Provides templates, guides, and scripts for writing documents in LaTeX and R markdown to produce guides, slides, and reports; and includes several vignettes to assist new users of literate programming. There is an Overview, a vignette on R Markdown Basics, and another on R Markdown HTML, and a comparison between Sweave and Knitr code chunks.

Visualization

balance v0.1.6: Provides an alternative scheme for visualizing balances (used in compositional data analysis) as described in Quinn (2018), as well as a method for principal balance analysis. See the vignette for details.

trelliscopejs v0.1.14: Provides methods that make it easy to create a Trelliscope display specification for TrelliscopeJS, including high-level functions for creating displays from within dplyr or ggplot2 workflows. There is a vignette on trelliscope Documentation and a trelliscope Package Reference.

A Mathematician's Perspective on Topological Data Analysis and R

Wed, 14 Nov 2018 00:00:00 +0000

A few years ago, when I first became aware of Topological Data Analysis (TDA), I was really excited by the possibility that the elegant theorems of Algebraic Topology could provide some new insights into the practical problems of data analysis. But time has passed, and the sober assessment of Larry Wasserman seems to describe where things stand.

TDA is an exciting area and is full of interesting ideas. But so far, it has had little impact on data analysis.

Nevertheless, TDA researchers have been quietly working the problem and at least some of them are using R (see below). Since I first read Professor Wasserman’s paper, I have been very keen on getting the perspective of a TDA researcher. So, I am delighted to present the following interview with Noah Giansiracusa, Algebraic Geometer, TDA researcher and co-author of a recent JCGS paper on a new visualization tool for persistent homology.

Q: Hello Dr. Giansiracusa. Thank you for making time for us at R Views. How did you get interested in TDA?

While doing a postdoc in pure mathematics (algebraic geometry, specifically) I, probably like many people, could not escape a feeling that crept up from time to time—particularly during the more challenging moments of research frustration—that perhaps the efforts I was putting into proving abstract theorems might have been better spent working in a more practical, applied realm of mathematics. However, pragmatic considerations made me apprehensive at that point in my career to take a sudden departure, for I finally felt like I was gaining some momentum in algebraic geometry, developing a nice network of supportive colleagues, etc., and also that I would very soon be on the tenure track job market and I knew that if I was hired (a big “if”!) it would be for a subject I had actually published in, not one I was merely curious about or had recently moved into. But around this same time I kept hearing about an exciting but possibly over-hyped topic called topological data analysis: TDA. It really seemed to be in the air at the time (this was about five years ago).

Q: Why do you think TDA took off?

I can only speak for myself, but I think there were two big reasons that TDA generated so much buzz among mathematicians at the early stages.

First, it was then, and still is impossible to escape the media blitz on “big data” and the “data revolution” and related sentiments. This is felt strongly within academic circles (our deans would love us all to be working in data it seems!) but also in the mainstream press. Yet, I think pure mathematicians often felt somewhat on the periphery of this revolution: we knew that the modern developments in data and deep learning and artificial intelligence would not be possible without the rigorous foundations our mathematical ancestors had laid, but we also knew that most of the theorems we are currently proving would in all likelihood play absolutely zero role in any of the contemporary story. TDA provided a hope for relevance, that in the end the pure mathematician would come out of the shadows of obscurity and strike a data science victory proving our ineluctable relevance and superiority in all things technical—and this hope quickly turned to hype.

I think I and many other pure mathematicians were rooting for TDA, to show the world that our work has value. We tired of telling stories of how mathematicians invented differential geometry before Einstein used it in relativity and your GPS would not be possible without this. We needed a more fresh, decisive victory in the intellectual landscape; number theory used in cryptography is great, but still too specialized: TDA had the promise of bringing us into the (big) data revolution. And so we hoped, and we hyped.

And second, from a very practical perspective, I simply did not have time to retrain myself in applied math, the usual form of applied math based heavily on differential equations, modeling, numerical analysis, etc. But TDA seemed to offer a chance to gently transition to data science mathematical relevance—instead of starting from scratch, pure mathematicians such as myself would simply need to add one more chapter to our background in topics like algebraic topology and then we’d be ready to go and could brand ourselves as useful! And if academia didn’t work out, Google would surely rather open the doors of employment to a TDA expert than to a traditional algebraic topologist (or algebraic geometer, in my case).

I think these are two of the main things that brought TDA so much early attention before it really had many real-world successes under its belt, and they are certainly what brought me to it; well, that and also an innocent curiosity to understand what TDA really is, how it works, and whether or not it does what it claims.

Q: So how did you get started?

I first dipped my toes in the TDA waters by agreeing to do a reading course with a couple undergraduates interested in the topic; then I led an undergraduate/master’s level course where we studied the basics of persistent homology, downloaded some data sets, and played around. We chose to use R for that since there are many data sets readily available, and also because we wanted to do some simple experiments like sampling points from nice objects like a sphere but then adding noise, so we knew we wanted to have a lot of statistical functions available to us and R had that plus TDA packages already. While doing this I grew to quite like R and so have stuck with it ever since. In fact, I’m now using it also on a (non-TDA) project to analyze Supreme Court voting patterns from a computational geometry perspective.

Q: Do you think TDA might become a practical tool for statisticians?

First of all, I think this is absolutely the correct way to phrase the question! A few years ago TDA seemed to have almost an adversarial nature to it, that topologists were going to do what data scientists were doing but better because fancy math and smart people were involved. So the question at the time seemed to be whether TDA would supplant other forms of data science, and this was a very unfortunate way to view things.

But, it was easy to entirely discredit TDA by saying that it makes no practical difference whether your data has the homology of a Klein bottle, or there were no real-world examples where TDA had outperformed machine learning. This type of dialogue was missing the point. As your question suggests, TDA should be viewed as a tool to be added to the quiver of data science arrows, rather than an entirely new weapon.
In fact, while this clarification moves the dialogue in a healthy direction (TDA and machine learning should work together, rather than compete with each other!) I think there’s still one further step we should take here: TDA is not really a well-defined entity. For instance, when I see topics like random decision forests, it looks very much like topology to me! (Any graph, of which a tree is an example, is a 1-dimensional simplicial complex, and actually if you look under the hood, the standard Rips complex approach in TDA builds its higher dimensional simplicial complexes out of a 1-dimensional simplicial complex, so both random forests and TDA—and most of network theory—are really rooted in the common world of graph theory.)

Another example: the 0-dimensional barcode for the Rips flavor of TDA encodes essentially the same information as hierarchical clustering. All I’m saying here is that there’s more traditional data science in TDA than one might first imagine, and there’s more topology in traditional data science than one might realize. I think this is healthy, to recognize connections like these—it helps one see a continuum of intellectual development here rather than a discrete jump from ordinary data science to fancy topological data science.

That’s a long-winded way of saying that you phrased the question well. The (less long-winded) answer to the question is: Yes! Once one sees TDA as one more tool for extracting structure and statistics from data, it is much easier to imagine it being absorbed into the mainstream. It need not outperform all previous methods or revolutionize data science, it merely needs to be, exactly as you worded it, a practical tool. Data science is replete with tools that apply in some settings and not others, work better with some data than others, reveal relevant information sometimes more than others, and TDA (whatever it is exactly) fits right into this. There certainly will be some branches of TDA that gain more traction over the years than others, but I am absolutely convinced that at least some of the methods used in TDA will be absorbed into statistical learning just as things like random decision trees and SVMs (both of which have very topological/geometric flavors to them!) have. This does not mean that every statistician needs to learn TDA, just as not every statistician needs to learn all the latest methods in deep learning.

Q: Where do you think TDA has had the most success?

Over the past few years I think the biggest strides TDA has made have been in terms of better interweaving it with other methods and disciplines—so big topics with lots of progress but still room for more have included confidence intervals, distributions of barcodes, feature selection and kernel methods in persistent homology. These are all exciting topics and healthy for the long-term development of TDA.

I think, perhaps controversially, the next step might actually be to rid ourselves of the label TDA. For one thing, TDA is very geometric and not just topological (which is to say: distances matter!). But the bigger issue for me is that we should refer to the actual tools being used (mapper, persistent homology in its various flavors, etc.) rather than lump them arbitrarily together under this common label. It took many years for statisticians to jump on the machine learning bandwagon, and part of what prevented them from doing so sooner was language; the field of statistical learning essentially translates machine learning into more familiar statistical terminology and reveals that it is just another branch of the same discipline. I suspect something similar will take place with TDA… er, I should say, with these recent topological and geometric data analysis tools.

Q: Do you think focusing on the kinds of concrete problems faced when trying to apply topological and algebraic ideas to data analysis will turn out to be a productive means of motivating mathematical research?

Yes, absolutely—and this is also a great question and a healthy way to look at things! Pure mathematicians have no particular agenda or preconceived notion of what they should and should not be studying: pure mathematics, broadly speaking, is logical exploration and development of structure and symmetry. The more intricate a structure appears to be, and the more connected to other structures we have studied, the more interested we tend to be in it. But that really is pretty much all we need to be interested—and to be happy.

So TDA provides a whole range of new questions we can ask, and new structures we can uncover, and inevitably many of these will tie back to earlier areas of pure mathematics in fascinating ways—all the while, throughout these explorations pure mathematicians likely will end up laying foundations that help provide a stable scaffolding for the adventurous data practitioners who jump into methodology before the full mathematical landscape has been revealed. So TDA absolutely will lead to new, important mathematical research: important both because we’ll uncover beautiful structures and connections, and important also because it will provide some certainty to the applied methods build on top of this.

Q: More specifically, what role might the R language play in facilitating the practice or teaching of mathematics?

Broadly speaking, I think many teachers—especially in pure mathematics—undervalue the importance of computer programming skills, though this is starting to change as pure mathematicians increasingly enjoy experimentation as a way of exploring small examples, honing intuition, and finding evidence for conjectures. While the idea of theorem-proof mathematics is certainly the staple of our discipline, it’s not the only way to understand mathematical structure. In fact, students often find mathematical material resonates with them much more strongly if they uncover a pattern by experimenting on a computer rather than just being fed it through lecture or textbooks. Concretely, if students play with something like the distribution of prime numbers, they might get excited to see the fascinating interplay between randomness and structure that emerges, and that can better prepare them to appreciate formally learning the prime number theorem in a classroom. So as things like TDA emerge, the number of pure mathematics topics that can be explored on a computer increases, and I think that’s a great thing.

Q: Where does R fit in?

Well, much of the mathematical exploration I’m referring to here is symbolic—so very precise and algebraic flavor—and R certainly has no limitations working precisely, but it’s not the main goal of the language so one likely would use a computer algebra system instead. But, one exciting thing TDA does help us see is that there’s a marvelous interface between the symbolic and numerical worlds (here represented by the topology and the statistics, respectively) and I think this is great for both teaching and research. The more common ground we find between topics that previously seemed quite disparate, the more chance we have of building meaningful interdisciplinary collaborations, the more perspectives we can provide our students to motivate and study something, and the more we see unity within mathematics. My favorite manifestation of this is that TDA is the study of the topology of discrete spaces—but discrete spaces have no non-trivial topology! What’s really going on then is that data gives us a discrete glimpse into a continuous, highly structured world, and TDA aims to restore the geometric structure lost due to sampling. In doing so one cannot, and should not, avoid statistics, so pure mathematics is brought meaningfully in contact with statistics and I absolutely love that. This means the R language finds a role in pure math where it may previously not have: topology with noise, algebra with uncertainty.

Thank you again! I think your ideas are going to inspire some R Views readers.

Editors note: here are some R packages for doing TDA:
* TDA contains tools for the statistical analysis of persistent homology and for density clustering.
* TDAmapper enables TDA using Discrete Morse Theory.
* TDAstats offers a tool set for TDA including for calculating persistent homology in a Vietoris-Rips complex.
* pterrace builds on TDA and offers a new multi-scale and parameter free summary plot for studying topological signals.

Serendipity at R / Medicine

Tue, 16 Oct 2018 00:00:00 +0000

We knew we were on to something important early on in the process of organizing R / Medicine 2018. Even during our initial attempts to articulate the differences between this conference and R / Pharma 2018, it became clear that the focus on the use of R and statistics in clinical settings was going to be a richer topic than just the design of clinical trials. However, it wasn’t until the conference got underway that we realized there was magic in the mix of attendees. R / Medicine attracted quite a few clinicians who were themselves using R in their work, or were in the process of teaching themselves R. This group catalyzed the discussions that continued throughout the conference, enabling high-bandwidth exchanges that would have otherwise suffered from the effort to translate between the two cultures. The small, single-track nature of the conference helped to keep the conversations going, with the questions and answers at the end of a given talk helping to enrich the quality of successive discussions.

Rob Tibshirani set the collaborative tone for the conference with his opening keynote talk describing the clinical forecasting system he and his collaborators have built to predict platelet usage for the Stanford hospitals. Big-league and big-impact, the system shows the promise of delivering real clinical and financial benefits. Tibshirani’s presentation of the modeling process also set the bar for clarity.

The other keynotes were also “top shelf”. Michael Lawrence spoke about Scientific Software In-the-Large. He laid out three challenges for scientific programming at this scale:
     * Integration of independently developed modules
     * Translation of analyses and prototypes into software
     * Scalability
and addressed these issues using examples from the Bioconductor project.

Victoria Stodden’s Keynote, Computational Reproducibility in Medical Research: Toward Open Code and Data, was a meditation on the need to reassess scientific transparency in an age where big data and computational power are driving medical research, and deep intellectual contributions are encoded in software. I was particularly struck by the idea that progress towards computational reproducibility depends on the coordination of stakeholders.

Perhaps the highest-energy talk of the conference (and maybe all of the conferences I have attended this year) was given by Yale’s Dr. Harlan Krumholz. Unfortunately, we have neither video nor slides from this keynote, but to give you some ideal of Dr. Krumholz iconoclastic work, look at the 2010 Forbes Article and this more recent article published in HealthAffairs. The following are some notes I managed to take at the talk between moments of mesmerization. With respect to medicine in general Dr. Krumholz said that:

There could not be a more exciting era in medicine. Medicine is emerging as an information science and the clinician’s role is changing to be a guide or interpretor, not a shaman.

Commenting on evidence-based medicine:

More than half of the guidlines in cardiology are not based on evidence.

With respect to medical data, he said:

The goal should be to take high-dimensional data and make it low-dimensional. Instead of thinking that everyone should have the same data, we should move towards thinking: How dow we use the data that we do have? There should be no missing data.

I took these statements to mean that teams of clinicians, statisticians, and data scientists should be working towards building predictive models for individual patients based on whatever data is available for them and whatever big data is relevant. This was clearly the music the crowd wanted to dance to.

The slides for most of the rest of the talks are available on the website. One talk I would like to highlight here is Nathaniel Phillips’ talk on Fast and Frugal Trees.

This talk addressed a recurring theme throughout the conference: the difference in decision making between the two cultures of statisticians and physicians. Probabilistic estimates to characteristic risk and to inform decision making are central to a statisticians worldview. Physicians, on the other hand, are in general not comfortable with probabilities, and when push comes to shove, prefer unambiguous guidelines and thresholds, such as blood pressure ranges, to inform treatment decisions. A vexing cultural problem is to identify effective decision models that have a chance of actually being used by clinicians.

The conference finished with a roundtable discussion with the theme Bridging the Two Cultures, with panelists Beth Atkinson, Joseph Chou, Peter Higgins, Stephan Kadauke, Chinonyerem Madu, and Jack Wasey representing both the statistical and clinical points of view. The moderator (me) began by asking three questions: 1. How do clinicians engage with statisticians and data scientists? 2. What are some key ideas you should know about collaborating? 3. In your experience, what kinds of engagements have been the most successful?

Panelists were free to respond as they felt inclined to any or all of the questions. As I recall, a consensus emerged around three key ideas: make an effort to empathize with colleagues, meet frequently and go out of your way to interact with colleagues, and carefully select projects and then cultivate them.

Planning is already underway for R / Medicine 2019. Mark the week of September 23rd, and stay tuned!

September 2018: Top 40 New Packages

Mon, 08 Oct 2018 00:00:00 +0000

September was another relatively slow month for new package activity on CRAN: “only” 126 new packages by my count. My Top 40 list is heavy on what I characterize as “utilities”: packages that either extend R in some fashion or make it easier to do things in R. This month, the packages I selected fall into eight categories: Data, Finance, Machine Learning, Science, Statistics, Time Series, Utilities and Visualization.

Data

trigpoints v1.0.0: Contains a complete data set of historic GB trig points (fixed survey points that help mapmakers and hikers) in British National Grid (OSGB36) coordinate reference system.

UKgrid v0.1.0: Provides a time series of the national grid demand (high-voltage electric power transmission network) in the UK since 2011. The vignette shows how to use the package.

Finance

jubilee v0.2-5: Implements a long-term forecast model called Jubilee-Tectonic model to forecast future returns of the U.S. stock market, Treasury yield, and gold price. The vignette shows the math.

portsort v0.1.0: Provides functions to sort assets into portfolios for up to three factors via a conditional or unconditional sorting procedure. There is an Introduction.

Machine Learning

crfsuite v0.1.1: Wraps the CRFsuite library allowing users to fit a conditional random field model. The focus is Natural Language Processing, and there are models for named entity recognition, text chunking, part of speech tagging, intent recognition, and classification. The vignette shows how to use the package.

ELMSO v1.0.0: Implements the algorithm described in Paulson, Luo, and James (2018); see here for a full-text version of the paper. The algorithm allocates budget across a set of online advertising opportunities.

embed v0.0.1: Provides functions to convert factor predictors to one or more numeric representations using simple generalized linear models or nonlinear models.

newsmap v0.6: Implements a semi-supervised model for geographical document classification ([Watanabe (2018)])(doi:10.¹⁰⁸⁰⁄_21670811.2017.1293487) with seed dictionaries in English, German, Spanish, Japanese, and Russian. See the README for an example.

splinetree v0.1.0: Provides functions to build regression trees and random forests for longitudinal or functional data using a spline projection method. Implements and extends the work of Yu and Lambert (1999). There is an Introduction and vignettes on trees and forests.

stylest v0.1.0: Provides functions to estimate the distinctiveness in speakers’ (authors’) style. Fits models that can be used for predicting speakers of new texts. See Spirling et al (2018) for the details and the vignette for an example on how to use the package.

Science

conStruct v1.0.0: Provides a method for modeling genetic data as a combination of discrete layers, within each of which relatedness may decay continuously with geographic distance. There are vignettes for formatting data, model construction, and on running and visualizing consStruct analyses.

episcan v0.0.1: Provides some efficient mechanisms to scan epistasis in genome-wide interaction studies (GWIS), and supports both case-control status (binary outcome) and quantitative phenotype (continuous outcome) studies. See Kam-Thong and Cxamara et al. (2011), Kam-Thong and Pütz et al. (2011), and the vignette.

Statistics

ahpsurvey v0.2.2: Implements the Analytic Hierarchy Process, a versatile multi-criteria decision-making tool introduced by Saaty (1987) that allows decision-makers to weigh attributes and evaluate alternatives presented to them. The vignette provides examples.

empirical v0.1.0: Implements empirical univariate probability density functions (continuous functions) and empirical cumulative distribution functions (step functions or continuous). The vignette provides examples.

basisMCMCplots v0.1.0: Provides functions for examining posterior MCMC samples from a single and multiple chains that interface with the NIMBLE software package. See de Valpine et al. (2017).

MetaStan v0.0.1: Provides functions to perform Bayesian meta-analysis using Stan. Includes binomial-normal hierarchical models and option to use weakly informative priors for the heterogeneity parameter and the treatment effect parameter, which are described in Guenhan, Roever, and Friede (2018). The vignette contains an example.

Opt4PL v0.1.1: Provides functions to obtain and evaluate various optimal designs for the 3-, 4-, and 5-parameter logistic models. The optimal designs are obtained based on the numerical algorithm in Hyun, Wong, Yang (2018).

rmatalog v1.0.0: Implements the metalog distribution, a modern, highly flexible, data-driven distribution. See Keelin (2016). The vignette provides an example.

rwavelet v0.1.0: Provides functions to perform wavelet analysis (orthogonal and translation invariant transforms) with applications to data compression or denoising. Most of the code is a port of the MATLAB Wavelab toolbox written by Donoho, Maleki and Shahram. The vignette provides examples.

samplingBigData v1.0.0: Provides methods for sampling large data sets, including spatially balanced sampling in multi-dimensional spaces with any prescribed inclusion probabilities. Written in C, it uses efficient data structures such as k-d trees that scale to several million rows on a modern desktop computer.

survivalAnalysis v0.1.0: Implements a high-level interface to perform survival analysis, including Kaplan-Meier analysis and log-rank tests and Cox regression. There are vignettes for univariate and multivariate survival analyses.

ungroup v1.1.0: Provides functions to implement a penalized composite link model for efficient estimation of smooth distributions from coarsely binned data. For a detailed description of the method and applications, see Rizzi et al. (2015). The vignette provides examples.

Time Series

bayesdfa v0.1.0: Implements Bayesian dynamic factor analysis, a dimension-reduction tool for multivariate time series, with Stan. The vignette shows how to identify extremes and latent regimes with glmmfields.

tbrf v0.1.0: Provides rolling statistical functions based on date and time windows instead of n-lagged observations. The vignette offers examples.

Utilities

atable v0.1.0: Provides functions to create tables for reporting clinical trials, calculate descriptive statistics and hypotheses tests, and arrange the results in a table with LaTeX or Word. The vignette provides examples.

av v0.2: Implements bindings to the FFmpeg AV library for working with audio and video in R.

binb v0.0.2: Provides a collection of LaTeX styles using Beamer customization for PDF-based presentation slides in RMarkdown. The vignette provides an example.

broom.mixed v0.2.2: Converts fitted objects from various R mixed-model packages into tidy data frames along the lines of the broom package.

codified v0.2.0: Allows authors to augment clinical data with metadata to create output used in conventional publications and reports. See the vignette for examples.

duawrangler v0.6.3: Allows users to create shareable data sets from raw data files that contain protected elements. There are vignettes on the motivation for the package and on securing data.

ipc v0.1.0: Provides tools for passing messages between R processes with Shiny Examples showing how to perform useful tasks. The vignette shows how to use the package.

piggyback v0.0.8: Works around git’s 50MB commit limit to allow larger (up to 2 GB) data files to piggyback on a repository as assets attached to individual GitHub releases. There is a package overview and a vignette on alternatives.

pysd2r v0.1.0: Uses reticulate to implement an interface to the pysd toolset, provides a number of pysd functions, and can read files in Vensim, mdl, and xmile formats. The vignette provides an overview.

radix v0.5: Provides functions to format scientific and technical articles for the web with Radix reader-friendly typography, flexible layout options for visualizations, and full support for footnotes and citations.

rbtc v0.1-5: Implements the RPC-JSON API for Bitcoin and provides utility functions for address creation and content analysis of the blockchain.

salty v0.1.0: Lets users take real or simulated data and salt it with errors commonly found in the wild, such as pseudo-OCR errors, Unicode problems, numeric fields with nonsensical punctuation, bad dates, etc. See README for examples.

Visualization

customLayout v0.2.0: Offers an extended version of the graphics::layout() function that also supports grid graphics, allowing users to create complicated drawing areas for multiple elements by combining much simpler layouts. The vignette for PowerPoint.

echarts4r v0.1.1: Allows users to create interactive charts by leveraging the Echarts JavaScript library. It includes 33 chart types, themes, Shiny proxies, and animations. Look here for an example.

ggparliament v2.0.0: Provides parliament plots to visualize election results as points in the architectural layout of the legislative chamber. There are vignettes for arranging parliament, basic plots, drawing majorities, emphasizing parliamentarians, faceting, hanging seats, highlighingt government, and labeling parties.

ggTimeSeries v1.0.1: Provides additional time series visualizations, such as calendar heat map, steamgraph, and marimekko. There is a vignette.

Some Thoughts on R / Pharma 2018

Wed, 03 Oct 2018 00:00:00 +0000

It’s no secret that there are few industries more competitive than the pharmaceutical industry. Big money placed on long-shot bets for block-buster drugs where being first makes all the difference means a constant struggle to gain a competitive edge. So, you might find it surprising that the inaugural R / Pharma Conference held this past August on the Harvard campus in a very classy auditorium was all about collaboration.

Some might also find it surprising that data scientists from competitive companies would gather to share information, but this is quite common. I have seen it before in other competitive industries, for example in IEEE-led standards initiatives, where engineers gather to forge a common technology. Not only is there the human need to share and learn from peers (and also brag a little), there is a larger force at play: a kind of market clearing operation where experts gather to gain as much of an advantage as they can by ensuring that no easily exploitable arbitrage opportunities remain.

It was a surprise, though (and I think a source of general amusement as the conference proceeded), that nearly every talk seemed to be about Shiny. Looking back, it is clear that it should not have been: 49% of the abstracts explicitly mention Shiny. This word cloud was built from the abstract submissions.

Shiny is basically a technology for sharing complex information across multiple organizations and stakeholders with different skill sets. Shiny, too, is all about collaboration. For a look into the large, production-grade Shiny app, bioWARP, see Sebastian Wolf’s recent post.

Other major themes addressed at the conference were: reproducible research, package administration, scaling R for production, and using R in a regulatory environment. This last theme was underscored by a strong FDA presence. Lilliam Rosario from the FDA Center for Drug Evaluation & Research delivered the opening keynote, in which she addressed the regulatory role of CDER and the use of R. FDA speaker Mat Souktup spoke about the need to transcend the compartmentalized culture common in medical research, and how open-source tools are helpful in working towards this goal. He explicitly noted along the way that the FDA does not specify what software may be used. The third FDA speaker, Paul Schuette, filled in some details associated with topics raised by Rosario and talked about the use of R and Shiny at CDER. Along these same lines, Andy Nicholls from GSK conducted a well-attended and very informative workshop on The Challenges of Validating R. You can find Andy’s slides here.

Other keynote speakers were Max Kuhn, who talked about Modeling in the tidyverse (slides here); Joe Cheng, who described how to use Shiny responsibly in pharma (slides here); and Michael Lawrence, who spoke about enabling open-source analytics in the enterprise.

Slides for some of the other presentations made at the conference may be found here. I expect more will become available soon.

My very biased impression was that R / Pharma was an unqualified success at accomplishing the major objectives of bringing together data scientists and statisticians working in the Pharmaceutical industry, and of presenting a high quality program that explored several issues relating to the production use of R in a regulatory environment.

The following chart shows that representatives from quite a few pharmaceutical companies attended in spite of organization problems that artificially limited the overall number of attendees to about 140.

Planning has already begun for R / Pharma 2019. The exact date has not yet been locked in, but I expect it will be mid-August. Please stay tuned for more information.

August 2018: Top 40 New Packages

Wed, 26 Sep 2018 00:00:00 +0000

Package developers relaxed a bit in August.; only 160 new packages went to CRAN that month. Here are my “Top 40” picks organized into seven categories: Data, Machine Learning, Science, Statistics, Time Series, Utilities, and Visualization.

Data

nsapi v0.1.1: Provides an interface to the Nederlandse Spoorwegen (Dutch Railways) API, allowing users to download current departure times, disruptions and engineering work, the station list, and travel recommendations from station to station. There is a vignette.

repec v0.1.0: Provides utilities for accessing RePEc (Research Papers in Economics) through a RESTful API. You can request an access code and get detailed information here.

rfacebookstat v1.8.3: Implements an interface to the Facebook Marketing API, allowing users to load data by campaigns, ads, ad sets, and insights.

UCSCXenaTools v0.2.4: Provides access to data sets from UCSC Xena data hubs, which are a collection of UCSC-hosted public databases.

ZipRadius v1.0.1: Generates a data frame of US zip codes and their distance to the given zip code, when given a starting zip code and a radius in miles. Also includes functions for use with choroplethrZip, which are detailed in the vignette.

Machine Learning

dials v0.0.1: Provides tools for creating model parameters that cannot be directly estimated from the data. There is a vignette.

tosca v0.1-2: Provides a framework for statistical analysis in content analysis. See the vignette for details.

tsmap v0.3.1: Implements the Matrix Profile concept for classification.

Science

DSAIRM v0.4.0: Provides a collection of Shiny apps that implement dynamical systems simulations to explore within-host immune response scenarios. See the package Tutorial.

epiflows v0.2.0: Provides functions and classes designed to handle and visualize epidemiological flows between locations, as well as a statistical method for predicting disease spread from flow data initially described in Dorigatti et al. (2017). For more information, see the RECON toolkit for outbreak analysis. There is an Overview and a vignette on Data Preparation.

fieldRS v0.1.1: Provides functions for remote-sensing field work using best practices suggested by Olofsson et al. (2014). See the vignette for details.

Rnmr1D v1.2.1: Provides functions to perform the complete processing of proton nuclear magnetic resonance spectra from the free induction decay raw data. For details see Jacob et al. (2017) and the vignette.

Statistics

bcaboot v0.2-1: Provides functions to compute bootstrap confidence intervals in an almost automatic fashion. See the vignette.

bivariate v0.2.2: Contains convenience functions for constructing and plotting bivariate probability distributions. See the vignette for details.

DesignLibrary v0.1.1: Provides a simple interface to build designs and allow users to compare performance of a given design across a range of combinations of parameters, such as effect size, sample size, and assignment probabilities. Look here for more information.

doremi v0.1.0: Provides functions to fit the dynamics of a regulated system experiencing exogenous inputs using differential equations and linear mixed-effects regressions to estimate the characteristic parameters of the equation. See the vignette.

eikosograms v0.1.1: An eikosogram (probability picture from the ancient Greek εὶκὀσ - likely or probable) divides the unit square into rectangular regions whose areas, sides, and widths represent various probabilities associated with the values of one or more categorical variates. For a discussion on the eikosogram and its superiority to Venn diagrams in teaching probability, see Cherry and Oldford (2003), and for a discussion of its value in exploring conditional independence structure and relation to graphical and log-linear models, see Oldford (2003). There is an Introduction and vignettes on Data Analysis and Independence Relations.

localIV v0.1.0: Provides functions to estimate marginal treatment effects using local instrumental variables. See Heckman et al. (2006) and Zhou and Xie (2018) for background.

merlin v0.0.1: Provides functions to fit linear, non-linear, and user-defined mixed effects regression models following the framework developed by Crowther (2017). See the vignette for details.

MRFcov v1.0.35: Provides functions to approximate node interaction parameters of Markov Random Fields graphical networks. The general methods are described in Clark et al. (2018). There are vignettes on Preparing Datasets, Gaussian and Poisson Fields, and an example using Bird parasite data.

SCPME v1.0: Provides functions to estimate a penalized precision matrix via an augmented ADMM algorithm as described in Molstad and Rothman (2018). There is a Tutorial and a vignette describing Algorithm Details.

survxai v0.2.0: Contains functions for creating a unified representation of survival models, which can be further processed by various survival explainers. There are vignettes on Local explanations, global explanations, comparing models, and on a custom prediction function.

Time Series

hpiR v0.2.0: Provides functions to compute house price indexes and series, and evaluate index goodness based on accuracy, volatility and revision statistics. For the background on model construction, see Case and Quigley (1991), and for hedonic pricing models, see Bourassa et al. (2006). There is an an introduction to the package and a vignette on Classes.

STMotif v0.1.1: Provides functions to identify motifs (previously identified sub-sequences) in spatial-time series. There are vignettes on motif discovery, examples, candidate generation, and candidate validation.

trawl v0.2.1: Contains functions for simulating and estimating integer-valued trawl processes as described in Veraart (2018), and for simulating random vectors from the bivariate negative binomial and the bi- and trivariate logarithmic series distributions. There is a vignette on trawl processes, and another on the binomial distributions.

Utilities

arkdb v0.0.3: Provides functions for exporting tables from relational database connections into compressed text files, and streaming those text files back into a database without requiring the whole table to fit in working memory. See the vignette for a tutorial.

aws.kms v0.1.2: Implements an interface to AWS Key Management Service, a cloud service for managing encryption keys. See the README for details.

DatapackageR v0.15.3: Provides a framework to help construct R data packages in a reproducible manner. It maintains data provenance by turning the data-processing scripts into package vignettes, as well as enforcing documentation and version checking of included data objects. There is a Guide to using the package, and a vignette on YAML configuration.

hedgehog v0.1: Enables users to test properties of their programs against randomly generated input, providing far superior test coverage compared to unit testing. There is a general tutorial and a description of the Hedgehog state machine.

jsonstat v0.0.2: Implements an interface to JSON-stat, a simple, lightweight ‘JSON’ format for data dissemination. There is a short quickstart quide.

nseval v0.4: Provides an API for Lazy and Non-Standard Evaluation with facilities to capture, inspect, manipulate, and create lazy values (promises), “…” lists, and active calls. See README.

runner v0.1.0: Provides running functions (windowed, rolling, cumulative) with varying window size and missing handling options for R vectors. See the vignette for details.

RTest v1.1.9.0: Provides an XML-based testing framework for automated component tests of R packages developed for a regulatory environment. There is a short vignette.

sparkbq v0.1.0: Extends sparklyr by providing integration with Google BigQuery. It supports direct import/export from/to BigQuery, as well as intermediate data extraction from Google Cloud Storage. See README.

vapour v0.1.0: Provides low-level access to GDAL, the Geospatial Data Abstraction Library. There is a vignette.

Visualization

mapdeck v0.1.0: Provides a mechanism to plot interactive maps using Mapbox GL, a JavaScript library for interactive maps, and Deck.gl, a JavaScript library which uses WebGL for visualizing large data sets. The vignette explains how to use the package.

rayshader v0.5.1: Provides functions that use a combination of raytracing, spherical texture mapping, lambertian reflectance, and ambient occlusion to produce hillshades of elevation matrices. Includes water-detection and layering functions, programmable color palette generation, built-in textures, 2D and 3D plotting options, and more. See README for details and examples.

sigmajs v0.1.1: Provides an interface to the sigma.js graph-visualization library, including animations, plugins, and shiny proxies. There is a brief Get Started Guide, and vignettes on Animation, Buttons, Coloring by Cluster, Dynamic graphs, igraph & gexf, Layout, Plugins, Settings, Shiny, and Crosstalk.

survsup v0.0.1: Implements functions to plot survival curves. The vignette provides examples.

tidybayes v1.0.1: Provides functions for composing data and extracting, manipulating, and visualizing posterior draws from Bayesian models (JAGS, Stan, rstanarm, brms, MCMCglmm, coda, …) in a tidy data format. There is a vignette on Using tidy data with Bayesian Models, and vignettes for brms and rstanarm models.

July 2018: Top 40 New Packages

Mon, 27 Aug 2018 00:00:00 +0000

July was a big month for submitting new packages to CRAN; by my count, 251 unique and truly new packages were accepted. In addition to quantity, I was pleased to see quality and variety. For instance, tropicalSparse, a package for exploring some abstract mathematics, and eChem, a package for teaching analytical chemistry, exemplify R’s expansion into new fields.

Below are my “Top 40” picks organized into ten categories: Computational Methods, Data, Econometrics, Machine Learning, Mathematics, Science, Statistics, Time Series, Utilities, and Visualization

Computational Methods

osqp v0.4.0: Provides bindings to the OSQP solver, a numerical optimization package for solving convex quadratic programs written in C based on the alternating direction method of multipliers. See Stellato et al. (2018) for details.

sundailr v0.1.1: Provides a way to call the functions in SUNDIALS C ODE solving library. There is a vignette.

Data

fredr v1.0.0: Provides an R client for the Federal Reserve Economic Data (FRED). There are vignettes on FRED Categories, Releases, Series, Sources, and Tags, as well as a Getting Started Guide.

jstor v0.3.2: Provides functions to import metadata, ngrams, and full-texts delivered by Data for Research by JSTOR. There is an Introduction, and vignettes on Automating File Import and Known Quirks.

rLandsat v0.1.0: Provides functions to search and acquire Landsat data using an API built by Development Seed and the U.S. Geological Survey. See README for how to use the package.

weathercan v0.2.7: Provides tools for downloading historical weather data from the Environment and Climate Change Canada website. Data can be downloaded from multiple stations over large date ranges, and automatically processed into a single dataset. There is an Introduction, a Glossary, and vignettes on Flags and Interpolation.

Econometrics

beezdemand v0.1.0: Provides tools to facilitate analyses performed in studies of behavioral economic demand such as data screening proposed by Stein et al.(2015), and model fitting, including linear Hursh et al. (1989), exponential Hursh & Silberberg (2008), and modified exponential Koffarnus et al. (2015) models. The vignette provides examples.

sgmodel v0.1.0: Provides functions to compute the solutions of a generic stochastic growth model for a given set of user-supplied parameters. See Merton (1971) and Tauchen (1986). There is a vignette.

Machine Learning

bigdatadist v1.0: Provides functions to compute distances between probability measures, entropy measures for samples of curves, distances and depth measures for functional data, and the Generalized Mahalanobis Kernel distance for high dimensional data. For further details see Martos et al (2014) and Martos et al (2018).

L0Learn v1.0.2: Provides an optimized toolkit for approximately solving L0-regularized learning problems. The algorithms are based on coordinate descent and local combinatorial search. For more details see Hazimeh and Mazumder (2018). There is a vignette.

Mathematics

tropicalSparse v0.1.0: Implements some basic tropical algebra functionality for sparse matrices by applying sparse matrix storage techniques. These include addition and multiplication of vectors and matrices, dot product of the vectors in tropical form, and some general equations are also solved using tropical algebra. Look here for the math.

Science

eChem v1.0.0: Provides tools for use in courses in analytical chemistry. Functions simulate cyclic voltammetry, linear-sweep voltammetry, single-pulse and double-pulse chronoamperometry, and chronocoulometry experiments using the implicit finite difference method outlined in Brown (2015). There is an Overview and vignettes on Using eChem, Computational details, and Examples.

RaceID v0.1.1: Enables inference of cell types and prediction of lineage trees using the StemID2 algorithm of Herman, Sagar and Grün D. (2018). There is a vignette.

updog v1.0.1: Implements empirical Bayes approaches to genotype polyploids from next-generation sequencing data while accounting for allelic bias, over dispersion, and sequencing error. See Gerard et al. (2018) for implementation details, along with vignettes on Oracle Calculations, Parallization, Simulating Sequeencing Data, and an Example.

Statistics

adaptMT v1.0.0: Implements adaptive p-value thresholding (AdaPT), including a framework that allows users to specify any algorithm to learn local false-discovery rate, as well as a pool of convenient functions that implement specific algorithms. See Lei and Fithian (2016). The vignette provides an introduction to the package.

biglmm v0.9-1: Provides regression for data too large to fit in memory. This package functions exactly like the biglm package, but works with later versions of R.

circumplex v0.1.2: Provides tools for analyzing and visualizing circular data, including a generalization of the bootstrapped structural summary method from Zimmermann & Wright (2017), and functions for creating publication-ready tables and figures from the results. There is an Introduction and a vignette on Analysis.

MultiFit v0.1.2: Provides functions to test for independence of two random vectors and learn and report the dependency structure. For more information, see Gorsky and Ma (2018) and the vignette.

PHEIndicatormethods v1.0.8: Provides functions to calculate commonly used public health statistics and their confidence intervals using methods approved for use in the production of Public Health England indicators, such as those presented via Fingertips. The statistical methods are referenced in the following publications: Breslow and Day](doi:10.1002/sim.4780080614), Dobson et al (1991), Armitage and Berry (2002), and Wilson (1927). There is a vignette.

robmixglm v1.0-2: Implements robust generalized linear models (GLM) using a mixture method, as described in Beath (2018). See the vignette for details.

SingelCaseES v0.4.0: Provides functions for calculating basic effect size indices for single-case designs, including several non-overlap measures and parametric effect size measures, and for estimating the gradual effects model developed by Swan and Pustejovsky (2018). There is a vignette on Definitions and Mathematical Details and another on Calculations.

spCP v1.0: Implements a spatially varying change-point model with unique intercepts, slopes, variance intercepts and slopes, and change points at each location. Inference is within the Bayesian setting using Markov chain Monte Carlo (MCMC). See the vignette for an example.

TDAstats v0.3.0: Provides a tool set for topological data analysis, specifically via the calculation of persistent homology in a Vietoris-Rips complex. For a general background on computing persistent homology for topological data analysis, see Otter et al. (2017). To learn more about how the permutation test is used for nonparametric statistical inference in topological data analysis, read Robinson & Turner (2017). There is an Introduction and a vignette on Hypothesis Testing with TDA.

trafo v1.0.0: Provides functions to estimate, select, and compare several families of transformations, including Bickel-Doksum Bickel and Doksum (1981), Box-Cox, Dual Yang (2006), Glog Durbin et al. (2002), Gpower1, Log, Log-shift opt Feng et al. (2016), Manly, Modulus John and Draper (1980), Neglog Whittaker et al. (2005), Reciprocal and Yeo-Johnson. See the vignette for the math.

uniformly v0.1.0: Provides functions to uniformly sample from various geometric shapes, such as spheres, ellipsoids, simplices. See the vignette.

Time Series

rollRegress v0.1.0: Implements methods for fast-rolling and expanding linear regression models. The methods use rank-one updates and downdates of the upper triangular matrix from a QR decomposition. See Dongarra et al.(1979). The vignette provides some details.

Utilities

anyLib v1.0.4: Provides functions to install and load a list of packages from CRAN, Bioconductor or GitHub. For GitHub, if you do not have the full path with the maintainer name in it (e.g. “achateigner/topReviGO”), it will be able to load it but not to install the package. There is a brief vignette.

dbx v0.2.1: Provides select, insert, update, upsert, and delete database operations for PostgreSQL, MySQL, SQLite, and other databases. See the README for usage

envnames v0.3.0: Provides functions to keep track of user-defined environment names that cannot be retrieved with the base R function environmentName(). The main function in this package, environment_name(), returns the name of the environment given as parameter. The vignette offers an overview of the package.

librarian v1.3.0: Provides functions to automatically install, update, and load CRAN and GitHub packages in a single function call. See README for usage.

makeParallel v0.1.1: Provides functions to automate the transformation of serial R code into more efficient parallel versions. There is a Quickstart Guide and a vignette on Parallel Concepts.

metaDigitise v1.0.0: Provides functions to extract, summarize and digitize data from published figures in research papers. The vignette shows how to use the package.

RSuite v0.32-244: Provides a set of tools to be used with the R Suite for developing data-science workflows.

Visualization

ceterisParibus v0.3.0: Provides functions to create “What if?” plots of model responses around selected points in a feature space. The four vignettes offer several examples, including a Random Forests Example and a Classification Example.

cytofan v0.1.0: Implements fan plots for cytometry data in ggplot2. See Britton et al. (1998) for information on fan plots, and README for package usage.

fingertipscharts v0.0.1: Provides tools to recreate the visualizations that are displayed on the Fingertips website of U.K. public health data. The vignette explains how to use the package.

ggvoronoi v0.8.0: Provides functions to create, manipulate and visualize Voronoi diagrams using the deldir and ggplot2 packages. The vignette shows how.

Two Big Ideas from JSM 2018

Tue, 07 Aug 2018 00:00:00 +0000

The Joint Statistical Meetings offer an astounding number of talks. It is impossible for an individual to see more than a small portion of what is going on. Even so, a diligent attendee ought to come away with more than a few good ideas. The following are two big ideas that I got from the conference.

Session 149, an invited panel on Theory versus Practice which featured an All-Star team of panelists (Edward George, Trevor Hastie, Elizaveta Levina, John Petkau, Nancy Reid, Richard J Samworth, Robert Tibshirani, Larry Wasserman and Bin Yu), covered a lot of ground and wove a rich tapestry of ideas. A persistent theme among many of the discussions was the worry that the paper publication process was undermining the quality of statistical results. Pressed to “sell” their ideas to journal editors, and constrained by publication space authors are being conditioned to emphasize the evidence for their results and neglect limitations or cases where their methods don’t perform well.

The big idea that really struck me was the notion articulated by Rob Tibshirani that simulation is the practical way to express healthy scientific skepticism that can be incorporated into both theoretical and applied papers without significantly increasing the papers length or complexity. (For the purposes of reproducibility, almost all of the simulation work can be submitted as supplementary material or stashed on a GitHub site.) For theoretical papers, authors could use simulation to examine underlying assumptions and determine which are most important, while authors of applied papers could point out cases where their methods or algorithms don’t work particularly well. Tibshirani noted that every model has its Achilles heel, and went so far as to suggest that every paper ought to have at least one table that exposes the weaknesses of a model or algorithm.

For researchers working in R, including simulations should add no additional burden as Monte Carlo simulation capabilities are built into the core of the language. (If you are new to R, you might find this brief tutorial by Corey Chivers helpful in getting started with simulating from statistical models.)

The second big idea came in Section 271, the Invited Special Presentation: Introductory Overview Lecture: Reproducibility, Efficient Workflows, and Rich Environments]. In her talk, How Computational Environments Can (Unexpectedly) Influence Statistical Findings Victoria Stodden elaborated on the idea that “As statistical research typically takes place in a constructed environment in silico, the findings may not be independent of this environment”. To help establish the pedigree of her ideas, Stodden briefly quoted David Donoho’s famous remark on a scientific paper only being the advertising for scientific work and not the scholarship itself. The paragraph surrounding this remark is illuminating. In his 2010 paper, An invitation to reproducible computational research, Donoho writes:

I was inspired more than 15 years ago by John Claerbout, an earth scientist at Stanford, to begin practicing reproducible computational science. See Claerbout and Karrenbach (1992). He pointed out to me, in a way paraphrased in Buckheit and Donoho (1995): “an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result.” This struck me as getting to the heart of a central problem in modern scientific publication. Most of the work in a modern research project is hidden in computational scripts that go to produce the reported results. If these scripts are not brought out into the open, no one really knows what was done in a certain project…

Here we have an assertion of the essential scientific value of software and code in a thread that traces the need for reproducible research back quite a few years to a collaboration between scientists and statisticians.

Immediately following Stodden, in his talk on Living a Reproducible Life Hadley Wickham gave a virtuoso presentation on using modern, R centric reproducible tools. (He even managed to rebase a GitHub repo without calling attention to it).

My main “takeaways” from these two talks were, first of all, an affirmation that the CRAN and Bioconductor repositories are themselves extremely valuable contributions to statistics. Not only do they enable the daily practice of statistics for many statisticians, by providing reference implementations (and documentation) for a vast number of models and algorithms they are the repositories of statistical knowledge.

The second takeaway is that reproducible research, long acknowledged to be essential to the scientific process, is now feasible for a large number of practitioners. Using coding tools such as R Markdown along with infrastructure such as GitHub, it is possible to develop reproducible workflows for significant portions of a research process. R-centric reproducible tools are helping to put the science in data science.

Note that both Victoria Stodden and Rob Tibshirani, along with R Core member Michael Lawrence, will be delivering keynote presentations at the inaugural R / Medicine conference coming up September 7th and 8th in New Haven, CT.

June 2018: Top 40 New Packages

Sun, 29 Jul 2018 00:00:00 +0000

Approximately 144 new packages stuck to CRAN in June. That fact that 31 of these are specialized to particular scientific disciplines or analyses provides some evidence to my hypothesis that working scientists are actively adopting R. Below are my Top 40 picks for June, organized into the categories of Computational Methods, Data, Data Science, Economics, Science, Statistics, Time Series, Utilities and Visualizations. The Data packages, especially rtrek and opensensmapr, look like they have some interesting new data to explore.

Computational Methods

nnTensor v0.99.1: Provides methods for n-negative matrix factorization and decomposition. See Cichock et al (2009) for details.

RcppEigenAD v1.0.0: Provides functions to compile C++ code using Rcpp, Eigen, and CppAD to produce first- and second-order partial derivatives, and also provides an implementation of Faa’ di Bruno’s formula to combine the partial derivatives of composed functions. See Hardy (2006).

rcrane v1.0: Provides optimization algorithms to estimate coefficients in models such as linear regression and neural networks. Includes batch gradient descent, stochastic gradient descent, minibatch gradient descent, and coordinate descent. See [Kiwiel, (2001)](doi:10.1007/PL00011414, Yu Nesterov (2004), Ferguson (1982), Zeiler (2012), and Wright (2015). The vignette introduces the package.

Data

bjscrapeR v0.1.0: Scrapes crime data from the National Crime Victimization Survey, which tracks personal and household crime in the USA.

genesysr v0.9.1: Implements an API to access data on plant genetic resources from genebanks around the world published on Genesys. The vignette offers a short tutorial.

opensensmapr v0.4.1: Allows users to download real-time environmental measurements and sensor station metadata from the OpenSenseMap API. There are vignettes for Visualization, Exploration, and Caching Data for Reproducibility.

readabs v0.2.1: Provides functions to read Excel files from the Australian Bureau of Statistics into Tidy Data Sets. See the vignette.

rppo v1.0: Implements an interface to the Global Plant Phenology Data Portal. See the vignette.

rtrek v0.1.0: Provides datasets related to the Star Trek fictional universe, functions for working with the data, and access to real-world datasets based on the televised series and other related licensed media productions. It interfaces with Wikipedia, the Star Trek API (STAPI), Memory Alpha, and Memory Beta to retrieve data, metadata, and other information relating to Star Trek. See the README for usage information.

skynet v1.2.2: Implements methods for generating air transport statistics based on publicly available data from the U.S. Bureau of Transport Statistics (BTS). See the vignette.

Data Science

AdaSampling v1.1: Implements the adaptive sampling procedure, a framework for both positive unlabeled learning and learning with class label noise. See Yang et al. (2018) and the vignette.

AROC v1.0: Provides functions to estimate the covariate-adjusted Receiver Operating Characteristic (AROC) curve and pooled (unadjusted) ROC curve. See de Carvalho and Rodriguez-Alvarez (2018).

cloudml v0.5.1: Provides an interface to the Google Cloud Machine Learning Platform. There is a Getting Sarted Guide and vignettes on Deploying Models, Cloud storage, Training, and Hyperparameter Tuning.

reclin v0.1.0: Provide functions to assist in performing probabilistic record linkage and deduplication: generating pairs, comparing records, em-algorithm for estimating m- and u-probabilities, forcing one-to-one matching. There is an Introduction and a vignette on Duplication.

vip v0.1.0: Provides a general framework for constructing variable importance plots from various types of machine learning models, based on a novel approach using partial dependence plots and individual conditional expectation curves as described in Greenwell et al. (2018). See the README for details and examples.

wevid v0.4.2: Provides functions to quantify the performance of a binary classifier through weight of evidence. These can be used with any test dataset on which you have observed case-control status, and have computed prior and posterior probabilities of case status using a model learned on a training dataset. Look at this website for details and examples.

Economics

trade v0.5.3: Provides tools for working with trade model, including the ability to calibrate different consumer-demand systems and simulate the effects of tariffs and quotas under different competitive regimes. The vignette provides details.

Science

linpk v1.0: Provides functions and a shiny application to generate concentration-time profiles from linear pharmacokinetic (PK) systems. Single or multiple doses may be specified. The vignette offers details and examples.

ratematrix v1.0: Provides functions to estimate the evolutionary rate matrix ® using Markov chain Monte Carlo (MCMC), as described in Caetano and Harmon (2017). There is a vignette on Setting a custom starting point and another on Using prior distributions.

spectralAnalysis v3.12.0: Provides a toolkit for spectral-analysis, enabling users to pre-process, visualize, and analyse process analytical dat, by spectral data measurements made during a chemical process.

Statistics

betaboost v1.0.1: Implements boosting beta regression for potentially high-dimensional data Mayr et al. (2018) using the same parametrization as betareg Cribari-Neto and Zeileis (2010). The underlying algorithms are implemented via the R add-on packages mboost Hofner et al. (2014) and gamboostLSS Mayr et al. (2012). The vignette offers examples.

bfw v0.1.0: Provides a framework for conducting Bayesian analysis using Markov chain Monte Carlo with the JAGS sampler. There are vignettes on Fitting Latent Data, Fitting Observed Data, the Predict Metric, Plotting, and Regression.

CaseBasedReasoning v0.1: Given a large set of problems and their individual solutions, case-based reasoning seeks to solve a new problem by referring to the solution of that problem that is “most similar” to the new problem. See Dippon et al. (2002), the vignette on Motivation, and examples of case-based reasoning with a Cox-Beta Model and a Random Forest Model.

coxed v0.1.1: Provides functions for generating, simulating, and visualizing expected durations and marginal changes in duration from the Cox proportional hazards model. There is a vignette on using the coxed() function and another on simulating survival data.

GLMMadaptive v0.2-0: Provides functions to fit generalized linear mixed models for a single grouping factor under maximum likelihood approximating the integrals over the random effects with an adaptive Gaussian quadrature rule. See Pinheiro and Bates (1995) and the vignettes on Custom Models, GLMMadaptive Basics, Methods for MixMod Objects, and Zero-Inflated and Two-Part Mixed Effects Models.

glmmfields v0.1.0: Implements generalized linear mixed models with robust random fields for spatiotemporal modeling. The vignette provides examples.

kendallRandomWalks v0.9.3: Provides functions for simulating Kendall random walks, continuous-space Markov chains generated by the Kendall generalized convolution. See Jasiulis-Gołdyn (2014) for details and the vignettes Kendall Random Walks and Studying the Behavior of Kendall Random Walks.

netSEM v0.5.0: Provides functions for structural equation modeling. There is an Introduction and vignettes on Backsheet Degradation, Backsheet Cracking, Current Voltage Features, and Modeling of the Weathering Driven Degradation of Poly(ethylene-terephthalate) Films.

umap v0.1.0.3: Implements the uniform manifold approximation and projection technique for dimension reduction as described in McInnes and Healy (2018). The vignette shows how to use the package.

vimp v1.0.0:Provides functions to calculate point estimates of, and valid confidence intervals for, non-parametric variable importance measures in high and low dimensions. For information about the methods, see Williamson et al. (2017). The vignette contains an introduction to the package.

vsgoftest v0.3-2: Implements Vasicek and Song goodness-of-fit tests (based on Kullbach-Leibler divergence) for a family of distributions that include uniform, Gaussian, log-normal, exponential, gamma, Weibull, Pareto, Fisher, Laplace, and beta distributions. See Lequesne and Regnault (2018) for details and the Tutorial.

Time Series

anomaly v1.0.0: Implements the CAPA (Collective And Point Anomaly) algorithm of Fisch, Eckley and Fearnhead (2018) for the detection of anomalies in time series data.

exuber v0.1.0: Provides functions for testing and dating periods of explosive dynamics (exuberance) in time series using recursive unit root tests as proposed by Phillips et al. (2015). See the README to get started.

Simulate a variety of periodically-collapsing bubble models. The estimation and simulation utilizes the matrix inversion lemma from the recursive least squares algorithm, which results in a significant speed improvement.

Utilities

BiocManager v1.30.1: Implements a tool to install and update Bioconductor packages. The vignette shows how to use the package.

IntervalSurgeon v1.0: Provides functions for manipulating integer-bounded intervals including finding overlaps, piling, and merging. The vignette shows how to use the package.

pkgbuild v1.0.0: Provides functions used to build R packages. Locates compilers needed to build R packages on various platforms and ensures the PATH is configured appropriately.

rqdatatable v0.1.2: Implements the rquery piped query algebra using data.table. There is a vignette on Grouped Sampling and a Logistic Example.

ssh v0.2: Provides functions to connect to a remote server over SSH to transfer files via SCP, setup a secure tunnel, or run a command or script on the host while streaming stdout and stderr directly to the client. There is a vignette.

Visualization

mgcViz v0.1.1: An extension of the mgcv package, providing visual tools for Generalized Additive Models (GAMs) that exploit the additive structure of GAMs, scale to large data sets, and can be used in conjunction with a wide range of response distributions. See the vignette for examples.

tiler v0.2.0: Provides functions to create geographic map tiles from geospatial map files or non-geographic map tiles from simple image files. The vignette provides an introduction.

May 2018: “Top 40” New Packages

Tue, 26 Jun 2018 00:00:00 +0000

While looking over the 215 or so new packages that made it to CRAN in May, I was delighted to find several packages devoted to subjects a little bit out of the ordinary; for instance, bioacoustics analyzes audio recordings, freegroup looks at some abstract mathematics, RQEntangle computes quantum entanglement, stemmatology analyzes textual musical traditions, and treedater estimates clock rates for evolutionary models. I take this as evidence that R is expanding beyond its traditional strongholds of statistics and finance as people in other fields with serious analytic and computational requirements become familiar with the language. And, when I see a package from a philologist and scholar of “Ancient and Medieval Worlds”, I am persuaded to think that R is making a unique contribution to computational literacy.

Below are my “Top 40” package picks for May 2018, organized into the following categories: Computational Methods, Data, Data Science, Finance, Mathematics, Music, Science, Statistics, Time Series, Utilities and Visualization.

Computational Methods

dqrng v0.0.4: Provides fast random number generators with good statistical properties, including the 64-bit variant of the Mersenne-Twister, pcg64, and Xoroshiro128 and Xoroshiro256. There is an Introduction and a vignette on Parallel Usage.

optimParallel v.7-2: Provides a parallel versions of the gradient-based stats::optim() function. The vignette is informative.

Data

childesr v0.1.0: Implements an interface to CHILDES, an open repository for transcripts of parent-child interaction. There is a vignette.

PetfindeR v1.1.3: Is a wrapper of the Petfinder API that implements methods for interacting with and extracting data from the Petfinder database, one of the largest online, searchable databases of adoptable animals and animal welfare organizations across North America. See the Getting Started Guide: part1 and part2.

Data Science

catch v1.0: Provides functions to perform classification and variable selection on high-dimensional tensors (multi-dimensional arrays) after adjusting for additional covariates (scalar or vectors) as CATCH model in Pan, Mai and Zhang (2018).

SemiSupervised v1.0: Implements several safe graph-based semi-supervised learning algorithms. For technical details, refer to Culp and Ryan (2013), Ryan and Culp (2015) and the package vignette.

spFSR v1.0.0: Offers functions to perform feature selection and ranking via simultaneous perturbation stochastic approximation (SPSA-FSR) based on works by Aksakalli and Malekipirbazari (2015) and Yenice et al. (2018). See the Introduction.

Finance

PortfolioAnalytics v1.1.0: Provides functions for portfolio analysis, including numerical methods for portfolio optimization. There is an Introduction and vignettes on Optimization, Custom Moment and Objective Functions, and Portfolio Optimization with CVaR Budgets.

sparseIndexTracking v0.1.0: Provides functions to compute sparse portfolios for financial index tracking, i.e., joint selection of a subset of the assets that compose the index and computation of their relative weights (capital allocation) based on the paper Benidis et al. (2018). The vignette shows how to design a portfolio to track an index.

Mathematics

freegroup v1.0: Provides functions to elements of the free group, including inversion, multiplication by a scalar, group-theoretic power operation, and Tietze forms. See the vignette for details.

ODEsensitivity v1.1.1: Provides functions to perform sensitivity analysis for ordinary differential equation (ode) models using the ODEnetwork package, which simulates a network of second-order ODEs. See [Weber et al. (2018)(https://eldorado.tu-dortmund.de/handle/2003/36875)] for details, and the vignette to get started.

Music

stemmatoloty v0.3.1: Allows users to explore and analyze the genealogy of textual or musical traditions from their variants, with various stemmatological methods, mainly the disagreement-based algorithms suggested by Camps and Cafiero (2015). The vignette provides details.

Science

bioacoustics v0.1.2: Provides functions to analyze audio recordings and automatically extract animal vocalizations. Contains all the necessary tools to process audio recordings of various formats (e.g., WAV, WAC, MP3, ZC), filter noisy files, display audio signals, and detect and extract automatically acoustic features for further analysis such as classification. The vignette provides an example.

epiphy v0.3.4: Provides a toolbox for analyzing plant disease epidemics and a common framework for plant disease intensity data recorded over time and/or space. There is a vignette on Definitions and Relationships between Parameters and another with Examples.

treedater v0.2.0: Offers functions for estimating times of common ancestry and molecular clock rates of evolution using a variety of evolutionary models. See Volz and Frost (2017). The vignette provides an example using the Influenza H3N2 virus.

RQEntangle v0.1.0: Provides functions to compute the Schmidt decomposition of bipartite quantum systems, discrete or continuous, and their respective entanglement metrics. See Ekert and Knight (1995) and the vignettes Entanglement in Coupled Harmonics and Entanglement between Two Coupled Two-Level Systems for details.

Statistics

glmmboot v0.1.2: Provides two functions to perform bootstrap resampling for most models that update() works for. BootGlmm() performs block resampling if random effects are present, and case resampling if not; BootCI() converts output from bootstrap model runs into confidence intervals and p-values. See Humphrey and Swingley (2018) for the details and the vignette to get started.

glmmEP v1.0-1: Allows users to solve Generalized Linear Mixed Model Analysis via Expectation Propagation. In this version, the random effects can be any reasonable dimension. However, only probit mixed models with one level of nesting are supported. See the methodology in Hall et al. (2018), and the user manual in the vignette.

groupedSurv v1.0.1: Provides Rcpp-based functions to compute the efficient score statistics for grouped time-to-event data (Prentice and Gloeckler (1978)), with the optional inclusion of baseline covariates. The vignette gives an example.

modeldb v0.1.0: Provides functions to fit models inside databases with dplyr backends. There are vignettes showing how to implement linear regression and kmeans models.

MLZ v0.1.1: Provides estimation functions and diagnostic tools for mean length-based total mortality estimators based on Gedamke and Hoenig (2006). There is a vignette.

NFWdist v0.1.0: Provides density, distribution function, quantile function, and random generation for the 3D Navarro, Frenk & White (NFW) profile. For details see Robotham & Howlett (2018) and the vignette.

survBootOutliers v1.0: Offers three new concordance-based methods for outlier detection in a survival context. The methodology is described in two papers by Pinto J., Carvalho A. and Vinga S.: paper1 and paper2. The vignette provides an introduction.

vinereg v0.3.0: Implements D-vine quantile regression models with parametric or non-parametric pair-copulas. See Kraus and Czado (2017) and Schallhorn et al. (2017). There is a vignette showing how to use the package, and another covering an Analysis of bike rental data.

Time Series

ASSA v1.0: Provides functions to model and decompose time series into principal components using singular spectrum analysis. See de Carvalho and Rua (2017) and de Carvalho et al (2012).

DTWBI v1.0: Provides functions to impute large gaps within time series based on Dynamic Time Warping methods. See Phan et al. (2017). The website has examples.

permutes v0.1: Uses permutation testing (Maris & Oostenveld (2007)) to help determine optimal windows for analyzing densely-sampled time series. The vignette provides details.

Utilities

bench v1.0.1: Provides tools to benchmark and analyze execution times for R expressions.

conflicted v0.1.0: Provides an alternative to R’s default conflict-resolution strategy for R packages.

diffdf v1.0.0: Provides functions for comparing two data frames. The vignette describes how to use the package.

pkgdown v1.1.0: Provides functions to generate a website from a source package by converting your documentation, vignettes, README, and more to HTML. See the vignette.

rtika v0.1.8: Provides functions to extract text or metadata from over a thousand file types, using Apache Tika to produce either plain-text or structured XHTML content. See the vignette.

tabulizer v0.2.2: Provides bindings for the Tabula Java library, which can extract tables from PDF documents. The tabulizerjars packages provides versioned .jar files. There is an Introduction.

shinytest v1.3.0: Enables automated testing of Shiny applications, using a headless browser.

vcr v0.1.0: A port of the Ruby gem of the same name, vcr enables users to record test suite HTTP requests and replay them during future runs. There is an Introduction and vignettes on Configuration and Request Matching

Visualization

c3 v0.2.0: Implements a wrapper (htmlwidget) for the C3.js charting library that includes all types of C3.js plots, enabling interactive web-based charts to be embedded in R Markdown documents and Shiny applications. The vignette shows basic usage.

chromoMap v0.1: Provides interactive, configurable graphics visualizations of human chromosomes, allowing users to map chromosome elements (like genes, SNPs, etc.) on the chromosome plot, and introduces a special plot, the “chromosome heatmap”, which enables visualizing data associated with chromosome elements. See the vignette.

ExPanDaR v0.2.0: Provides a shiny-based front end and a set of functions for exploratory panel data analysis. Run as a web-based app, it enables users to assess the robustness of empirical evidence without providing them access to the underlying data. There is a vignette on using the package and another on panel data exploration.

ggdistribute v1.0.1: Extends ggplot2 for plotting posterior or other types of unimodal distributions that require overlaying information about a distribution’s intervals. The vignette provides examples.

r2d3 v0.2.2: Provides a suite of tools for using the D3 library to produce dynamic, interactive data visualizations. There are vignettes on Advanced Rendering with Callbacks, R to D3 Data Conversion, CSS & JavaScript Dependencies, Learning D3, Package Development, and D3 Visualization Options.

April 2018: “Top 40” New Packages

Thu, 24 May 2018 00:00:00 +0000

Below are my “Top 40” picks from the approximately 212 new packages that made it to CRAN in April. They are organized into ten categories: Computational Methods, Data, Data Science, Machine Learning, Music, Science, Statistics, Time Series, Utilities, and Visualizations.

Computational Methods

diffeqr v0.1.1: Provides an interface to DifferentialEquations.jl which offers high performance methods for solving ordinary differential equations (ODE), stochastic differential equations (SDE), delay differential equations (DDE), differential-algebraic equations (DAE), and more. There are vignettes for solving differential algebraic equations, delay differential equations, ordinary differential equations, and stochastic differential equations.

SimRepeat v0.1.0: Provides functions to simulate correlated systems of statistical equations with multiple variable types. There is a vignette describing the underlying theory, as well as vignettes on systems with Continuous variables, multiple variable types, and hierarchical linear models.

Data

echor v0.1.0: Implements an interface to the United States Environmental Protection Agency (EPA) Environmental Compliance History Online ECHO API. The vignette shows how to use the package.

essurvey v1.0.0: Provides an interface to download data from the European Social Survey. There is a vignette.

malariaAtlas v0.0.1: Provides a suite of tools to allow downloading all publicly available parasite rate survey points and raster surfaces from the Malaria Atlas Project servers as well as utility functions for plotting the downloaded data. The vignette shows how to use the package.

salesforcer v0.1.2: Implements the Salesforce Platform APIs (REST, SOAP, Bulk 1.0, Bulk 2.0, and Metadata). There is a Getting Started guide, a vignette on Transitioning from RForecom and another on Working with Bulk API.

Data Science

mleap v0.1.1: Provides a sparklyr extension to MLeap, an open source library that enables exporting and serving Apache Spark pipelines.

rdfp v0.1.2: Implements Google’s DoubleClick for Publishers (DFP) API. There are vignettes on Ad trafficing setup, administrative tasks, checking availability, manipulating orders and line items, and reporting.

workflowr v1.0.1: Implements a framework for reproducible and collaborative data science. There is a getting started guide, and vignettes on customizing a website, migrating projects, package details, and frequently asked questions.

Machine Learning

aws.translate v0.1.3: Implements a client for AWS Translate, a machine translation service that will convert a text input in one language into a text output in another.

CBDA v1.0.0: Implements an ensemble predictor, a ‘SuperLearner’, using knockoff filtering and concepts from comprehensive sensing to classify Big Data. See the Guide to Compressive Big Data Analytics (CBDA).

featurefinder v1.0: Implements a method to find model features through a detailed analysis of model residuals using rpart classification and regression trees. See Chatterjee and Hadi (2006) and the vignette.

iilassso v0.0.1: Implements efficient algorithms for fitting linear / logistic regression models with an independently interpretable Lasso. See Takada et al. (2018) and the vignette.

live v1.5.4: Implements a methodology for interpreting a complex machine learning model by understanding the key factors. This is achieved through local approximations based on additive regression or CART like models that allow for higher interactions. See Ribeiro et al. (2016) and Staniak and Biecek (2018) for details. The vignette offers a case study.

mlrCPO v0.3.3: Provides a tool set that enriches the mlr with a diverse set of composable preprocessing operators (CPOs). There are several vignettes including First Steps, the mlrCPO Core, Built-in CPOs, Custom CPOs and more.

varrank v0.1: Provides a computational toolbox of heuristic approaches for performing variable ranking and feature selection based on mutual information adapted for multivariate system epidemiology datasets. The core function is a general implementation of the minimum redundancy maximum relevance model: Battiti (1994). The vignette is informative.

Music

tabr v0.2.0: Provides functions to creates guitar tablature from R code using LilyPond, open source music engraving software for generating high quality sheet music based on markup language. tabr also offers nominal MIDI file support. There are several vignettes: Basic example, Chords, Engraving, Phrase Helpers, Musical Phrases, Repeats, Tracks and Scores, and Non-standard Tuning.

Science

DrInsight v0.1.1: Implements a connectivity mapping-based drug repurposing tool that identifies drugs that can potentially reverse query disease phenotype or have similar functions with query drugs. The vignette provides an example.

Statistics

cultevo v1.0.2: Provides tools for measuring the composition of signalling systems: in particular, the information-theoretic measure due to Spike (2016) and the Mantel test for distance matrix correlation Dietz (1983) as well as an implementation of the Page test for monotonicity of ranks Page (1963). The vignette provides and example.

indirect v0.2.0: Provides functions to facilitate prior elicitation for Bayesian generalized linear models using independent conditional means priors. Various methodologies for eliciting fractiles are supported, including versions of the approach of Hosack et al. (2017). The vignette describes the methodology.

konfound v0.1.0: Implements methods that reflect recent developments in sensitivity analysis including functions to calculate robustness indices. See Frank (2000) and Frank et al. (2013) for details and the vignette for an introduction to the package.

mgc v1.0.1: Implements Multiscale Graph Correlation (MGC), a framework developed by Shen et al. (2017), that extends global correlation procedures to be multiscale. There are vignettes on the Discriminability statistic, the mgc statistic, and Canonical Dependence Simulations.

netregR v0.3.1: Provides functions to regress network responses (both directed and undirected) onto covariates of interest that may be actor, relation, or network-valued, and compute principled variance estimates of the coefficients. For details see Marrs et al. (2017) and the vignette.

RNGforGPD v1.0: Provides functions to generate univariate and multivariate data that follow the generalized Poisson distribution. See Demirtas (2017) for the details, and the vignette for an example.

SmartEDA v0.2.0: Provides functions that automatically describe the structure and relationships in data to facilitate exploratory data analysis. There is an Introduction to the package and a vignette on building custom tables.

stratifyR 1.0-1: Implements stratified sampling designs for univariate populations using the method of Khan et al. (2002), Khan et al. (2008) and Khan et al. (2015). The vignette goes through the math.

WeMix v1.0.1: Provides functions to run mixed-effects models that include weights at every level. Models are fit using adaptive quadrature following the methodology of Rabe-Hesketh et. al (2006). There is a vignette.

Time Series

anomalize v0.1.1: Implements a “tidy” workflow for detecting anomalies in time series data. Functions decompose time series, detect anomalies, and create bands separating the “normal” data from the anomalous data for multiple time series. There is a Quick Start Guide and a vignette on Methods.

LPWC v0.99.2: Provides functions to compute a time series distance measure for clustering based on weighted correlation and introduction of lags. See Chandereng and Gitter (2018) and the vignette.

Utilities

table1 v1.0: Provides functions to create HTML tables of descriptive statistics, as one would expect to see as the first table (i.e. “Table 1”) in a medical/epidemiological journal article. There is a vignette that shows how to create a table of descriptive statistics.

foolbox v0.1.0: Provides a framework for manipulating functions and translating them with metaprogramming. There is a tutorial and vignettes on Partial Evaluation and Transforming Functions.

graphframes v0.1.0: Implements a sparklyr extension that interfaces to GraphFrames, an Apache Spark package that provides a DataFrame-based API for working with graphs. See README for some details.

komaletter v0.2.0: Offers a versatile R Markdown template for writing letters, using the KOMA-Script letter class scrlttr2 and an adaptation of the pandoc-letter template. There is an Introduction as well as three letter templates: letter1, letter2, and letter3.

pkgnet v0.2.0: Uses tools from graph theory to analyze the dependencies between functions in an R package and between its imported packages and quantify their complexity and vulnerability to failure. There is a vignette.

promises v1.0.1: Provides fundamental abstractions for doing asynchronous programming in R using promises. There are several vignettes: 1. Why use promises?, 2. An informal introduction to async programming, 3. Working with promises in R, 4. Launching tasks with future, 5. Using promises with Shiny, and 6. Combining promises.

Visualization

ERSA v0.1.0: Provides functions for displaying the results of a regression calculation, packaged together as a shiny app. See the Vignette

ggplotify v0.0.2: Provides the means to convert plot function calls to grob or ggplot enabling users to align plots produced by base graphics, grid, lattice and vcd functions. See the vignette.

ggstatsplot v0.0.3: Extends ggplot2 to produce graphics with details from statistical tests. There are vignettes for producing publication ready violin plots, correlation plots, histograms, pie charts, scatter plots as well as vignettes describing the default theme and showing how to combine plots.

panelView v1.0.1: Provides functions to visualize panel data with dichotomous treatments. Look here for details.

prepplot v0.7: Contains a single function to prepare a figure region for base graphics that makes it easy to produce customized graphics and produce ggplot2 like plots. The vignette explains how.

2018 R Conferences

Fri, 11 May 2018 00:00:00 +0000

rstudio::conf 2018 and the New York R Conference are both behind us, but we are rushing headlong into the season for conferences focused on the R Language and its applications.

The European R Users Meeting (eRum) begins this coming Monday, May 14th, in Budapest with three days of workshops and talks. Headlined by R Core member Martin Mächler and fellow keynote speakers Achim Zeileis, Nathalie Villa-Vialaneix, Stefano Maria Iacus, and Roger Bivand, the program features an outstanding array of accomplished speakers including RStudio’s own Barbara Borges Ribeiro, Andrie de Vries, and Lionel Henry.

Second only to useR! in longevity, the tenth consecutive R / Finance will be held in Chicago on June 1st and 2nd. Keynote speakers Norm Matloff, J.J. Allaire, and Li Deng head a strong program. Produced by the same committed crew of Chicago quants with the unwavering support of UIC, R / Finance is the epitome of a small, tightly focused, single-track R conference. If you are interested in the quantitative side of Finance, there is no better place to network.

The relatively new CascadiaRConf will feature keynote speakers Alison Hill and Kara Woo in a one-day event on June 2nd in Portland, OR that promises to be good time with several hands-on workshops.

A SatRday mini-conference will be held in Cardiff on June 23rd. Stephanie Locke, Heather Turner, and Maelle Salmon will be leading the event. The recent conference in Capetown appears to have been a great day for working with R, and a lot of fun. I expect that Cardiff will also be a blast.

Why R? July 2nd through 5th in Wrocław, Poland is an ambitious undertaking with five keynote speakers (Bernd Bischl, Thomas Petzoldt, Leon Eyrich Jessen, Tomasz Niedzielski, and Maciej Eder), a hackathon, and several “pre-meetings” spread across Poland, Germany, and Denmark. I expect this to be a top-tier series of events.

R in Montreal will be held from July 4th through 6th. Pleanary speakers Julie Josse, Arun Srinivasan, and Daniel Stubbs will headline the program.

The 14th useR! conference, the first to happen in the Southern Hemisphere, will be held in Brisbane, Australia from July 10th through the 13th. The mother of all R conferences, useR! attracts R afficianados from around the globe and provides a window to what is au courant in the R universe. Keynote speakers Jenny Bryan, Steph De Silva, Heike Hofmann, Thomas Lin Pedersen, Roger Peng, and Bill Venables head the program. The tutorials, always a major attraction at useR! conferences, will take place over two days.

Insurance Data Science, the direct successor to the series of R in Insurance conference will be held in London on July 16th. Although renamed, and presumably refocused, the program for the conference still indicates quite a bit of R content. Garth Peters and Eric Novic will deliver the keynotes.

BioC 2018, the flagship conference for the BioConductor project and a major event in the computational genomics world, will be held from July 25 through the 27th at Victoria University, Toronto. The program is still coming together, but confirmed speakers include Brenda Andrews, Benjamine Haibe-Kains, Elana Fertig, Charlotte Sonneson, Michael Hoffman, and Tim Hughes.

The Latin American R/BioConductor Developers Workshop will be held between July 30th and August 3rd at the center for Genomic Sciences in Cuernavaca, Mexico. Invited speakers include Martin Morgan and Heather Turner. The workshop is aimed at students and researchers, with a goal of teaching participants the principles of reproducible data science through the development of R/Bioconductor packages.

Two brand-new conferences directly modeled on the R / Finance experience will make their debuts this year. R / Pharma, a conference devoted to the use of R for reproducible research, regulatory compliance and validation, safety monitoring, clinical trials, drug discovery, R&D, PK/PD/pharmacometrics, genomics, and diagnostics in the pharmaceutical industry will be held on August 15th and 16th at Harvard University. This will be a small, collegial gathering limited to 150 attendees; it will undoubtedly sell out soon after registration opens.

R / Medicine, which will focus on the use of R in medical research and clinical practice, with talks addressing Phase I clinical trial design; the analysis and visualization of clinical trial data, patient records, and genetic data; personalized medicine; and reproducible research, will take place in New Haven, CT on September 7th and 8th. This will also be a small gathering that is likely to sell out soon after registration opens.

LatinR, which will focus on the use of R in R&D, will be held at th University of Palermo in Buenos Aires on September 4th and 5th. Keynote speakers Jenny Bryan and Walter Sosa Escudero will head the program.

The last R conference on my radar for the 2018 season, the enterprise-focused EARL (Enterprise Applications of the R Language) Conference, will take place in London from September 11th through the 13th. Edwin Dunn and Garrett Grolemund will deliver the keynotes, and the list of speakers comprises an impressive roster of industrial-strength R users. This is clearly the event for data scientists looking to put R into production.

March 2018: "Top 40" New Package Picks

Mon, 30 Apr 2018 00:00:00 +0000

By my count, just over 200 new packages made it to CRAN and stuck during March. The trend for specialized, and sometimes downright esoteric science packages continues. I counted 40 new packages in this class. Most, but not all of these, are focused on bio-science applications. For example, the foreSIGHT package profiled below focuses on climate science. I was also pleased to see two new packages (not from RStudio) in the Data Science category, h2o4gpu and onnx, built on the reticulate package for interfacing with Python. I hope this also becomes a trend.

The following are my “Top 40” picks for March in nine categories: Computational Methods, Data, Data Science, Political Science, Science, Statistics, Time Series, Utilities and Visualizations.

Computational Methods

dynprog v0.1.0: Implements a domain-specific language for specifying translating recursions into dynamic-programming algorithms.

fmlogcondens v1.0.2: Implements a fast solver for the maximum likelihood estimator of the family of multivariate log-concave probability function. Includes well-known parametric densities including the normal, uniform, and exponential distributions and many more. For details, see Rathke et al. (2015). The vignette shows how to use the package.

knor v0.0-5: Provides access to knor, a NUMA-optimized, in-memory, distributed library for computing k-means.

Data

daymetr v1.3.1: Provides programmatic interface to the Daymet climate data. The vignette shows how to use it.

NOAAWeather v0.1.0: Provides functions to retrieve real-time weather data from all NOAA stations, and plot time series, boxplot, calendar heatmap, and geospatial maps to analyze trends. The vignette shows how to use the package.

ppitables v0.1.2: Contains country-specific lookup data tables used as reference to determine the poverty likelihood of a household based on their PPI score (Poverty Probability Index), with documentation from Innovations for Poverty Action.

usfertilizer v0.1.5: Provides county-level estimates of fertilizer, nitrogen and phosphorus, from 1945 to 2012 in the United States of America. There is an Introduction and a vignette on Data Scources and Processes.

Data Science

greybox v0.2.0: Implements tools for model selection and combinations via information criteria based on the values of partial correlations. The vignette provides details.

h2o4gpu v0.2.0: Implements an interface to H2O4GPU, a collection of GPU solvers for machine learning algorithms. There is a vignette.

iml v0.3.0: Provides interpretability methods to analyze the behavior and predictions of any machine learning model, including feature importance, partial dependence plots, [individual conditional expectation (ice plots), local models, the Shapley Value, and tree surrogate models.

iTOP v1.0.1: Provides functions to infer a topology of relationships between different datasets, such as multi-omics and phenotypic data recorded on the same samples. The methodology is based on the extension of the RV coefficient, a measure of matrix correlation to partial matrix correlations and binary data. See Aben et al. (2018) for details and the vignette introduction to the package.

onnx v0.0.1: Implements an interface to ONNX, the Open Neural Network Exchange, which provides an open-source format for machine-learning models.

rcqp v0.5: Implements Corpus Query Protocol functions based on the CWB software, a collection of open-source tools for managing and querying large text corpora. The vignette provides a roadmap.

Political Science

coalitions v0.6.2: Implements an MCMC method to calculate probabilities for a coalition majority based on survey results. See Bender and Bauer (2018). There are vignettes on Workflows, Pooling, and Diagnostics.

Science

diagmeta v0.2-0: Implements methods by Steinhauser et al. (2016) for meta-analysis of diagnostic accuracy studies with several cutpoints.

NetworkExtinction v0.1.0: Provides functions to simulate the extinction of species in the food web, and analyze the cascading effects as described in Dunne et al. (2002). There is a vignette.

foreSIGHT v0.9.2: Provides a tool to create hydroclimate scenarios, stress test systems, and visualize system performance in scenario-neutral climate-change impact assessments. Functions generate perturbed time series using a range of approaches, including simple scaling of observed time series (Culley et al. (2016)) and stochastic simulation of perturbed time series. (Guo et al. (2018)). The vignette offers a tutorial.

PINSPlus v1.0.0: Implements PINS: Perturbation clustering for data INtegration and disease Subtyping Nguyen et al. (2017), a novel approach for integration of data and classification of diseases into various subtypes There is a vignette.

Statistics

chandwich v1.0.0: Provides functions to adjustment user-supplied independence loglikelihood functions using a robust sandwich estimator of the parameter covariance matrix, based on the methodology in Chandler and Bate (2007). The vignette shows how it works.

ciuupi v1.0.0: Provides functions to compute a confidence interval for a specified linear combination of regression parameters in a linear regression model with iid normal errors and known variance, when there is uncertain prior information that a distinct specified linear combination of the regression parameters takes a given value. See Kabaila and Mainzer (2017) and the vignette for details.

CoxPhLb v1.0.0: Provides functions to analyze right-censored, length-biased data using Cox model, including model fitting and checking, and the stationarity assumption test. The model fitting and checking methods are described in Qin and Shen (2010) and Lee, Ning, and Shen (2018).

cutpointr v0.7.3: Provides functions to estimate cutpoints that optimize a specified metric in binary classification tasks and validate performance using bootstrapping. The vignette shows how to use the functions.

fcr v1.0: Provides a function for dynamic prediction in functional concurrent regression that extends the pffr() function from the refund package to handle the scenario where the functional response and concurrently measured functional predictor are irregularly measured. See Leroux et al. (2017) and the vignette.

ggdag v0.1.0: Builds on the DAGitty web tool to provide functions to tidy, analyze, and plot directed acyclic graphs (DAGs). There is an Introduction to DAGS, an Introduction to ggdag, and a vignette on Common Structures of Bias.

hdme v0.1.1: Provides a function for penalized regression for generalized linear models for measurement error problems including the lasso (L1-penalization), which corrects for measurement error (Sorensen et al. (2015), and an implementation of the Generalized Matrix Uncertainty Selector (Sorensen et al. (2018). The vignette gives the details.

joineRmeta v0.1.1: Extends the joint models proposed by Henderson et. al. (2000) to include multi-study, meta-analytic cases. See the vignette for details.

rare v0.1.0: Implements the alternating direction method of multipliers algorithm of Yan and Bien (2018) for fitting linear models with tree-based lasso regularization. The vignette shows how to use the package.

Time Series

rMEA v1.0.0: Provides tools to read, visualize, and export bivariate motion energy time-series. Lagged synchrony between subjects can be analyzed through windowed cross-correlation. See Ramseyer & Tschacher (2011) for an application, and the README for how to use the package.

tsfknn v0.1.0: Provides a function to forecast time series using nearest neighbors regression. See Martinez et al. (2017) and the vignette for details.

spGARCH v0.1.4: Provides functions to analyze spatial and spatiotemporal autoregressive conditional heteroscedasticity Otto, Schmid, Garthoff (2017), simulation of spatial ARCH-type processes, quasi-maximum-likelihood estimation of the parameters of spARCH models, spatial autoregressive models with spARCH disturbances, diagnostic checks, and visualizations.

Utilities

base2grob v0.0.2: Provides a function to convert a base plot function call (using expression or formula) to grob objects that are compatible to the grid ecosystem so that cowplot can be used to align base plots with ggplot objects. The vignette shows how things work.

cranly v0.1: Provides functions to clean, organize, summarize, and visualize CRAN package database information, and also for building package directives networks (depends, imports, suggests, enhances) and collaboration networks. The vignette shows how to use the package.

osrmr v0.1.28: Implements a wrapper around the Open Source Routing Machine (OSRM) API. See the vignette for details.

fasterize v1.0.0: Provides a fast, drop-in replacement for rasterize() from the raster package that takes sf-type objects and uses the scan line algorithm attributed to [Wylie et al. (1967)](doi:10.¹¹⁴⁵⁄_1465611.1465619 There is a vignette.

jsr223 v0.3.1: Provides a high-level integration that makes Java objects easy to use from within R, and an unified interface for integrating R with several programming languages, including Groovy, JavaScript, JRuby, (Ruby), Jython (Python), and Kotlin. See the manual for details.

Visualization

clustree v0.1.2: Provides functions to produce clustering tree visualizations for interrogating clusterings as resolution increases. See the vignette for details.

datamaps v0.0.2: Enables users to create interactive choropleth maps with bubbles and arcs by coordinates or region name that can be used directly from the console, from RStudio, in Shiny apps, and in R Markdown documents. The vignette will help you get started.

funnelR v0.1.0: Provides functions for creating funnel plots for proportion data, and supports user-defined benchmarks, confidence limits, and estimation methods (e.g., exact or approximate) based on Spiegelhalter (2005). See the Introduction to get started.

nVennR v0.2.0: Provides an interface for the nVenn algorithm of Perez-Silva et al. (2018). See the vignette for an introduction to the package, and the R package UpSetR for help interpreting the results.

smovie v1.0.1: Uses the rpanel package to create interactive movies to help students understand statistical concepts. There are movies to: visualize probability distributions (including user-supplied ones); illustrate sampling distributions of the sample mean (central limit theorem); the sample maximum (extremal types theorem); and more. See the vignette for an overview.

An Introduction to Greta

Mon, 23 Apr 2018 00:00:00 +0000

I was surprised by greta. I had assumed that the tensorflow and reticulate packages would eventually enable R developers to look beyond deep learning applications and exploit the TensorFlow platform to create all manner of production-grade statistical applications. But I wasn’t thinking Bayesian. After all, Stan is probably everything a Bayesian modeler could want. Stan is a powerful, production-level probability distribution modeling engine with a slick R interface, deep documentation, and a dedicated development team.

But greta lets users write TensorFlow-based Bayesian models directly in R! What could be more charming? greta removes the barrier of learning an intermediate modeling language while still promising to deliver high-performance MCMC models that run anywhere TensorFlow can go.

In this post, I’ll introduce you to greta with a simple model used by Richard McElreath in section 8.3 of his iconoclastic book: Statistical Rethinking: A Bayesian Course with Examples in R and Stan. This model seeks to explain the log of a country’s GDP based on a measure of terrain ruggedness while controlling for whether or not the country is in Africa. I am going to use it just to illustrate MCMC sampling with greta. The extended example in McElreath’s book, however, is a meditation on the subtleties of modeling interactions, and is well worth studying.

First, we load the required packages and fetch the data. DiagrammeR is for plotting the TensorFlow flow diagram of the model, and bayesplot is used to plot trace diagrams of the Markov chains. The rugged data set which provides 52 variables for 234 is fairly interesting, but we will use a trimmed-down data set with only 170 counties and three variables.

library(rethinking)
library(greta)
library(DiagrammeR)
library(bayesplot)
library(ggplot2)

# Example from section 8.3 Statistical Rethinking
data(rugged)
d <- rugged
d$log_gdp <- log(d$rgdppc_2000)
dd <- d[complete.cases(d$rgdppc_2000), ]
dd_trim <- dd[ , c("log_gdp","rugged","cont_africa")]
head(dd_trim)

##     log_gdp rugged cont_africa
## 3  7.492609  0.858           1
## 5  8.216929  3.427           0
## 8  9.933263  0.769           0
## 9  9.407032  0.775           0
## 10 7.792343  2.688           0
## 12 9.212541  0.006           0

set.seed(1234)

In this section of code, we set up the TensorFlow data structures. The first step is to move the data into greta arrays. These data structures behave similarly to R arrays in that they can be manipulated with functions. However, greta doesn’t immediately calculate values for new arrays. It works out the size and shape of the result and creates a place-holder data structure.

#data
g_log_gdp <- as_data(dd_trim$log_gdp)
g_rugged <- as_data(dd_trim$rugged)
g_cont_africa <- as_data(dd_trim$cont_africa)

In this section, we set up the Bayesian model. All parameters need prior probability distributions. Note that the parameters a, bR, bA, bAR, sigma, and mu are all new greta arrays that don’t contain any data. a is 1 x 1 array and mu is a 170 x 1 array with one slot for each observation.

The distribution() function sets up the likelihood function for the model.

# Variables and Priors

a <- normal(0, 100)
bR <- normal(0, 10)
bA <- normal(0, 10)
bAR <- normal(0,10)
sigma <- cauchy(0,2,truncation=c(0,Inf))

a  # Look at this greata array

## greta array (variable following a normal distribution)
## 
##      [,1]
## [1,]  ?

# operations
mu <- a + bR*g_rugged + bA*g_cont_africa + bAR*g_rugged*g_cont_africa

dim(mu)

## [1] 170   1

# likelihood
distribution(g_log_gdp) = normal(mu, sigma)

The model() function does all of the work. It fits the model and produces a fairly complicated object organized as three lists that contain, respectively, the R6 class, TensorFlow structures, and the various greta data arrays.

# defining the model
mod <- model(a,bR,bA,bAR,sigma)

str(mod,give.attr=FALSE,max.level=1)

## List of 3
##  $ dag                 :Classes 'dag_class', 'R6' <dag_class>
##   Public:
##     adjacency_matrix: function () 
##     build_dag: function (greta_array_list) 
##     clone: function (deep = FALSE) 
##     compile: TRUE
##     define_gradients: function () 
##     define_joint_density: function () 
##     define_tf: function () 
##     example_parameters: function (flat = TRUE) 
##     find_node_neighbours: function () 
##     get_tf_names: function (types = NULL) 
##     gradients: function (adjusted = TRUE) 
##     initialize: function (target_greta_arrays, tf_float = tf$float32, n_cores = 2L, 
##     log_density: function (adjusted = TRUE) 
##     make_names: function () 
##     n_cores: 4
##     node_list: list
##     node_tf_names: variable_1 distribution_1 data_1 data_2 operation_1 oper ...
##     node_types: variable distribution data data operation operation oper ...
##     parameters_example: list
##     send_parameters: function (parameters, flat = TRUE) 
##     subgraph_membership: function () 
##     target_nodes: list
##     tf_environment: environment
##     tf_float: tensorflow.python.framework.dtypes.DType, python.builtin.object
##     tf_name: function (node) 
##     trace_values: function ()  
##  $ target_greta_arrays :List of 5
##  $ visible_greta_arrays:List of 9

Plotting mod produces the TensorFlow flow diagram that shows the structure of the underlying TensorFlow model, which is simple for this model and easily interpretable.

# plotting
plot(mod)

Next, we use the greta function mcmc() to sample from the posterior distributions defined in the model.

# sampling
draws <- mcmc(mod, n_samples = 1000)
summary(draws)

## 
## Iterations = 1:1000
## Thinning interval = 1 
## Number of chains = 1 
## Sample size per chain = 1000 
## 
## 1. Empirical mean and standard deviation for each variable,
##    plus standard error of the mean:
## 
##          Mean      SD Naive SE Time-series SE
## a      9.2225 0.13721 0.004339       0.004773
## bR    -0.2009 0.07486 0.002367       0.002746
## bA    -1.9485 0.23033 0.007284       0.004435
## bAR    0.3992 0.13271 0.004197       0.003136
## sigma  0.9527 0.04892 0.001547       0.001744
## 
## 2. Quantiles for each variable:
## 
##          2.5%     25%     50%     75%    97.5%
## a      8.9575  9.1284  9.2306  9.3183  9.47865
## bR    -0.3465 -0.2501 -0.1981 -0.1538 -0.05893
## bA    -2.3910 -2.1096 -1.9420 -1.7876 -1.50781
## bAR    0.1408  0.3054  0.3954  0.4844  0.66000
## sigma  0.8616  0.9194  0.9520  0.9845  1.05006

Now that we have the samples of the posterior distributions of the parameters in the model, it is straightforward to examine them. Here, we plot the posterior distribution of the interaction term.

mat <- data.frame(matrix(draws[[1]],ncol=5))
names(mat) <- c("a","bR","bA","bAR","sigma")
#head(mat)
# http://www.cookbook-r.com/Graphs/Plotting_distributions_(ggplot2)/
ggplot(mat, aes(x=bAR)) + 
  geom_histogram(aes(y=..density..), binwidth=.05, colour="black", fill="white") +
  geom_density(alpha=.2, fill="#FF6666")

Finally, we examine the trace plots for the MCMC samples using the greta function mcmc_trace(). The plots for each parameter appear to be stationary (flat, i.e., centered on a constant value) and well-mixed (there is no obvious correlation between points). mcmc_intervals() plots the uncertainty intervals for each parameter computed from posterior draws with all chains merged.

mcmc_trace(draws)

mcmc_intervals(draws)

So there it is - a Bayesian model using Hamiltonian Monte Carlo sampling built in R and evaluated by TensorFlow.

For an expert discussion of the model, have a look at McElreath’s book, described at the link above. For more on greta, see the package documentation. And please, do take the time to read about greta’s namesake: Greta Hermann, a remarkable woman - mathematician, philosopher, educator, social activist, and theoretical physicist who found the error in John von Neuman’s “proof” of the “No hidden variables theorem” of Quantum Mechanics.

Feb 2018: "Top 40" New Package Picks

Thu, 29 Mar 2018 00:00:00 +0000

Here are my picks for the “Top 40” packages of the 171 new packages that made it to CRAN (and stuck) in February, organized into the following categories: Computational Methods, Data, Finance, Science, Statistics, Time Series, and Utilities.

Computational Methods

adnuts v1.0.0: Provides an implementation of the no-U-turn (NUTS) algorithm by Hoffman and Gelman (2014) for ADMB and TMB models. The vignette will get you started.

CholWishart v0.9.2: Provides functions to sample from the Cholesky factorization of a Wishart random variable, the inverse Wishart distribution and the Cholesky factorization of an inverse Wishart random variable. See the vignette for details.

particles v0.2.1: Provides functions to simulate particle movement in 2D space using the ideas behind the ‘d3-force’ JavaScript particles library. It implements all forces defined in d3-force, as well as others such as vector fields, traps, and attractors. The vignette explains how to use the package.

rosqp v0.1.0: Provides bindings to the OSQP solver, which can solve sparse convex quadratic programming problems with optional equality and inequality constraints.

SolveLS v1.0: Implements methods including Jacobi, Gauss-Seidel, Successive Over-Relaxation, SSOR and non-stationary, Krylov subspace methods. See this book for details.

Data

Cluster.OBeu v1.2.1: Provides functions to estimate and return the needed parameters for visualizations designed for OpenBudgets data. There is a vignette for Using Cluster.OBeu with OpenCPU and one for Cluster analysis.

photobiologySun v0.4.0: Contains data for extraterrestrial solar spectral irradiance and ground-level solar spectral irradiance and irradiance. See Aphalo P. J. (2015) and the User Guide for more information.

SympluR v0.3.0: Provides functions to analyze data from the Healthcare Social Graph via access to the Symplur API. Look here for related research articles.

totalcensus v0.3.0: Allows users to download summary files from the Census Bureau and extract data - in particular, high resolution data at block, block group, and tract level - from decennial census and American Community Survey 1-year and 5-year estimates.

Finance

estudy2 v0.8.4: Implements event study models, including rate-of-return estimation and classical models. Tests include those proposed by [Brown and Warner (1980)](doi:10.¹⁰¹⁶⁄₀₃₀₄-405X(80)90002-1], [Brown and Warner (1985)](doi:10.¹⁰¹⁶⁄₀₃₀₄-405X(85)90042-X], [Boehmer et al. (1991)](doi:10.¹⁰¹⁶⁄₀₃₀₄-405X(91)90032-F>] and more. The vignette provides an introduction.

Machine Learning

DALEX v0.1.1: Provides various explainers that help to understand the link between input variables and model output in machine learning models. See this website for explanations.

forestControl v0.1.1: Allows approximate false positive rate control in selection frequency for random forest using the methods described by Konukoglu and Ganz (2015).

kmed v0.0.1: Implements the distance-based k-medoids clustering algorithm from Park and Jun (2009). Cluster validation applies bootstrap procedure producing a heatmap with a flexible reordering matrix algorithm. There is a vignette.

lolR v1.0.1: Implements optimal low-rank projection algorithms to obtain a lower-dimensional representation of data before applying supervised learning techniques in situations where the dimensionality exceeds the sample size. There are several vignettes including: Class Condidtional PCA, Low-Rank Canonical Correlation Analysis, and HDLSS Simulations.

projpred v0.7.0: Provides functions to perform projection predictive feature selection for generalized linear models; see, for example, Piironen and Vehtari (2017). The package is compatible with rstanarm. There is a Quick Start Guide.

RGF v1.0.1: Implements a wrapper for the python package Regularized Greedy Forest. It also includes a multi-core implementation called FastRGF.

Science

cRegulome v0.1.1: Provides functions to build a SQLite database file of pre-calculated transcription factor/microRNA-gene correlations (co-expression) incancer from the Cistrome and miRCancerdb databases. There is an Introduction and a Case Study.

CENFA v0.1.0: Provides tools for climate- and ecological-niche factor analysis of spatial data, including methods for visualization of spatial variability of species sensitivity, exposure, and vulnerability to climate change. See Hirzel et al. (2002) and Basille et al. (2008). The vignette introduces the package.

detectRUNS v0.9.5: Provides functions to detect runs of homozygosity and of heterozygosity in diploid genomes using the sliding windows ( Purcell et al (2007) ) and consecutive runs ( Marras et al (2015) ) methods. The vignette provides an overview.

Statistics

cosa v1.1.0: Implements generalized constrained optimal sample allocation framework for two-group multilevel regression discontinuity studies and multilevel randomized trials with continuous outcomes. There is a short Tutorial.

DirectEffects v0.1: Provides functions to estimate the controlled direct effect of treatment fixing a potential mediator to a specific value. Implements the sequential g-estimation estimator described in Vansteelandt (2009) and Acharya et al. (2016). The vignette introduces the package.

dnr v0.3.2: Provides functions to fit temporal lag models to dynamic networks built on top of exponential random graph models (ERGM) framework. The vignette describes the method.

geozoning v1.0.0: Provides a zoning method and a numerical criterion for assessing zoning quality. There are vignettes on Geozoning Structures and Simulated Data.

GpGp v0.1.0: Provides functions for Gaussian process predictions and conditional simulations, along with covariance functions for spatial and spatial-temporal data on Euclidean domains and spheres. The original approximation is due to Vecchia (1988), and the reordering and grouping methods are from Guinness (2018). The vignette contains an example using wind speed.

idealstan v0.2.7: Offers item-response theory (IRT) ideal-point scaling/dimension reduction methods that incorporate additional response categories and missing/censored values. Full and approximate Bayesian inference is done via the Stan engine. There is an Introduction and a vignette on Evaluating Models.

kdensity v1.0.0: Provides methods for univariate non-parametric density estimation with parametric starts and asymmetric kernels. See Chen (2000), Chen (1999), and Jones & Henderson (2007). There is a Tutorial.

NetLogoR v0.3.2: Provides functions to create agent-based models in R following the NetLogo framework. See Wilensky (1999). The NetLogo models Ants and Wolf-Sheep-Predation have been translated in R. See the Programming Guide and Data Dictionary.

riskyr v0.1.0: Provides functions to express risk-related information in terms of probabilities or frequencies to make the teaching and training of risk literacy more transparent. There is a User Guide and Quick Start Primer, along with vignettes on Data Formats, the Confusion Matrix and Metrics, and Functional Perspectives.

rsimsum v0.3.0: Provides functions to summarize results from simulation studies and compute Monte Carlo standard errors of commonly used summary statistics. This package is modeled on the simsum user-written command in Stata. There is an Introduction and vignettes on Visualization, Simulating a simulation study, and rsimsum and the tidyverse.

SimCorrMix v0.1.0: Provides functions to generate continuous (normal, non-normal, or mixture distributions), binary, ordinal, and count (regular or zero-inflated, Poisson or Negative Binomial) variables with a specified correlation matrix, or one continuous variable with a mixture distribution. This package can be used to simulate data sets that mimic real-world clinical or genetic data sets (i.e., plasmodes, as in Vaughan et al. (2009). There are vignettes on Continuous Mixture Distributions, Expected Cumulants and Correlations for Continuous Mixture Variables, Comparison of Correlation Methods, Variable Types, and Overall Workflow for Generation of Correlated Data

tree.bins v0.1.0: Allows users to recategorize the factors variables through a decision tree method derived from the rpart() function of the rpart package. For details, see Hastie et al (2009) and the vignette.

Time Series

segclust2d v0.1.0: Provides two methods for segmentation and joint segmentation/clustering of bivariate time-series. The segmentation method is a bivariate extension of Lavielle’s method available in adehabitatLT Lavielle (1999) and Lavielle (2005). The segmentation/clustering method is an extension of Picard et al (2007). The vignette contains several examples.

tstools v0.3.6: Provides functions to plot official statistics time series with automatic legends, highlight windows, stacked bar chars with positive and negative contributions, and other options. It includes a fast, data.table backed time series I/O that allows the user to export / import long format, wide format, and transposed wide format data to various file types. See the vignette for details.

Utilities

codemetar v0.1.5: Provides utilities to generate, parse, and modify codemeta.json files automatically for R packages, as defined in the Codemeta Project. There is an Introduction to the Codemeta Project, and vignettes on Translating Between Data Formats, Validating JSON-LD, and Examples.

knitrProgressBar v1.1.0: Provides a progress bar similar to dplyr that can write progress out to a variety of locations, including stdout(), stderr(), or from file(). There is an Example and a vignette for setting up.

msgpack v1.0: Implements a fast, C-based encoder and streaming decoder for the messagepack data format.

pmatch v0.1.3: Implements type constructions and pattern matching. See the README for details.

shinyalert v1.0: Provides functions to create pretty popup messages (modals) in Shiny that may contain text, images, OK/Cancel buttons, an input to get a response from the user, and many more customizable options.

trackr v0.7.5: Provides functions to automatically annotate R-based artifacts with relevant descriptive and provenance-related notes, and provides a back-end-agnostic storage and discoverability system for organizing, retrieving, and interrogating such artifacts. There is an Introduction and a vignette on Extending trackr.

Alternative Design for Shiny

Tue, 13 Mar 2018 00:00:00 +0000

Shiny’s Design

Most Shiny apps out there have a similar design style. It is usually easy for a seasoned Shiny developer to tell the difference between a Shiny app and a standard website. Why is this? Shiny apps ARE websites for all intents and purposes. Why do they not vary as greatly as the rest of the sites we encounter when surfing the web?

This is partly due to the fact that many Shiny developers leverage pre-written code (e.g., the layout was presented to them). There is another major factor here, though: the web framework that powers Shiny, called Bootstrap.

Bootstrap

Shiny implements the most widely used and established CSS, JavaScript, and HTML framework that exists today. Bootstrap was originally written for internal use at Twitter, and has since blossomed into the most utilized web framework across the web.

People with experience in Shiny will recognize these examples (think actionButton and selectInput) from the Bootstrap website:

Alternatives

While Bootstrap is the most widely used web framework, other frameworks do exist that are also appealing. There are too many good ones to name here, so I will go directly to the one which is the basis for this article: Material Design.

Material Design

Material Design is an extremely popular set of design standards created by Google. Anyone who uses Google web products (or Android) will quickly recognize some of their work.

Here is an example of some UI elements that adhere to Material Design:

Standards vs. Framework

Material Design is not the same thing as Bootstrap. Bootstrap is the design standards, along with the CSS, JavaScript, and HTML code written to implement those standards. Material Design is only the standards. While Google has the resources to implement the standards themselves, for an average developer this would be extremely difficult to take on. Thankfully, open-source development has a solution: an excellent framework called Materialize CSS.

Materialize CSS implements Google’s Material Design with such precision that is indistinguishable from Google’s own implementation. The UI elements found in Materialize CSS are slick and are suitable for any web site or application.

Material Design in Shiny

Let’s assume someone wants to implement Material Design in a Shiny app. Where does one start? Yes, Materialize CSS exists, but that is code written for the web and not for native R users (hence Shiny). This leads to, yes, an R package. The package is called shinymaterial, and it allows Shiny developers to implement Material Design using only R code. The package contains many of the standard types of Shiny inputs, but with its own API (e.g., material_select vs. selectInput). Shiny developers will quickly pick up the similarities between shinymaterial inputs and standard shiny inputs.

Along with inputs, the package also has more broad UI functionality, including the ability for Shiny developers to create Material dashboards, as shown here, along with other unique features such as parallax scrolling, shown here.

Conclusion

Shiny changed the game. It turned R programmers into web developers overnight. What shinymaterial aims to do is to provide Shiny developers with a little more design flexibility, while maintaining the spirit of Shiny: simple APIs that R developers can use to turn their scientific analyses into great web apps.

Jan 2018: "Top 40" New Package Picks

Thu, 22 Feb 2018 00:00:00 +0000

Here are my “Top 40” picks from the two hundred or so new packages that stuck to CRAN in January, listed under seven categories: Data, Data Science, Science, Statistics, Time Series, Utilities and Visualizations (I say “stuck to” because I counted at least six packages that were accepted onto CRAN in January but removed within the month. Having packages quickly removed from CRAN is a phenomenon I have observed in recent months.)

While looking over the packages that I have listed under Data and Science, it struck me that in addition to being the world’s largest repository of statistical knowledge, CRAN is becoming a repository for practical, hard-won scientific knowledge.

Data

cancensus v0.1.7: Provides an interface to Canadian census and geographic data using the CensusMapper API. There is an Introduction and a vignette for Making maps.

elevatr v0.1.4: Provides access to several services offering elevation data, and returns the data either as a SpatialPointsDataFrame from point elevation services or as a raster object from raster elevation services. Currently, the package supports access to the Mapzen Elevation Service, Mapzen Terrain Service, Amazon Web Services Terrain Tiles, and the USGS Elevation Point Query Service. The vignette shows how to use the package.

fabricatr v0.2.0: Provides functions to simulate hierarchical and correlated data. There are several vignettes including a Getting Started guide and an Advanced Features guide, as well as introductions to Resampling and Generating Discrete Random Variables.

getTBinR v0.5.2: Facilitates easy import of analysis-ready World Health Organisation Tuberculosis data, and provides plotting functions for exploratory data analysis. There is a vignette.

homologene v1.1.68: Provides a wrapper for the homologene database by the National Center for Biotechnology Information, which allows searching for gene homologs across species.

photobiologyFilters v0.4.4: Is a data-only package with spectral ‘transmittance’ data for frequently used filters and materials, including plastic sheets and films, optical glass and ordinary glass, and some labware. It complements the photobiology package. See this website and the vignette for details.

tfdatasets v1.5: Provides an interface to TensorFlow Datasets, a high-level library for building complex input pipelines from simple, re-usable pieces.

washdata v0.1.2: Provides access to the urban water and sanitation survey data set collected by Water and Sanitation for the Urban Poor (WSUP), with technical support from Valid International. There is a vignette.

Data Science

CRPClustering v1.0: Provides a clustering method using the Chinese restaurant process Pitman (1995) that does not need to decide the number of clusters in advance. Also provides functions to calculate the ambiguity of clusters as entropy Yngvason (1999). The vignette shows how to use the package.

kerasformula v0.1.1: Provides a high-level interface for keras neural nets. See the vignette for details.

multiROC v1.0.0: Provides tools to solve problems with multiple classes by computing the areas under ROC curve via micro- and macro-averaging. The methodology is described in Van Asch (2013) and Pedregosa et al. (2011). See the vignette for a quick tour.

reinforcelearn v0.1.0: Implements reinforcement learning environments and algorithms. as described in Sutton & Barto (1998). There are vignettes for Agents and Environments.

stranger v0.3.2: Provides a framework for unsupervised anomalies detection There is a Vignette for the Impatient, and vignettes for Methods and Anomalies manual selection.

tidypredict v0.1.0: Provides functions to parse a fitted ‘R’ model object, and return a SQL query. There are vignettes for GLM and Random Forest models.

Science

annovarR v1.0.0: Provides unctions and database resources to offer an integrated framework to annotate genetic variants from genome and transcriptome data. The wrapper functions unify the interface of many published annotation tools, such as VEP, ANNOVAR, vcfanno, and AnnotationDbi. There is an Introduction and a vignette on Databases.

pubh v0.1.7: Offers a toolbox for making R functions and capabilities more accessible to students and professionals from Epidemiology and Public Health related disciplines. There is an Introduction and a Regression Example.

trajr v1.0.0: Provides a toolbox to assist with statistical analysis of two-dimensional animal trajectories. The vignette provides several examples.

Statistics

dalmatian v0.3.0: Automates fitting a double GLM in JAGS. There is a vignette on weighted regression and a two-part example using the Pied Flycatcher Data: Part 1 and Part 2.

dirichletprocess v0.2.0: Enables the creation of Dirichlet process objects that can be used as infinite mixture models. Examples include density estimation, Poisson process intensity inference, hierarchical modelling, and clustering. See Teh, Y. W. (2011) and the vignette for details.

detpack v1.0.1: Enables density estimation for possibly large data sets and conditional/unconditional random number generation with distribution element trees. For details on distribution element trees, see Meyer (2016), Meyer (2017), and Meyer (2017).

gnorm v1.0.0: Provides functions for obtaining generalized normal/exponential power distribution probabilities, quantiles, densities, and random deviates. See the vignette for details.

IROmiss v1.0.1: Provides a general algorithm, the Imputation Regularized Optimization (IRO) algorithm, for high-dimensional missing data problems. See Liang et al. (2018) for details.

KRIG v0.1.0: Provides functions for Kriging models and various methods for spatial statistics, including multivariate sensitivity analysis using reproducing kernel Hilbert spaces and computation of Sobol indexes. There are vignettes on Ordinary Kriging, Simple Kriging, Universal Kriging, and a worked example.

natural v0.9.0: Implements two error variance estimation methods in high-dimensional linear models. See Yu, Bien (2017) and the vignette.

OpVar v1.0: Provides functions for modeling operational (value-at-)risk, including loss frequencies and loss severities with plain, mixed (Frigessi et al. (2012)) or spliced distributions using Maximum Likelihood estimation and Bayesian approaches (Ergashev et al. (2013)). The vignette shows some examples.

netrankr v0.2.0: Implements methods for centrality-related analyses of networks, focusing on index-free assessment of centrality via partial rankings obtained by neighborhood-inclusion or positional dominance. See Schoch (2018). There are vignettes for benchmarks, centrality indices, indirect relations, neighborhood inclusion, partial centrality, positional dominance, probabilistic centrality, uniquely ranked graphs, and a use case.

palmtree v0.9.0: Implements the PALM tree algorithm, an extension to the MOB algorithm (implemented in the partykit package), where some parameters are fixed across all groups. See Seibold et al. (2016) for details.

PMCMRplus v1.0.0: Provides functions to calculate many different types of pairwise multiple comparisons tests. See the vignette for charts listing the tests covered.

seminr v0.4.0: Implements a domain-specific language for building PLS structural equation models, allowing for the latest estimation methods for Consistent PLS as per Dijkstra & Henseler (2015), adjusted interactions as per Henseler & Chin (2010), and bootstrapping utilizing parallel processing as per Hair et al. (2017). There is a vignette.

Time Series

santaR v1.0: Provides a graphical, automated pipeline for the analysis of short time series that has been designed to accommodate asynchronous time sampling, inter-individual variability, noisy measurements and large numbers of variables. There is a Getting Started Guide and vignettes on advanced command line functions, automated command line functions, plotting options, preparing input, selecting degrees of freedom, the theoretical background, and the GUI.

TSrepr v1.0.0: Provides methods for representations (e.g., dimensionality reduction, preprocessing, feature extraction) of time series. There is an Introduction to the Framework, a vignettes on representations, and a Use Case.

TSstudio v0.1.1: Provides a set of interactive visualization tools for time series analysis supporting ts, mts, zoo and xts objects including visualization functions for forecasting model performance (forecasted vs. actual), time series interactive plots (single and multiple series), and seasonality plots. The vignette shows the features available.

Utilities

arrangements v1.0.2: Provides fast generators and iterators for permutations, combinations and partitions, allowing users to generate arrangements in a memory-efficient manner. Benchmarks may be found here.

fs v1.1.0: Implements a cross-platform interface to file system operations, built on top of the libuv C library. See README for details.

googlePolylines v0.4.0: Provides functions to encode simple feature (sf) objects and coordinates using the Google polyline encoding algorithm. The vignette introduces the package.

prrd v0.2.0: Provides functions to queue reverse depends for a given package, such that multiple workers can run the tests in parallel. Look here or in the README for functionality details.

rquery v0.3.1: Implements a query generator based on Edgar F. Codd’s relational algebra and operator names, which is aimed at enhancing the experience using ‘SQL’ at big-data scale. There is a vignette on the Assignment Partitioner and one on Query Generation.

tsibble v0.1.3: Provides a tbl_ts class, the tsibble, to store and manage temporal-context data in a data-centric format. There is an Introduction.

Visualizations

breakDown v0.1.3: Implements break-down plots which show the contribution of every variable present in the model. Vignettes cover linear models, GLMs, and ranger models.

sigmaNet v1.0.3: Offers functions to create interactive graph visualizations using Sigma.js. The vignette shows how to get started.

Dec 2017: "Top 40" New Package Picks

Thu, 25 Jan 2018 00:00:00 +0000

Sometimes it appears to me that the invisible hand economists speak of guides the market for new R packages. Eight of the 129 new packages that stuck to CRAN in December fall under Computational Methods, a category I have only recently begun using. All of them made it into the list below of my “Top 40” picks. One day, I would like to go back and reexamine the categories I have been using to see if package developers really do respond to some idea that is “in the air” or whether the variation in categories is just one more of my many hidden biases.

Computational Methods

alphashape3d v1.3: Provides functions to compute the alpha-shape (a generalization of the convex hull) of a finite set of points in the three-dimensional space.

deGradInfer v1.0.0: Implements efficient Bayesian parameter inference for systems of ordinary differential equations based on adaptive gradient matching. See Dondelinger et al. (2013) and Macdonald (2017). The vignette provides examples.

FixedPoint v0.1: Provides algorithms for finding fixed-point vectors, including iterative procedures for non-linear integral equations Anderson (1965), epsilon extrapolation methods Wynn (1962), and minimal polynomial methods Cabay & Jackson (1976). The vignette provides a very nice introduction to the subject.

grapherator v1.0.0: Aimed at research in single- and multi-objective combinatorial optimization, the package provides functions for step-wise generation of weighted graphs. There are vignettes on Graph Generation, Custom Generators and using pipes.

HMMEsolver v0.1.0: Implements a fast solver for Henderson Mixed Model Equation via row operations without computing a matrix inverse. See Kim (2017) for more details.

kexpmv v0.0.3: Implements functions from EXPOKIT to calculate matrix exponentials. See Sidje RB, (1998) for both small dense matrices and large sparse matrices based on Krylov subspace methods.

sparseEigen v0.1.0: Provides methods to compute sparse eigenvectors of a matrix with running time two to three orders of magnitude lower than existing methods. The methods are based on the paper by (Benidis et al. (2016). The vignette includes performance benchmarks.

TukeyRegion v0.1.2: Provides fast computation of Tukey regions, polytopes in the Euclidean space giving upper-level sets of the Tukey depth function for given data. For details, see Liu, Mosler, and Mozharovskyi (2017).

Data

mlbgameday v0.0.1: Implements methods for multi-core processing of Gameday data from Major League Baseball Advanced Media (http://gd2.mlb.com/components/game/mlb/). There are vignettes on Database Connedtions, Parallel Processing, Plotting Pitches, and Searching Games.

robis v1.0.0: Implements a client for the Ocean Biogeographic Information System (http://iobis.org). See README to get started.

seaaroundus v1.2.0: Provides access to Sea Around Us fish catch data. See README to get started.

tidyhydat v0.3.2: Provides functions to extract historical and real-time national ‘hydrometric’ data from Water Survey of Canada data sources here and here. There is an Introduction and an Example.

Machine Learning

afCEC v1.0.2: Implements active function cross-entropy clustering partitions the n-dimensional data into the clusters by finding the parameters of the mixed generalized multivariate normal distribution, that optimally approximates the scattering of the data in the n-dimensional space. For details see P. Spurek et al (2017).

dissever v0.2-2: Enables spatial down-scaling of coarse-grid mapping to fine-grid mapping using predictive covariates and a model fitted using the ‘caret’ package. The original dissever algorithm was published by Malone et al. (2012) and extended by Roudier et al. (2017). There is a short tutorial.

mlapi Provides R6 abstract classes for building machine-learning models with scikit-learn-like API. (scikit-learn is a popular module for Python programming language.) There is a vignette.

Numero v1.0.3: Implements an unsupervised statistical framework for defining subgroups in complex datasets based on visual cues Makinen et al. (2011). The vignette shows how to use the package.

PPforest v0.1.0: Implements projection pursuit forest algorithm for supervised classification. The vignette provides details.

qCBA v0.3.1: Implements quantitative classification by association rules. See Kliegr (2017).

tfestimators v1.5: Implements an interface to TensorFlow Estimators, a high-level API that provides implementations of many different model types, including linear models and deep neural networks. There is an Introduction and vignettes on Custom Estimators, the Dataset API, Basic Components, Feature Columns, Input Functions, Parsing Utilites, Run Hooks, and TensorBoard Visualization.

Science

ePCR v0.9.9-4: Provides the top-performing ensemble-based Penalized Cox Regression (ePCR) framework developed during the DREAM 9.5 mCRPC Prostate Cancer Challenge. See Guinney J et al. (2017) for details.

simRVPedigree v0.1.0: Provides routines to simulate and manipulate pedigrees ascertained to contain multiple family members affected by a rare disease. See Nieuwoudt et al. (2017) for the science and the vignette to get started.

theseus v0.1.0: Provides analysis and visualization tools for the interpretation of microbial community composition data, especially those originating from amplicon sequencing. The vignette describes how to use the package.

Statistics

ForecastComb v1.1: Provides geometric and regression-based forecast combination methods under a unified user interface for the packages ForecastCombinations and GeomComb. For details see Hsiao C, Wan SK (2014), Hansen BE (2007), [Elliott G, Gargano A, Timmermann A (2013)](doi:10.1016/j.jeconom.2013.04.017] and Clemen RT (1989).

hesim v0.1.0: Provides functions to develop and analyze health-economic simulation models, including random sampling functions for probabilistic sensitivity analyses Claxton et al. (2005), individual patient simulations Brennan et al. (2006), cost-effectiveness analysis Basu and Meltzer (2007), and Ioannidis and Garber (2011). The vignette provides an overview.

PlackettLuce v0.2-1: Implements a generalization of the model jointly attributed to Plackett (1975) and Luce (1959) for modelling rankings data. The vignette introduces the model.

PUlasso v2.1: Implements an efficient algorithm for solving the Positive and Unlabeled problem in low- or high-dimensional settings with lasso or group lasso penalty. See Hyebin 7 Raskutti (2017) for details and the vignette for an introduction.

recurse v1.0.1: Computes revisitation metrics for trajectory data, such as the number of revisitations for each location as well as the time spent for that visit and the time since the previous visit. The vignette works through a case study of using the package to analyze revisitations in animal movement data.

samplesizeCMH v0.0.0: Provides functions to calculate the power and sample size for Cochran-Mantel-Haenszel tests, and for working with probability, odds, relative risk, and odds ratio values. There is an Introduction to the Cochran-Mantal-Haenszel Test, and vignettes on Power Calculations and Sample Size Calculations.

skimr v1.0.1: Provides a function to display summary statistics at the console. There is an Introduction and vignettes on Fonts and defining summary objects.

ZOIP v0.1: Implements the ZOIP (Zeros Ones Inflated Proportional), proportional data distribution inflated with zeros and/or ones. See Jørgensen and Barndorff-Nielsen (1991), Ferrari and Cribari-Neto (2004), and Rigby and Stasinopoulos (2005) for details, and the vignette for a summary.

Time Series

OSTSC v0.0.1: Provides functions for oversampling imbalanced univariate time series classification data using integrated Enhanced Structure Preserving Oversampling (ESPO) and Adaptive Synthetic (ADASYN) methods. The vignette describes the method and provides examples.

Utilities

JuniperKernel v1.2.2.0: Implements the Jupyter kernel for R, providing an interface to libraries that exist in the Jupyter ecosystem for building widgets, plotting, and more. Look here for details.

labelVector v0.0.1: Supports labels for atomic vectors in a light-weight design that is suitable for use in other packages. The vignette provides details.

ncmeta v0.0.1: Provides functions to extract metadata from NetCDF data sources, which can be files, file handles, or servers. The provides a framework for the in-development tidync project.

RPostgres v1.0-4: Implements a fully DBI-compliant, Rcpp-backed interface to PostgreSQL, an open-source relational database.

swatches v0.5.0: Provides functions to read and inspect Adobe Color (ACO), Adobe Swatch Exchange (ASE), GIMP Palette (GPL), OpenOffice palette (SOC) files, and KDE Palette (colors) files.

stylr v1.0.0: Provides functions to pretty-print R code without changing the user’s formatting intent. There is an Introduction, and vignettes on Customization and Performance Improvements.

Visualizations

BioCircos.png v0.2.2: Implements interactive Circos-like visualizations of genomic data, to map information such as genetic variants, genomic fusions, and aberrations to a circular genome. The vignette shows how to use the package.

cubing v1.0-1: Provides functions for visualizing, animating, solving and analyzing the Rubik’s cube. See Rokicki (2008) for the underlying Kociemba solver. The vignette shows how to get started.

Downtime Reading

Fri, 29 Dec 2017 00:00:00 +0000

Not everyone has the luxury of taking some downtime at the end the year, but if you do have some free time, you may enjoy something on my short list of downtime reading. The books and articles here are not exactly “light reading”, nor are they literature for cuddling by the fire. Nevertheless, you may find something that catches your eye.

The Syncfusion series of free eBooks contains more than a few gems on a variety of programming subjects, including James McCaffrey’s R Programming Succinctly and Barton Poulson’s R Succinctly.

For a more ambitious read, mine the rich vein of SUNY Open Textbooks. My pick is Hiroki Sayama’s Introduction to the Modeling and Analysis of Complex Systems.

If you just can’t get enough of data science, then a few articles that caught my attention are:

Christopher Olah’s brief but mind-stretching post on Neural Networks, Manifolds, and Topology, which is good preparation for the Fujitsu Laboratories paper on Time Series Classification via Topological Data Analysis
The paper by Nguyen and Holmes on their Bayesian Unidimensional Scaling (BUDS) method for detecting patterns in high-dimensional data
Bou-Hamad et. al’s A review of survival trees, a valuable introduction to the literature on the subject
Rob Hyndman’s recent post on Some new time series packages
Mike Bostock’s beautiful and mind-altering post on Visualizing Algorithms.

Finally, if you really have some time on your hands, try searching through the 318M+ papers on PDFDRIVE.

Happy reading, and have a Happy and Prosperous New Year from all of us at RStudio!!

Nov 2017: New Package Picks

Fri, 22 Dec 2017 00:00:00 +0000

Two hundred thirty-seven new packages made it to CRAN in November. Here are my picks for the “Top 40” organized into the categories: Computational Methods, Data, Data Science, Science, Social Science, Utilities and Visualizations.

Computational Methods

CVXR v0.94-4: Implements an object-oriented modeling language for disciplined convex programming (DCP) which allows users to formulate and solve convex optimization problems. The vignette introduces the package. Look here for examples and theory.

PreciseSums v0.1: Implements the Kahan (1965) sum, Neumaier (1974) sum, pairwise-sum adapted from ‘NumPy’ and arbitrary precision sum.

Data

ballr v0.1.1: Provides functions for accessing data/tables from basketball-reference.com. There is a vignette.

biofiles v1.0.0: Provides functions to parse GenBank/GenPept records into native R objects, access and manipulate the sequence and annotation information. There is an Introduction.

ipumsr v0.1.1: Enables users to import census, survey and geographic data from IPUMS. There is an Introduction and vignettes on CPS Extraction, Geographic Data, NHDIS Datasets and on Using Value Labels.

proPubBills v0.1: Implements an API wrapper around the ProPublica API. The brief vignette shows how to use it.

Rpolyhedra v0.1.0: Contains a 142 polyhedra database scraped from PHD files as R6 objects, and provides rgl visualizing capabilities. The PHD format was created to describe the geometric polyhedra definitions derived mathematically by Andrew Hume and by the Kaleido program of Zvi Har’El. The vignette will get you started.

voteogram v0.2.0: Provides tools to retrieve United States Congressional voting data from ProPublica, prepare the data for plotting with ggplot2 and create vote cartograms and themes. The vignette provides examples.

Data Science

imbalance v0.1.1: Provides algorithms to treat unbalanced datasets. See the vignette for details.

intrinsicdimension v1.1.0: Implements a variety of methods for estimating intrinsic dimension of data sets (i.e the manifold or Hausdorff dimension of the support of the distribution that generated the data) as reviewed in Johnsson et al.(2015). The vignette provides an Overview.

ppclust v0.1.0: Implements probabilistic clustering algorithms for partitioning datasets including Fuzzy C-Means (Bezdek, 1974), Possibilistic C-Means (Krishnapuram & Keller, 1993), Possibilistic Fuzzy C-Means (Pal et al, 2005), Possibilistic Clustering Algorithm (Yang et al, 2006), Possibilistic C-Means with Repulsion (Wachs et al, 2006) and the other variants. There are vignettes on Fuzzy C-Means, Probabilistic C-Means, Probabilistic Fuzzy C-Means and Unsupervised Probabilistic Fuzzy C-Means.

textrank v0.2.0: Implements the textrank algorithm, an extension of the Pagerank algorithm for text. See the paper Mihalcea & Tarau (2004) and the vignette.

TrajDataMining v0.1.4: Contains a set of methods for trajectory data preparation, such as filtering, compressing and clustering, and for trajectory pattern discovery. The vignette provides examples.

Science

benthos v1.3-4: Provides preprocessing tools and biodiversity measures for analyzing marine benthic data. See Van Loon et al. (2015) for an application and the vignette for an introduction to the package.

nlmixr v0.9.0-1: Provides functions to fit and compare nonlinear mixed-effects models in differential equations with flexible dosing information commonly seen in pharmacokinetics and pharmacodynamics. See Almquist et al. (2015). The vignette shows how to use the package.

PCRedux v0.2.5-1: Provides functions to extract Polymerase Chain Reactions (qPCR) amplification curve data for machine learning purposes. For details see Pabinger et al.(2014) and the vignette.

PDN v0.1.0: Provides tools for building patient level networks for predicting medical outcomes based on the paper by Cabrera et al. (2016). See the vignette for an introduction.

Rraven v1.0.0: Provides a tool to exchange data between R and Raven sound analysis software. The vignette shows how to use the software.

spew v1.3.0: Provides tools for generating Synthetic Populations and Ecosystems. See Gallagher et al. (2017) for details and the vignette for a brief tour.

Statistics

[bang(https://CRAN.R-project.org/package=bang)] v1.0.0: Provides functions for the Bayesian analysis of some simple common models, without using Markov Chain Monte Carlo (MCMC) methods such as Gibbs sampling. There is an Introduction and vignettes on Hierarchical 1-way ANOVA, Conjugate Hierarchical Models and Posterior Predictive Checking.

beast v1.0: Provides a method for the Bayesian estimation of change-points in the slope of multivariate time series. See Papastamoulis et al (2017) for a detailed presentation of the method.

CorShrink v0.1.1: Offers functions to perform adaptive shrinkage of correlation and covariance matrices using a mixture model prior over the Fisher z-transformation of the correlations. See Stephens (2016) for details. The vignette contains examples.

dvmisc v1.1.1: Provides faster versions of base R functions (e.g. mean, standard deviation, covariance, weighted mean), mostly written in C++, along with other miscellaneous functions.

inlabru v2.1.2: Facilitates spatial modeling using integrated nested Laplace approximation via the INLA package and also implements a log Gaussian Cox process likelihood for modeling univariate and spatial point processes. Yuan et al. (2017).

outbreaker2 v1.0-0: Allows users to reconstruct disease outbreaks using epidemiological and genetic information. See Jombart et al. (2014) for the details. There is a package Overview as well as an Introduction and vignettes on Using Custom Priors and The Rcpp API.

probout v1.0: Provides functions to estimate unsupervised outlier probabilities for multivariate numeric data with many observations from a nonparametric outlier statistic. There is a vignette.

quokar v0.1.0: Provides diagnostics for quantile regression models including detecting influential observations, robust distance methods, generalized Cook’s distance and Q-function distance (see Benites et al. (2015)) and mean posterior probability and Kullback–Leibler divergence methods (see Santos & Bolfarine (2016)). The vignette introduces the package.

tidyposterior v0.0.1: This memorably named package implements a Bayesian approach for examining the differences between models and aims to answer the question: “When looking at resampling results, are the differences between models real?” The methods included are similar to those described in Benavoli et al (2017). There is a Getting Started Guide and a vignette on Bayesian Models.

trialr v0.0.1: Offers a showcase of Bayesian clinical trial designs, implemented in RStan and R including some designs implemented in R for the first time (e.g. EffTox’ by Thall & Cook (2004). There are vignettes on the BEBOP Design, EffTox and Hierarchical Bayesian Models for Binary Responses.

tvReg v0.2.1: Provides functions for fitting simultaneous equations with time varying coefficients, for both independent and correlated equations. The vignette contains examples.

Utilities

cli v1.0.0: Implements a suite of tools designed to build attractive command line interfaces. Includes tools for drawing rules, boxes, trees, and ‘Unicode’ symbols with ‘ASCII’ alternatives. See README for details.

float v0.1-1: Extends R’s linear algebra facilities to include 32-bit float (single precision) data. There is a vignette.

mudata2 v1.0.0: Offers functions and data structures designed to easily organize and visualize spatiotemporal data. There are vignettes for usinng and creating mudata2 objects.

rhub v1/0.2: Provides an interface to the R-Hub package build system sponsored by the R Consortium. Run R CMD check on Windows, macOS, Solari and various Linux distributions.

Visualizations

ALEPlot v1.0: Offers functions to visualizes the main effects of individual predictor variables and their second-order interaction effects in black-box supervised learning models. The vignette contains several examples.

dbplot v0.1.1: Leverages dplyr to process the calculations for a plot inside a database. Helper functions abstract the work at three levels: outputs the ggplot, the calculations and the formula needed to calculate bins. See README to get started.

ggalluvial v0.5.0: Implements ggplot2 stat and geom layers for alluvial diagrams, charts that use x-splines (alluvia and flows), sometimes augmented with stacked bars (lodes or strata), to visualize incidence structures derived from several data types. The vignette provides examples.

shinyaframe v1.0.1: Enables users to make R data available in Web-based virtual reality experiences for immersive, cross-platform data visualizations. It provides functions to create 3-dimensional data visualizations with Mozilla A-Frame. The vignette shows how.

tactile v0.1.0: Extends lattice, providing new high-level functions, methods for existing functions, panel functions, and a theme. There are vignettes for New High-Level Functions, New Methods for Lattice and the tactile Theme.

Connecting R to Keras and TensorFlow

Mon, 11 Dec 2017 00:00:00 +0000

It has always been the mission of R developers to connect R to the “good stuff”. As John Chambers puts it in his book Extending R:

One of the attractions of R has always been the ability to compute an interesting result quickly. A key motivation for the original S remains as important now: to give easy access to the best computations for understanding data.

From the day it was announced a little over two years ago, it was clear that Google’s TensorFlow platform for Deep Learning is good stuff. This September (see announcment), J.J. Allaire, François Chollet, and the other authors of the keras package delivered on R’s “easy access to the best” mission in a big way. Data scientists can now build very sophisticated Deep Learning models from an R session while maintaining the flow that R users expect. The strategy that made this happen seems to have been straightforward. But, the smooth experience of using the Keras API indicates inspired programming all the way along the chain from TensorFlow to R.

The Keras Strategy

TensorFlow itself is implemented as a Data Flow Language on a directed graph. Operations are implemented as nodes on the graph and the data, multi-dimensional arrays called “tensors”, flow over the graph as directed by control signals. An overview and some of the details of how this all happens is lucidly described in a paper by Abadi, Isard and Murry of the Google Brain Team,

and even more details and some fascinating history are contained in Peter Goldsborough’s paper, A Tour of TensorFlow.

This kind of programming will probably strike most R users as being exotic and obscure, but my guess is that because of the long history of dataflow programming and parallel computing, it was an obvious choice for the Google computer scientists who were tasked to develop a platform flexible enough to implement arbitrary algorithms, work with extremely large data sets, and be easily implementable on any kind of distributed hardware including GPUs, CPUs, and mobile devices.

The TensorFlow operations are written in C++, CUDA, Eigen, and other low-level languages optimized for different operation. Users don’t directly program TensorFlow at this level. Instead, they assemble flow graphs or algorithms using a higher-level language, most commonly Python, that accesses the elementary building blocks through an API.

The keras R package wraps the Keras Python Library that was expressly built for developing Deep Learning Models. It supports convolutional networks (for computer vision), recurrent networks (for sequence processing), and any combination of both, as well as arbitrary network architectures: multi-input or multi-output models, layer sharing, model sharing, etc. (It should be pretty clear that the Python code that makes this all happen counts as good stuff too.)

Getting Started with Keras and TensorFlow

Setting up the whole shebang on your local machine couldn’t be simpler, just three lines of code:

install.packages("keras")
library(keras)
install_keras()

Just install and load the keras R package and then run the keras::install_keras() function, which installs TensorFlow, Python and everything else you need including a Virtualenv or Conda environment. It just works! For instructions on installing Keras and TensorFLow on GPUs, look here.

That’s it; just a few minutes and you are ready to start a hands-on exploration of the extensive documentation on the RStudio’s TensorFlow webpage tensorflow.rstudio.com, or jump right in and build a Deep Learning model to classify the hand-written numerals using

MNIST data set which comes with the keras package, or any one of the other twenty-five pre-built examples.

Beyond Deep Learning

Being able to build production-level Deep Learning applications from R is important, but Deep Learning is not the answer to everything, and TensorFlow is bigger than Deep Learning. The really big ideas around TensorFlow are: (1) TensorFlow is a general-purpose platform for building large, distributed applications on a wide range of cluster architectures, and (2) while data flow programming takes some getting used to, TensorFlow was designed for algorithm development with big data.

Two additional R packages make general modeling and algorithm development in TensorFlow accessible to R users.

The tfestimators package, currently on GitHub, provides an interface to Google’s Estimators API, which provides access to pre-built TensorFlow models including SVM’s, Random Forests and KMeans. The architecture of the API looks something like this:

There are several layers in the stack, but execution on the small models I am running locally goes quickly. Look here for documentation and sample models that you can run yourself.

At the deepest level, the tensorflow package provides an interface to the core TensorFlow API, which comprises a set of Python modules that enable constructing and executing TensorFlow graphs. The documentation on the package’s webpage is impressive, containing tutorials for different levels of expertise, several examples, and references for further reading. The MNIST for ML Beginners tutorial works through the classification problem described above in terms of the Keras interface at a low level that works through the details of a softmax regression.

While Deep Learning is sure to capture most of the R to TensorFlow attention in the near term, I think having easy access to a big league computational platform will turn out to be the most important benefit to R users in the long run.

As a final thought, I am very much enjoying reading the MEAP from the forthcoming Manning Book, Deep Learning with R by François Chollet, the creator of Keras, and J.J. Allaire. It is a really good read, masterfully balancing theory and hands-on practice, that ought to be helpful to anyone interested in Deep Learning and TensorFlow.

October 2017 New Packages

Wed, 22 Nov 2017 00:00:00 +0000

Of the 182 new packages that made it to CRAN in October, here are my picks for the “Top 40”. They are organized into eight categories: Engineering, Machine Learning, Numerical Methods, Science, Statistics, Time Series, Utilities and Visualizations. Engineering is a new category, and its appearance may be an early signal for the expansion of R into a new domain. The Science category is well-represented this month. I think this is the result of the continuing trend for working scientists to wrap their specialized analyses into R packages.

Engineering

FlowRegEnvCost 0.1.1: Calculates the daily environmental costs of river-flow regulation by dams based on García de Jalon et al. (2017).

rroad v0.0.4: Computes and visualizes the International Roughness Index (IRI) given a longitudinal road profile for a single road segment, or for a sequence of segments with a fixed length. For details on The International Road Roughness Experiment establishing a correlation and a calibration standard for measurements, see the World Bank technical paper. The vignette shows an example of a Road Condition Analysis. The following scaleogram was produced from a continuous wavelet transform of a 3D accelerometer signal.

Machine Learning

detrendr v0.1.0: Implements a method based on an algorithm by Nolan et al. (2017) for detrending images affected by bleaching. See the vignette

MlBayesOpt v0.3.3: Provides a framework for using Bayesian optimization (see Shahriari et al. to tune hyperparameters for support vector machine, random forest, and extreme gradient boosting models. The vignette shows how to set things up.

rerf v1.0: Implements an algorithm, Random Forester (RerF), developed by Tomita (2016), which is similar to the Random Combination (Forest-RC) algorithm developed by Breiman (2001). Both algorithms form splits using linear combinations of coordinates.

Numerical Methods

episode v1.0.0: Provides statistical tools for inferring unknown parameters in continuous time processes governed by ordinary differential equations (ODE). See the Introduction.

KGode v1.0.1: Implements the kernel ridge regression and the gradient matching algorithm proposed in Niu et al. (2016), and the warping algorithm proposed in Niu et al. (2017) for improving parameter estimation in ODEs.

Science

adjclust v0.5.2: Implements a constrained version of hierarchical agglomerative clustering, in which each observation is associated with a position, and only adjacent clusters can be merged. The algorithm, which is time- and memory-efficient, is described in Alia Dehman (2015). There are vignettes on Clustering Hi-C Contact Maps, Implementation Notes, and Inferring Linkage Disequilibrium blocks from Genotypes.

hsdar v0.6.0: Provides functions for transforming reflectance spectra, calculating vegetation indices and red edge parameters, and spectral resampling for hyperspectral remote sensing and simulation. The Introduction offers several examples.

mapfuser v0.1.2: Constructs consensus genetic maps with LPmerge (See Endelman and Plomion (2014)) and models the relationship between physical distance and genetic distance using thin-plate regression splines (see Wood (2003)). The vignette explains how to use the package.

mortAAR v1.0.0: Provides functions for the analysis of archaeological mortality data See Chamberlain (2006). There is a vignette on Lifetables and an Extended Discussion.

skyscapeR v0.2.2: Provides a tool set for data reduction, visualization and analysis in skyscape archaeology, archaeoastronomy and cultural astronomy. The vignette shows how to use the package.

Statistics

BayesRS v0.1.2: Fits hierarchical linear Bayesian models, samples from the posterior distributions of model parameters in JAGS, and computes Bayes factors for group parameters of interest with the Savage-Dickey density ratio ([See Wetzels et al.(2009). There is an Introduction.

CatPredi v1.1: Allows users to categorize a continuous predictor variable in a logistic or a Cox proportional hazards regression setting, by maximizing the discriminative ability of the model. See Barrio et al. (2015) and Barrio et al. (2017).

CovTools v0.2.1: Provides a collection of geometric and inferential tools for convenient analysis of covariance structures. For an introduction to covariance in multivariate statistical analysis, see Schervish (1987).

genlogis v0.5.0: Provides basic distribution functions for a generalized logistic distribution proposed by Rathie and Swamee (2006).

emmeans v0.9.1: Provides functions to obtain estimated marginal means (EMMs) for many linear, generalized linear, and mixed models, and computes contrasts or linear functions of EMMs, trends, and comparisons of slopes. There are twelve vignettes including The Basics, Comparisons and Contrasts, Confidence Intervals and Tests, Interaction Analysis, and Working with Messy Data.

ESTER v0.1.0: Provides an implementation of sequential testing that uses evidence ratios computed from the Akaike weights of a set of models. For details see Burnham & Anderson (2004). There is a vignette.

FarmTest v1.0.0: Provides functions to perform robust multiple testing for means in the presence of latent factors. It uses Huber’s loss function to estimate distribution parameters and accounts for strong dependence among coordinates via an approximate factor model. See Zhou et al.(2017) for details. There is a vignette to get you started.

miic v0.1: Implements an information-theoretic method which learns a large class of causal or non-causal graphical models from purely observational data, while including the effects of unobserved latent variables, commonly found in many datasets. For more information see Verny et al. (2017).

modcmfitr v0.1.0: Fits a modified version of the Connor-Mosimann distribution ( Connor & Mosimann (1969)), a Connor-Mosimann distribution, or a Dirichlet distribution to elicited quantiles of a multinomial distribution. See the vignette for details.

pense v1.0.8: Provides a robust penalized elastic net S and MM estimator for linear regression as described in Freue et al. (2017).

paramtest v0.1.0: Enables running simulations or other functions while easily varying parameters from one iteration to the next. The vignette shows how to run a power simulation.

rENA v0.1.0: Implements functions to perform epistemic network analysis ENA, a novel method for identifying and quantifying connections among elements in coded data, and representing them in dynamic network models, which illustrate the structure of connections and measure the strength of association among elements in a network.

rma.exact v0.1.0: Provides functions to compute an exact CI for the population mean under a random-effects model. For details, see Michael, Thronton, Xie, and Tian (2017).

Time Series

carfima v1.0.1: Provides a toolbox to fit a continuous-time, fractionally integrated ARMA process CARFIMA on univariate and irregularly spaced time-series data using a general-order CARFIMA(p, H, q) model for p>q as specified in Tsai and Chan (2005).

colorednoise v0.0.1: Provides tools for simulating populations with white noise (no temporal autocorrelation), red noise (positive temporal autocorrelation), and blue noise (negative temporal autocorrelation) based on work by Ruokolainen et al. (2009). The vignette describes colored noise.

nnfor v0.9: Provides functions to facilitate automatic time-series modelling with neural networks. Look here for some help getting started.

Utilities

hdf5r v1.0.0: Provides an object-oriented wrapper for the HDF5 API using R6 classes. The vignette shows how to use the package.

geoops v0.1.2: Provides tools for working with the GeoJSON geospatial data interchange format. There is an Introduction.

linl v0.0.2: Adds a LaTeX Letter class to rmarkdown, using the pandoc-letter template adapted for use with markdown. See the vignette and README for details.

oshka v0.1.2: Expands quoted language by recursively replacing any symbol that points to quoted language with the language itself. There is an Introduction and a vignette on Non Standard Evaluation Functions.

rcreds v0.6.6: Provides functions to read and write credentials to and from an encrypted file. The vignette describes how to use the package.

RMariaDB v1.0-2: Implements a DBI-compliant interface to MariaDB and MySQL databases.

securitytxt v0.1.0: Provides tools to identify and parse security.txt files to enable the analysis and adoption of the Web Security Policies draft standard.

usethis v1.1.0: Automates package and project setup tasks, including setting up unit testing, test coverage, continuous integration, Git, GitHub, licenses, Rcpp, RStudio projects, and more that would otherwise be performed manually. README provides examples.

xltabr v0.1.1: Provides functions to produce nicely formatted cross tabulations to Excel using [openxlsx]((https://cran.r-project.org/package=openxlsx), which has been developed to help automate the process of publishing Official Statistics. Look here for documentation.

Visualizations

iheatmapr v0.4.2: Provides a system for making complex, interactive heatmaps. Look at the webpage for examples.

otvPlots v0.2.0: Provides functions to automate the visualization of variable distributions over time, and compute time-aggregated summary statistics for large datasets. See the README for an introduction.