Welcome!

We used to have a language war, then peaceful polyglot co-existence. Now, it’s clear that we’re actively better off learning from each other. A reflection on 2025, posit::conf, what R and python stacks can learn from each other, and the criticality of diversity in open source

Polars provides a consistent API for conducting transformations against a DataFrame. But what do you do when you need to apply a user-defined function beyond the native API? This post surveys the patterns for transformation and aggregation with UDFs, including vector / object output, expression expansion, and when to break the abstraction with partitions and list comprehensions

Playing with the potential, perils, and design principles of deploying ML models into the analytical database using orbital’s sklearn-to-sql translation, sqlglot, and dbt

Literate programming excels at capturing our stream of conscience. Our stream of conscience does not excel at explaining the impact of our work. Notebooks enable some of data scientists’ worst tendencies in technical communication, but Quarto’s embed feature bridges the gap between reproducible research and resonate story-telling.

The orbital package offers an interface for translating a fitted SciKitLearn pipeline to pure SQL for scaling predictions. In this tech note, I explore how this framework can (mostly) be used for xgboost models, as well, with a bit of wrangling (and a few limitations).

Switching languages is about switching mindsets - not just syntax. New developments in python data science toolings, like polars and seaborn’s object interface, can capture the ‘feel’ that converts from R/tidyverse love while opening the door to truly pythonic workflows. (Updated from 2025 for new tools).

A quick tech note on Netlify’s managed authentication solution

Data teams may struggle to quantify the benefits of good data documentation. But running countless ad hoc validation queries can incur both computational and cognitive cost.

Getting comfortable in a new language is more than the packages you use. Syntactic sugar in base python increases the efficiency, and aesthetics of python code in ways that R users may enjoy in packages like glue and purrr. This post collects a miscellaneous grab bag of tools for wrangling, formatting (f-strings), repeating (list comprehensions), faking data, and saving objects (pickle)

Data documentation isn’t a box to check; it’s an active member of your team with many jobs-to-be-done. In this cross-post with Select Star, I write about how effective documentation can be your data products’ developer advocate for users, project manager for developers, and chief of staff for data leaders

In this follow-up post to Python Rgonomics, we deep dive into some of the advanced data wrangling functionality in python’s polars package to see how it’s powertools like column selectors and nested data structures mirror the best of dplyr and tidyr’s expressive and concise syntax

Writing is thinking; documenting is planning and executing. In this cross-post with Select Star, I write about how teams can produce high-quality and maintainble documentation by smartly structuring planning and development documentation and effeciently recycling them into long-term, user-friendly docs

Switching languages is about switching mindsets - not just syntax. New developments in python data science toolings, like polars and seaborn’s object interface, can capture the ‘feel’ that converts from R/tidyverse love while opening the door to truly pythonic workflows

Five highlights and links to select talks

Proactive collection of data to comply or confront assumptions

Rounding out my three-part ETL series form Airbyte’s developer blog

Cross-post from guest post on Airbyte’s developer blog

After a prior post on the merits of grouped data quality checks, I demo my newly merged implementation for dbt

Out-of-memory processing of North Carolina’s voter file with DuckDB and Apache Arrow

How we do (or don’t) think about null values and why the polyglot push makes it all the more important

After a prior post on the merits of grouped data quality checks, I demo my newly merged implementation for dbt

Key issues when adding persistent storage to a Shiny application, featuring {golem} app development and Digital Ocean serving

Much like ice sculpting, applying powertools to absolutely frivolous pursuits

Which of these numbers doesn’t belong? -1, 0, 1, NA. You can’t judge data quality without data context, so our tools should enable as much context as possible.

A personal encounter with ‘intelligent’ data products gone wrong

Following up on ‘Embedding Column-Name Contracts… with dbt’ to demo my new dbtplyr package to further streamline the process

A right-sized solution to automated data monitoring, alerting, and reporting using R (pointblank, projmgr), GitHub (Actions, Pages, issues), and Slack

A data consumer’s guide to validating data based on the failure modes data producer’s try to avoid

Exploring how Playwright‘s headless browser automation (and its friends) can help unite the states’ data

dbt supercharges SQL with Jinja templating, macros, and testing – all of which can be customized to enforce controlled vocabularies and their implied contracts on a data model

An informal primer to causal analysis designs and data structures

Free books, lectures, blogs, papers, and more for a causal inference crash course

On the jobs-to-be-done and design principles for internal tools

An R package for maintaining controlled vocabularies to encode contracts between data producers and consumers

Marketing maintenance work with irrational exuberance

An introduction to browser-based interactivity of htmlwidgets – no Shiny server required!

Using controlled dictionaries for low-touch documentation, validation, and usability of tabular data

Don’t believe the documentation! Shiny modules aren’t just for advanced users; they might just be a great entry point for development

Readings and assorted ideas about creating and maintaining low-overhead documentation

A few tips and tools for finding the right selectors to style in RMarkdown

A walkthrough of using the projmgr package for GitHub-based project management via R

A recommended tech stack for implementing RMarkdown Driven Development

Case studies of the impact of R use on organizational culture and collaboration

An annotated bibliography of advice for getting started with reproducible research

A walkthrough of a GitHub template for making your own RMarkdown and ggplot2 theme package

Conference planning tips to design a good speakers experience

A workflow for refactoring one-time analyses to sustainable data products

A proposed workflow for methodically developing a good presentations

A Posit podcast hosted by Michael Chow, Hadley Wickham, and Wes McKiney. We discuss a wide range of topics spanning polyglot workflows, syntactic sugar and the origins of column selectors, and much more.

Casual Inference is a podcast on all things epidemiology, statistics, data science, causal inference, and public health. Sponsored by the American Journal of Epidemiology. As a guest on this episode, I discuss data science communication, the different challenges of causal analysis in industry versus academia, and much more.

A tour of data pipeline techniques and tools for use in academia

A survey of modern python tooling that “feels good” to R users

Panel discussion with Chad Sanderson and Joe Reis, hosted by Monte Carlo Data, on our thorniest brushes with data downtime, leading data teams to tackle data quality at scale with testing, contracts, observability and monitoring, and more.

An exploration of how data producers and consumers can use column names as interfaces, configuations, and code to improve data quality and discoverability. The second half of the talk demonstrates how to implement these ideas with my dbtplyr dbt package.

An overview of the data stack used to automate over 50,000 personalized emails to voter turnout volunteers using BigQuery, dbt, Census, and MailChimp

An overview of basic research design patterns in causal inference, modern extensions, and data management strategies to set up a causal inference initiative for success

Joined a panel of speakers to discuss tips and tricks for running dbt at scale

Each week, host Rachael Dempsey invites an accomplished data science leader to talk about their experience and answer questions from the audience. The discussion focuses mainly on the human elements of data science leadership. There’s no sales or marketing fluff, just great insights from inspiring professionals.

An introduction to inverse propensity of treatment weighting for program evaluation with applications to Two Million Texans’ relational organizing campaign during the 2022 midterms

An argument for the individual and organization-wide benefits of teaching new developers Shiny with a modules-first paradigm.

Interrogating the data generating process to devise better data quality tests.

Discussing how column names can serve as a light-weight alternative to data catalogs and contracts and how to implement this approach with dbtplyr

Discussing how to move from scripting to tool development, designing tools in enterprise, and navigating diverse data career paths

Exploring the benefits of using controlled vocabularies to encode metadata in column names, and demonstrations of implementing this approach with the convo R package or dbt extensions of SQL.

An overview of the unique design challenges and opportunities when building R packages for use inside of a single organization versus open-source. By using the jobs-to-be-done framework, this talk explores how internal packages can be better teammates by following specific design patterns for API design, testing, documentaiton, and more.

A lightning talk on key features of the projmgr package which brings enables code-based planning and reporting workflows grounded in GitHub issues and milestones

How and why to refactor one time analyses in RMarkdown into sustainable data products

A variety of related talks to creating innersource culture with R packages and related tools

A case study on building an internal R package for customer lifetime value modeling at Capital One and leading broader analyst adoption of open-source tooling and reproducible workflows through a community of practice.

We screened over 1,000 articles from 18 high-profile journals published from 2010-2019 to review the use of causal language to describe results in experimental and observational data.

With this in-depth book, data engineers will learn powerful, real-world best practices for managing data—both big and small. Contributors from companies including Google, Microsoft, IBM, Facebook, Databricks, and GitHub share their experiences and lessons learned on cleaning, prepping, wrangling, and storing data. I contributed 6 of the 97 essays.

This book is designed to provide a range of examples of how to extend the functionality of your R Markdown documents. As a cookbook, this guide is recommended to new or intermediate R Markdown users who desire to enhance the efficiency of using R Markdown and also explore the power of R Markdown. Read online at https://bookdown.org/yihui/rmarkdown-cookbook/