When Models See Ghosts - Investigating Why Adversarial Examples Break Our Models
Explore the trade-offs between accuracy, simplicity, and the chaos of high-dimensional data.
Howdy 🤠! This blog is my attempt at unpacking why adversarial examples exist and at piecing together why models are often wrong almost everywhere. I've spent an unhealthy number of hours reading arXiv papers (and should probably spend hundreds more). I'm not claiming this is the gospel truth. I've put together what makes sense to me & I'll do my best to explain manifolds, dimensions, dimples, and a bunch of other data geometry stuff as clearly as I can.
I've sat on publishing this for ~4 months now 🫠. Truth is, I feel awkward posting something like this in a field that's academically published this much. There's a constant flow of new defenses & research that shakes up how everyone thinks these systems work (Robbinson et al., 2025 😅). I wrote this back in late January & A LOT has changed since then. I want to emphasize that there's no single explanation that captures it all & not everyone sees it the same way. Consider this a snapshot in time of how I and perhaps others understand these vulnerabilities.
NOTE: This blog provides an overview of the broader field, drawing on the work of many others. I've chosen to emphasize concepts rather than a full history of who did what. I don’t have the luxury of 2,000 GPUs💔 while I've been able to make certain observations independently and through collaboration, I recognize that my understanding of these ideas has been shaped & influenced by the research of many people.
I don't have any real academic education or a PhD. Expect that I might abstract and oversimplify certain concepts. If for some reason this blog upsets you, refer to my Content Policy. I've included a messy/unsorted list of reading materials that have taught, shaped, and challenged my understanding → here.
I read a few papers a week. The best way I've found to stay up-to-date with the right arXiv papers is with Dreadnode's Paperstack instance. I find it better than setting up/using arxival, pasa, etc. It's free, I love it, and you might like it too.