Inspiration
Music producers face a fundamental disconnect between musical intent and DAW execution. When they think "make the bass line more syncopated" or "add a ii-V-I turnaround in bars 9 through 12," they have to manually translate that into dozens of precise mouse clicks in a piano roll. It's slow, it breaks creative flow, and it creates a steep skill barrier for newer producers.
Existing AI music tools don't solve this. Generative tools like Suno and Udio produce finished audio from prompts so you lose all creative control and can't edit what comes out. DAW assistants like DAWZY and Ableton Copilot MCP handle DAW level operations like track management and FX routing, but they never touch the actual notes. The gap we kept seeing was that no tool lets a producer conversationally edit MIDI data inside their real DAW with note-level precision.
We wanted to essentially build Cursor for music production - an AI copilot that sits alongside REAPER and translates musical intent into precise MIDI operations, while the producer stays in their familiar workflow.
What it does
Reaper AI is a chat based sidecar app that runs alongside REAPER. You describe what you want: a genre, a mood, an emotion, Twinkle Twinkle Little Star, and the AI builds a full production in real time including chord progressions, bass lines, drum patterns, melodies, instrument plugins, and a complete FX chain. It understands musical context, so "dark cinematic beat" produces a different key and feel than "happy pop track."
It also ships a real audio to MIDI transcription tool powered by Spotify's basic pitch neural network. Drop in any audio file and it transcribes the pitches directly into editable MIDI notes on a new REAPER track.
How we built it
The core challenge was getting an AI to communicate with REAPER, which has no native external API. We built a bridge where a Lua script running inside REAPER communicates with a Python server via HTTP and a shared file directory. The Python server translates requests into structured tool calls and writes responses back. This gave us reliable communication without needing a socket server or ReaPy dependency.
On the AI side, we use Gemini 2.5 Flash via litellm with a full agentic tool calling loop. The model receives the current project state at the start of each turn and can call over 30 tools to modify the DAW. Rather than having the LLM guess at music theory, we built a custom music theory engine in Python that generates humanized MIDI note lists for chords, bass lines, drum patterns, and melodies. The AI calls these tools and passes the results directly to REAPER where they are populated. For audio transcription, we run Spotify's basic-pitch ONNX model in a Python 3.12 subprocess, convert the detected note events from seconds to beats, and drop them onto a new REAPER track. The UI is a pywebview window rendering an HTML/CSS/JS chat panel that stays on top of REAPER.
Challenges we ran into
The bridge, not the AI, was the hardest part. REAPER has no standard external API, so we built the file based bridge from scratch. Getting request and response timings reliable without miscommunication between the Lua watcher and the Python server took significant debugging. We also went through two model switches, starting on Backboard.io, hitting rate limits, moving to OpenAI, and ultimately landing on Gemini 2.5 Flash which gave us the best results for structured tool calling at the speed we needed.
The AI also kept entering delete and restart loops. When it encountered an existing project state it didn't expect, it would delete all tracks and try to rebuild from scratch, repeatedly. Fixing this required careful system prompt engineering.
Cross platform compatibility was also a problem we solved relatively early. The project was originally built on Windows. Getting the project to work on macOS and windows required doing things such as fixing every hardcoded path, and replacing Windows only screen size APIs with those that worked on both kinds of machine.
Accomplishments that we're proud of
The music actually sounds good. The theory engine generates actual music and not just random notes. We're also proud that it runs inside a real professional DAW rather than a toy sandbox. You start from a blank project, describe a beat, and have a fully structured production a couple of minutes.
The tone to key mapping was also nice touch where telling the system you want something "spooky" or "dark" now picks minor keys like Dm or Em rather than defaulting to whatever the genre suggests. Musical intent actually goes all the way down to note selection.
What we learned
Prompt engineering in agentic systems is hard! Small ambiguities like not explicitly saying "always append tracks to the end" would cause the model to behave unpredictably in ways that were hard to debug. Every constraint the model needs to follow has to be stated explicitly and redundantly.
We also learned that music theory can't be left to the AI. Models can't yet reliably produce correct MIDI ingredients. The music theory engine doing most of the work and the AI just deciding what to generate was the right way to do things.
What's next for Reaper AI
The MP3 to MIDI transcription currently generates patterns from file properties rather than actual audio analysis so implementing better analysis for that is an immediate next step. We would also like to add support for a greater variety of instruments in REAPER, so almost anything can be composed with Reaper AI. Beyond that, we want to add better intelligence, having the AI structure full songs well rather than smaller pieces. Longer term, we would like to implement voice interface so producers can describe changes while they're listening, and support for additional DAWs beyond REAPER.
Log in or sign up for Devpost to join the conversation.