Kin Kitchen

Kin Kitchen — Indigenous Food Reconnection

Kin Kitchen reconnects people with Indigenous food traditions through AI-powered ingredient recognition, traditional recipes, and immersive 3D/AR experiences. Upload a photo or video of your kitchen or ingredients; the app identifies food, surfaces Indigenous context (e.g. Ojibwe names, cultural uses), suggests recipes from an Indigenous recipes dataset, and guides you through cooking in a step-by-step 3D kitchen with voice (say “next” or “continue”) and TTS.

Our story

What inspired us

We wanted to help people reconnect with Indigenous foodways—not as a history lesson, but as something you can do in your own kitchen. So many of us are disconnected from where our food comes from and from the knowledge that Indigenous cultures have carried for generations. We were inspired by the Three Sisters (squash, beans, corn) and by the idea that a photo of your counter could become a doorway: to Ojibwe names, to traditional recipes, and to a guided cooking experience that feels both modern (AI, 3D, voice) and rooted. We built Kin Kitchen so that “what’s in my kitchen?” leads to “how do I cook this with respect and care?”

What we learned

We learned how to weave multiple AI and media services into one flow: Cloudinary for storage and object detection (LVIS), Gemini for Indigenous context and analysis, ElevenLabs for natural-sounding step-by-step voice. We learned how to structure a 3D cooking tutorial as a sequence of small, scripted “step modules” (chop, dice, mince, pour, boil, add veggies, add beans & corn, stir and reveal) so the experience stays smooth and predictable. We also learned how important it is to keep Indigenous context accurate and respectful—surfacing nation and territory, using recipe and story data carefully, and letting the food and the process lead.

How we built it

We started with a Node + Express backend and a vanilla JS frontend (no framework) so we could move fast and keep the stack simple. The upload pipeline: Multer → Cloudinary (upload + LVIS bounding boxes) → Gemini (analysis + cultural context) → recipe-matcher (ingredient list vs. indigenous-recipes.json). The frontend is a single-page app: splash (with a 3D hero), nation picker, upload, detection, recipe list with a “View 3D Kitchen” button, and the 3D kitchen screen. The kitchen is a Three.js scene that loads GLB models for ingredients, a cutting board, pot, and finished stew; CookingGuide runs an 8-step overlay and coordinates step modules (Step1Chop through Step8Stir). Each step can speak via ElevenLabs TTS, and we added voice control so users can say “next” or “continue” to advance. Story narration and AR storyboard (CookingAR) round out the experience.

Challenges we faced

Integrating many APIs — Cloudinary, Gemini, ElevenLabs (and optionally Hugging Face) had different auth, rate limits, and response shapes. We had to normalize errors, handle missing keys gracefully, and keep the UI responsive while waiting on external calls.
3D pipeline and step choreography — Making chop, dice, mince, pour, boil, and stir feel consistent meant building separate step modules with clear phases (e.g. board in, ingredient to board, knife animation, pile reveal, board out). Syncing visibility and positions (cutting board, pot, stew) across steps and keeping performance smooth was tricky.
Merging different versions — We had one branch focused on recipe list, 3D button, and voice, and another with the “finish” (pot, stew, full 8-step flow). Combining them required carefully bringing over new assets and step modules without breaking the existing flow.
Voice and accessibility — We wanted “Listen” and voice “next/continue” to work across browsers; we had to handle Web Speech API support and fallbacks and keep TTS and UI in sync.

Quick start

cp .env.example .env   # add your API keys (see below)
npm run install:backend
npm run dev







video-analyzer/
├── backend/
│   ├── server.js
│   ├── routes/          # upload, story, hf, generate3d
│   ├── services/        # cloudinary, gemini, elevenlabs, recipe-matcher, food-filter, crop, …
│   ├── data/            # indigenous-recipes.json, recipe-stories.json
│   └── uploads/
├── frontend/
│   ├── index.html
│   ├── styles.css, kin-ui.css
│   └── js/
│       ├── app.js           # KinKitchenApp: goTo(), state, nation, nav
│       ├── upload.js        # POST /api/upload, progress, 3D vs detect
│       ├── render.js        # KinRender: renderDetection, renderRecipe, renderStory, renderWord
│       ├── generate3d.js    # Three.js scene, GLBs, CookingGuide, WebXR
│       ├── cookingGuide.js  # 8-step overlay, TTS, voice “next/continue”
│       ├── ingredientPositions.js
│       ├── steps/           # Step1Chop, Step2Chop, Step3Garlic, Step4Stock, Step5Boil,
│       │                    # Step6Veggies, Step7BeansCorn, Step8Stir
│       ├── story.js, ar.js, hero3d.js, glow.js, tiltCard.js, …
│       └── …
├── assets/3d/           # *.glb (ingredients, pot, stew, minced-garlic, cutting-board, …)
├── .env.example, .env
├── package.json
└── README.md