Inspiration
Yesterday, millions of shoppers walked out of stores empty-handed. For an average retailer with 1,000 daily visitors, that's tens of thousands of dollars in lost conversion down the drain. If pointicart existed before, those sales could have been saved. Retail shoppers repeatedly spend time rediscovering the same physical friction patterns: searching racks for their size, visualizing how an item looks, waiting in fitting room lines, and standing at registers—problems that e-commerce solved thousands of times over. As physical retail scales globally, this offline friction remains one of the largest sources of inefficiency in the consumer economy. We built pointicart from a simple insight: real-world shopping workflows are digital assets waiting to happen, and the ecosystem needs an intelligent bridge where consumers can discover, visualize, and execute checkouts instantly by pointing instead of waiting every time.
What it does
pointicart is an intelligent spatial commerce platform where shoppers autonomously identify, try on, and purchase physical products simply by pointing. Retailers can seamlessly digitize their catalogs, while consumers interact with them through a lightweight iOS application that exposes structured AR functionalities such as spatial identify, virtual try-on, and personalized auto-add to cart. Shoppers can therefore integrate pointicart directly into their real-world browsing loop, allowing them to instantly execute high-confidence checkout plans while keeping all sensitive interaction histories and engagement data local.
How we built it
We built pointicart as a full-stack spatial application natively deployed on iOS using SwiftUI and the Observation framework, coupled with a hybrid multimodal inference system powered by Google Cloud. A specialized tracking engine uses ARKit to decompose live camera frames and track fingertips, retrieve candidate product matches via Gemini visual inference, and recombine them into an executable spatial UI card that can be instantly checked out. We also built an autonomous recommendation engine backed by an InteractionVectorStore, which calculates real-time engagement intensity from dwell times and add-to-cart frequencies to surface high-confidence suggestions, alongside a Virtual Try-On pipeline leveraging Vertex AI endpoints to generate seamless digital fittings directly on the fly.
Challenges we ran into
One of the main challenges was achieving high-precision visual product matching while maintaining real-time interactive latency, which required carefully designing a spatial position caching layer and separating localized tracking bounding boxes from the semantic product catalog. Another challenge was building a reliable virtual try-on pipeline capable of handling complex apparel geometries where a standard camera snapshot was insufficient. Finally, designing a localized recommendation engine that fairly anticipates user intent without being obtrusive required implementing intensity-gated auto-add thresholds and time-weighted interaction tracking.
Accomplishments that we're proud of
Within the hackathon timeframe, we built a fully functional spatial commerce application, a live AR dwell-tracking interface, a working virtual try-on module, and a recommendation architecture capable of processing complex physical shopping behaviors into frictionless checkout flows. Our system demonstrated substantial improvements in identification latency, spatial context retention, and checkout efficiency across benchmark scenarios, validating that computer vision-powered spatial interactions can operate as a scalable interface for modern brick-and-mortar ecosystems.
What we learned
We learned that scalable spatial commerce systems depend on structured interaction artifacts rather than manual barcode scans, and that caching spatial object tracking asynchronously from cloud inference dramatically improves matching precision. We also learned that privacy-first execution, where interaction histories are vectorized entirely locally before triggering recommendations, is critical for the real-world adoption of AI-powered physical retail tools.
What's next for pointicart
Next, we plan to expand the retail ecosystem, introduce multi-store catalog verification pipelines, and deepen integrations across major payment frameworks so pointicart becomes a default physical-to-digital intelligence distribution layer for retailers. By combining spatial computing economics, verifiable AI try-on workflows, and seamless cloud-based multimodal identification, our goal is to enable a global retail economy where shoppers continuously engage with the best available products simply by pointing, instead of repeatedly browsing from scratch.
Log in or sign up for Devpost to join the conversation.