On this page
The Premise
PolyFish was built almost entirely through human-AI collaboration. Matt provided the creative vision, game design, and artistic direction. Claude (by Anthropic) handled the bulk of the code implementation - from translating a decade-old Unity prototype into modern JavaScript to building systems that neither of us had built before.
This wasn't a case of asking an AI to "make me a game." It was closer to an intensive pair programming sprint where one partner has deep domain knowledge and the other can write code very fast but needs constant steering. The result is roughly 20,000 lines of JavaScript across 63 source files, a full WebXR integration, a documentary camera system, and the Making Of site you're reading right now.
What follows is an honest accounting of how that process worked.
Our Approach
The Translation Layer
The project started as a port. Matt had a working Unity prototype from 2014 with C# scripts, HLSL shaders, and Unity-specific physics. Translating that into Three.js meant more than syntax conversion - Unity's component system, FixedUpdate loop, and rigidbody physics have no direct equivalents in a browser.
The approach was system-by-system. Matt would share the relevant C# code and describe the intended behavior. Claude would produce a JavaScript implementation, and the two would iterate until it felt right. The creature AI alone went through dozens of revisions before the fish moved the way Matt remembered.
The "Creative Director + Technical Lead" Dynamic
Over time, a natural rhythm emerged. Matt operated as creative director: deciding what to build, how it should feel, and when something was off. Claude operated as technical lead: proposing architectures, writing implementations, and debugging when things broke. Neither role was rigid - Matt would sometimes dive into code directly, and Claude would sometimes push back on a design direction with a technical argument.
The key insight was that this worked best when Matt described what he wanted to experience rather than what code to write. "The dolphins feel too robotic" was a better prompt than "change the steering algorithm." It gave Claude room to propose solutions Matt hadn't considered.
This experience-driven development approach unlocked better outcomes than detailed technical specifications. Instead of Matt being limited to the boundaries of his own technical knowledge or vocabulary, he could say "I want it to feel like a nature documentary" and Claude would figure out the camera system, timing, and transitions. Describing the desired experience meant the collaboration could reach further than either partner's individual constraints would allow.
What Worked
Rapid Prototyping of Complex Systems
Some of the most satisfying moments came from building systems that would have taken weeks to learn and implement solo. The documentary camera system is a good example: a Director that manages narrative phases, a Cinematographer that composes shots with 11 different shot types, and an Ecosystem Scout that detects interesting creature behavior to film. That's roughly 2,500 lines of interconnected state-machine code. Claude produced a working first draft in a single session, and Matt could immediately start tweaking shot feel instead of spending days getting a basic follow-cam working.
The caustic shader system followed the same pattern. Matt described the look he wanted for seabed light patterns, Claude produced a GLSL implementation using domain-warped Voronoi cells, and Matt tuned the colors and speed. The result looks better than anything either would have built independently.
The Port Itself
Translating Unity C# to Three.js JavaScript is a surprisingly nuanced job. Unity has a component system; Three.js has a scene graph. Unity has FixedUpdate at 50Hz; the browser has requestAnimationFrame at variable rates. Unity has built-in rigidbody physics; the browser has nothing unless you bring your own. Claude handled these translation decisions well because it could hold the mental model of both engines simultaneously and identify where a direct port would fail.
Verification as a Superpower
One of the most valuable practices was asking Claude to verify its own code - to check that the app was behaving according to its own expectations. This created a self-correcting loop where Claude could catch its own mistakes before they became bugs. Rather than relying solely on Matt's testing, having the AI reason through its own implementation and spot inconsistencies caught edge cases that might otherwise have shipped. It's a form of rubber-duck debugging elevated to a structural principle: the AI could debug itself.
Documentation as a Byproduct
This Making Of site was built collaboratively, and the process revealed something useful: because Claude had been involved in building every system, it could write accurate technical documentation about them. The interactive demos, architecture explanations, and code walkthroughs are all drawn from actual implementation knowledge rather than reverse-engineering.
Pain Points
Context Window Limits
The single biggest friction point. An AI can only hold so much code in its working memory at once, and PolyFish has 63 source files where changes to one system often ripple through several others. When Claude lost context on how the creature system's engine burn states connected to the instanced rendering pipeline, things broke in subtle ways. Re-sharing files and re-explaining architecture worked, but that overhead added up.
Long sessions were especially vulnerable. After hours of iteration, Claude would start making suggestions that contradicted earlier decisions. The solution was focused sprints - one system per session, with clear goals - rather than marathon conversations.
Overconfident Code Generation
Claude writes code quickly and confidently. That's great when it's right, and frustrating when it's wrong with the same confidence. A division by zero when no fish were alive. A NaN propagating through the physics system. A shader that compiled on desktop but failed on mobile GPUs. Each looked correct on review.
The lesson: AI-generated code needs the same rigor as any code. "It looks right" isn't the same as "it works."
The "Almost Right" Problem
Sometimes Claude would produce something 90% correct but subtly wrong in a way that took longer to diagnose than writing it from scratch would have. A Verlet constraint solver that was stable but too stiff. A camera transition that eased correctly but targeted the wrong position. An audio crossfade that faded out but never faded back in. The code was well-structured and readable - the kind you'd trust on visual inspection - which made the bugs harder to catch.
Regressions During Refactoring
Large refactors were risky. When Claude restructured a system, it would sometimes lose track of subtle behaviors the old code handled implicitly - a creature AI that forgot to check for dead targets, a rendering pipeline rework that dropped a caustic intensity uniform. Each regression was small, but they compounded during big changes.
Unexpected Wins
Systems Neither of Us Had Built Before
The most surprising outcome was how well the collaboration handled novel territory. Matt had never written a Verlet physics solver. The kelp system was uncharted for both of us - Matt described how kelp should behave physically, Claude proposed position-based dynamics, and iterative tuning of buoyancy, damping, and constraint stiffness produced something that genuinely looks organic.
The vertex-shader swim animation was another leap. Inspired by a GDC talk about Abzu, Matt wanted GPU-driven creature animation. Claude's first attempt was recognizable but stiff; after several rounds of tweaking amplitude curves and phase offsets, the fish undulate convincingly. Performance went from ~100 creatures with CPU animation to 200+ with the GPU approach.
Cross-Domain Problem Solving
The collaboration shines when a problem spans multiple domains. The narration system needed to coordinate audio timing, camera cuts, creature AI, and UI state - four separate systems working in concert. Claude could reason about all four simultaneously in a way that would have required constant context-switching for a solo developer.
Toolmaking for Collaboration
Another major win emerged from asking Claude to build tools that they'd then use together to dial in concepts. Instead of tweaking magic numbers deep in code, Claude would create interactive sliders, visualizers, and test harnesses that made the tuning process collaborative and visual. Building a caustic shader? Claude created a Voronoi explorer where Matt could adjust parameters in real-time and see results instantly. Tuning creature behavior? Tools let them visualize neural pathways and sensor ranges. These collaborative tools made iteration faster and the feedback loop tighter - neither partner was guessing anymore.
This Site
What started as a simple documentation exercise turned into a genuine technical writing project - interactive WebGL demos, magazine-style layout, deep dives into each system. Building it collaboratively meant the demos could be technically accurate (Claude built the original systems) and the writing could be editorially polished (Matt pushed for clarity and voice). It also served as a forcing function: explaining a system well enough to document it often revealed bugs or design improvements.
Accessibility
There's an aspect of this collaboration that doesn't fit neatly into a technical write-up, but it's one of the most important parts of the project for Matt personally.
Matt lives with autoimmune illness. Chronic pain and fatigue are baseline. Catching other illnesses is frequent and recovery is slow. On a bad day, the kind of sustained focus that complex systems programming demands - holding a mental model of interacting state machines, tracking down a NaN propagating through a physics pipeline, reasoning about race conditions in an audio system - is somewhere between difficult and impossible. Pain doesn't politely pause while you debug a shader.
Before this project, that meant creative work happened in unpredictable bursts. A good week might produce real progress. A bad month might produce nothing, and the re-ramp cost of picking the project back up after a gap could eat the next good week entirely. The emotional toll of that cycle - wanting to build things, having the ideas, but lacking the physical capacity to execute - is its own kind of exhausting.
Working with Claude changed the economics of that equation. The collaboration let Matt operate at the level where his illness doesn't reach: high-level creative direction, describing how something should feel, identifying when something is wrong. The implementation details - the part that requires unbroken concentration and exacts the highest cognitive tax - could be delegated. Not eliminated, but moved to a partner who doesn't experience pain, doesn't lose focus, and doesn't need to re-ramp after a bad week.
Some tasks carry an emotional cost that's invisible from the outside. Slogging through a complex refactor when your body is fighting you isn't just slower - it's demoralizing. Every misplaced semicolon feels heavier. Every failed build stings more. Claude doesn't experience any of that. It approaches a 500-line refactor with the same energy whether it's the first task of the day or the fiftieth. That asymmetry turned out to be genuinely freeing. Matt could bring the creative energy and directorial focus; Claude could handle the implementation grind that illness makes punishing.
The result is that PolyFish exists. Not a simplified version, not a scaled-back proof of concept - a full real-time ecosystem simulation with physics, shaders, spatial audio, a cinematic camera, and VR support. Built in spare time, during a period that included plenty of bad days. That wouldn't have happened without AI as a collaborator. The ideas were always there. The bottleneck was always execution capacity, and this collaboration removed it.
AI-as-accessibility-tool isn't a framing you hear much in the current conversation around these technologies. But for anyone whose creative ambitions outpace what their body allows on a given day, it's a meaningful one.
Refining the Collaboration
What We Learned
Over the course of the project, the collaboration got noticeably better. Some patterns that emerged:
Small, focused sessions beat marathons. A 2-hour session with a clear goal ("get the oxygen system working for dolphins") produced better results than an 8-hour session with a vague one ("work on creature AI"). Context stayed fresher and regressions were easier to catch.
Show, don't spec. Screenshots and live browser tabs were more effective than written specifications. Pointing at a swimming fish and saying "that turn looks too sharp" gave Claude more actionable information than a paragraph describing the desired turning radius in degrees.
Let the AI propose architecture. The results were usually better when Matt described desired behavior and let Claude choose the approach. Claude proposed using a spatial hash for nearest-neighbor queries instead of brute-force distance checks - a pattern Matt hadn't considered that turned out to be critical for performance.
Commit often. Small, frequent commits made it easy to roll back when an AI-generated change introduced regressions. Large, sweeping changes were harder to debug and easier to lose work in.
Project Rules as Guardrails
The project's CLAUDE.md file - a set of instructions that persist across sessions - became increasingly important. It started with a single writing style rule. Over time it grew to include architectural conventions, naming patterns, and performance constraints. These rules gave Claude consistent context even in fresh sessions, reducing ramp-up time significantly.
Going Forward
What We'd Do Differently
If we started over, we'd invest more in automated testing from day one. The project has tests, but they were added retroactively. AI-generated code is especially test-worthy because the author can't always predict edge cases the way a human who struggled through the implementation might.
We'd also be more deliberate about architecture documentation. The codebase has inline comments but no centralized architecture guide. When Claude starts a new session and needs to understand how the rendering pipeline connects to the creature system connects to the physics worker, it re-discovers that from code. A living architecture doc would save real time.
The Bigger Picture
Building PolyFish with AI didn't feel like using a tool. It felt like working with a collaborator who types very fast, never gets tired, and occasionally forgets what you talked about yesterday. A real-time ecosystem simulation with physics, spatial audio, a cinematic camera, and VR support would have been a multi-month project for a small team. What makes this remarkable is that Matt built it entirely in spare time, in under two weeks - this wasn't a full-time effort. As a solo developer with an AI partner working on evenings and weekends, he created something that would typically require a team and months of sustained focus.
That's not because AI replaces skill. Matt's design sensibility and willingness to say "no, that's not right, try again" were essential. The AI amplified what he could build, but it didn't decide what to build. The collaboration worked because both partners contributed something the other couldn't.