AI Has a Churn Problem

The treadmill

Every week there's a new model. Every month there's a new framework. Every quarter the "best practice" from three months ago is obsolete. If you're building with AI, you've felt this — the constant pressure to rip out what you just built and replace it with the new hotness.

This is churn. And it's not a minor annoyance — it's becoming one of the biggest hidden costs in AI-driven software development. Teams are spending more time migrating between tools than actually shipping products. The promise of AI was 10x productivity. The reality, for many, is 10x context switching.

Where the churn comes from

The churn isn't random. It has structural causes that make it almost inevitable in the current ecosystem.

Model releases outpace integration. By the time you've tuned your prompts, built your evaluation suite, and optimized your pipeline for one model, a better model drops. The new model has different strengths, different failure modes, different context windows, different pricing. Your carefully crafted system prompts that worked perfectly on GPT-4 produce garbage on Claude, and vice versa. So you rebuild. Again.

Frameworks are built on shifting sand. LangChain, LlamaIndex, CrewAI, AutoGen — each one promised to be the abstraction layer that would insulate you from the chaos below. Instead, they added another layer of churn on top. Breaking changes in minor versions. APIs that get deprecated before they're documented. The framework you chose six months ago now has a "legacy" mode.

Best practices haven't converged. In traditional software engineering, we've had decades to figure out patterns that work. REST APIs, relational databases, MVC architecture — these are stable because they've been battle-tested. AI engineering is still in its "try everything" phase. RAG vs fine-tuning vs long context. Agents vs chains vs workflows. Every team is rediscovering the same lessons from scratch because there's no shared body of proven practice.

The real cost

The obvious cost is engineering time. But the deeper cost is institutional knowledge that never accumulates.

When you rebuild your AI pipeline every few months, you lose the hard-won understanding of why certain decisions were made. The prompt that was carefully tuned to handle edge cases gets thrown away with the old framework. The evaluation dataset that took weeks to curate doesn't transfer cleanly to the new architecture. The team's mental model of "how our AI system works" resets to zero.

This is particularly brutal for smaller teams. A startup with five engineers can't afford to have two of them perpetually migrating infrastructure. But that's exactly what happens when every quarter brings a new "you should really be using X instead" moment.

The irony is thick: AI is supposed to make us more productive, but the AI ecosystem's instability is making us less productive at building with AI.

What to do about it

You can't stop the churn. The field is genuinely moving fast and that's mostly a good thing. But you can build in a way that absorbs churn instead of amplifying it.

Build thin abstraction layers you own. Don't marry a framework. Write a thin adapter between your application logic and whatever AI provider or tool you're using. When the next model drops, you swap the adapter, not the application. This isn't over-engineering — it's survival. The adapter should be simple enough that replacing it is a day's work, not a sprint.

Invest in evaluation over implementation. Your eval suite is more durable than your implementation. If you have a solid set of test cases that define "correct behavior" for your AI features, you can swap out the entire backend and know immediately whether the new thing works. Evals are the one asset that appreciates across model generations.

Delay adoption deliberately. Not every new release needs to be adopted immediately. Let the early adopters find the bugs. Read the migration guides. Wait for the first patch release. The cost of being two weeks behind the cutting edge is almost always lower than the cost of being the person who discovers a breaking change in production.

Document decisions, not just code. When you make an AI architecture decision, write down why. "We chose streaming over batch because latency matters more than throughput for our use case." When the next shiny thing arrives, you can evaluate it against your actual constraints instead of getting swept up in hype.

Our take

At System32, we've felt every bit of this churn. We've rewritten pipelines, migrated between models, and watched frameworks we depended on pivot in directions that didn't serve us.

Our response has been to bet on primitives over frameworks. We build on the lowest-level abstractions that are likely to remain stable: HTTP, containers, standard I/O, structured data formats. Everything above that is our code, our abstractions, our control. When a new model drops, we test it against our eval suite and swap it in if it's better. When a framework breaks, we don't notice because we never depended on it.

The churn problem is real, and it's not going away soon. But the teams that will win are the ones that build churn-resilient systems — not the ones chasing every new release. Stability is a competitive advantage in an unstable ecosystem.