Lately, while working on an AI-heavy app, I had to change how I was structuring things. At the start, everything ran inside normal API routes. A user clicks something, the server calls an AI model, waits, then responds. That felt fine when I was just getting features working.
But once the workflows grew, this approach started breaking down.
A single user action was no longer one call. It turned into research, document processing, planning, multiple generations, retries, and progress tracking. When all of that runs in a single request cycle, things get messy fast. Requests time out, retries repeat expensive work, and costs become hard to predict.
Moving Heavy Work Out of the Request Path
So I moved the heavy AI work out of the request path.
Now the UI responds immediately and the actual work runs in the background, step by step. If something fails halfway, it can retry from that point instead of starting over. This alone made the system much more stable.
Cost Control
The other problem I ran into was cost control.
AI bills usually do not blow up because of traffic alone. They blow up because the same work runs twice, users trigger too fast, or retries happen without limits. So I added an in-memory caching layer for rate limiting and usage checks before expensive operations execute.
This was not about premature scaling. It was about not leaking money quietly.
The Real Shift Was Mental
The main shift for me was mental.
Instead of thinking in terms of API calls, I began viewing systems as workflows that need rules around timing, frequency, and failure handling. This change in framing simplified the architecture, improved UI responsiveness, and made cost management possible.
For multi-step AI workflows, this kind of structure becomes necessary much earlier than it seems. I implemented it using Inngest and Upstash Redis.