Shipping an AI Chat Without Burning Money

Adding AI chat to a website sounds simple.

Create an API route, send the prompt to a model, and stream the response back.

Technically, that part is simple.

The problem starts when you put it on the public internet.

I recently added an AI chat experience to my portfolio. The goal was straightforward: let visitors ask questions about me, my work, projects, experience, and the things I build.

The interesting part was not connecting the model.

It was making sure nobody could turn a small portfolio feature into an unlimited API bill.

The first version was basically an open wallet

A naive implementation looks something like this:

Visitor
-> /api/chat
-> LLM provider
-> streamed response

If /api/chat is publicly accessible, anyone can call it.

Not just someone using the interface.

Bots can hit the endpoint directly. Someone can write a loop. Multiple requests can run at the same time. Prompts can become unnecessarily large.

Every one of those requests costs money.

The UI can look perfectly normal while the backend gets abused. So I started treating the chat endpoint less like a fun portfolio feature and more like a small public API.

Rate limiting was not enough

The obvious first solution is rate limiting. I added limits around both the visitor and their session.

That helps, but a simple "X requests per minute" rule does not fully solve the problem.

Suppose someone starts several expensive requests almost simultaneously. They may all pass the rate check before any of them finish. Now you are paying for several concurrent generations.

So I added a concurrency lock as well.

Conceptually:

request arrives
 
-> validate request
-> check rate limits
-> check spend limits
-> acquire concurrency lock
-> call model
-> stream response
-> release lock

Each protection handles a different failure mode.

I added an actual spending ceiling

Rate limits answer:

How frequently can someone use this?

But what I really cared about was:

How much money can this feature spend?

Those are different questions.

The portfolio now has spend budgets backed by Upstash. That gives the system a hard boundary instead of relying entirely on usage assumptions.

If the allowed budget is exhausted, the AI feature can stop. The rest of the website keeps working.

That is a tradeoff I am completely fine with. A portfolio chat going temporarily unavailable is much better than waking up to an unexpected API bill.

Validate before spending anything

Another rule became obvious:

Reject bad requests before they touch the model.

The API validates things like:

• request origin
• input size
• expected request structure
• session usage
• visitor usage
• concurrency state
• available budget

Only after those checks does the request reach the model gateway.

The cheapest LLM request is the one you never send.

Streaming still matters

Despite all the backend restrictions, I did not want the feature to feel restricted.

Responses still stream back to the browser. That makes the interaction feel immediate instead of waiting for an entire answer to generate before anything appears.

There is an interesting product lesson here: backend systems can be strict while the user experience remains simple.

The visitor should not need to know that behind one text box there are rate limits, locking, budget checks, validation, and failure states. They should just see:

Ask something.

I deliberately kept persistence simple

The chat does not need a complicated database-backed conversation system.

For my use case, browser storage is enough to preserve the visitor's messages locally.

That is an example of something I keep relearning while building products:

Production-ready does not mean architecturally maximal.

It means the architecture matches the actual risk.

The expensive part of this feature lives on the server, so that is where the stronger controls belong. The local conversation history does not need the same infrastructure.

"Call the AI API" is only the prototype

AI APIs make impressive prototypes extremely easy. That can also hide how much engineering comes afterward.

The model call was probably the smallest part of the system. The real feature became:

interface
+ streaming
+ validation
+ rate limiting
+ concurrency control
+ spend budgets
+ failure handling
+ persistence

And that is before even talking about whether the answers are actually useful.

The more AI products I build, the less I think of AI integration as simply:

Send prompt. Receive answer.

The model is just another dependency. Everything around it is what turns that dependency into a product.

You can try the implementation on this portfolio.