Start a project

Why apps built with AI need testing more than any other

4 min read

AI tools can build a working app in a weekend. Here is what they leave out, why the builder is the last person to notice, and what to test before real users arrive.

A laptop on a dark desk showing source code produced with an AI coding tool

You describe an app in a chat window, and an hour later it is on your screen. You click around, it works, and it looks better than you expected. So you share the link.

That is how a lot of software gets made now, and it is a good thing. People who could never afford a development team are shipping real products. But there is a gap in the process that used to be filled by other people, and with an AI tool nobody fills it unless you do.

The app works because you used it the way you asked for it

When you test your own app, you follow the path you described to the AI. You sign up with a normal email. You type a sensible amount. You press the button once.

The AI built that path, so that path works. This tells you very little.

Your users will not follow it. They will paste an email with a space at the end. They will press Back halfway through payment. They will open the app on a five-year-old Android phone, lose signal, and tap the button four times. Nobody asked the AI what should happen then, so the answer is whatever the code happens to do.

What AI tools tend to leave out

AI coding tools are very good at the part you can see. The weak spots are the parts you only find when something goes wrong.

Checks that live on the server

A common pattern: the app hides the Delete button from users who should not delete things. It looks secure. But the action behind the button is still open, and anyone who knows how to send the request directly can use it.

The same thing happens with prices and discounts worked out in the browser, where a user can change them, and with pages that show your data because your ID is in the address. Change the ID, see someone else's data.

What happens when things fail

Ask an AI for a checkout and you get a checkout. You rarely get the case where the payment succeeds but the confirmation never arrives, or the card is declined on the second attempt, or two people buy the last item in the same second.

These are not rare events. On a live product they happen every day.

Secrets in the wrong place

Keys for payment providers, email services and databases sometimes end up inside the code that is sent to the browser, where anyone can read them. The app runs perfectly. The key is also public.

The second and third feature

The first version usually holds together. Trouble starts when you ask for changes. The AI rewrites one part to add a feature and quietly breaks another part it is not looking at. You test the new feature, it works, and you do not go back to check the old one.

A development team catches this with a set of repeatable tests. A solo builder with an AI tool usually has none.

Why the builder is the last to find out

There are three reasons, and none of them are about skill.

  • You know how it is supposed to work. You cannot unsee the correct path, so you keep taking it.
  • You cannot read the code to check it. If you did not write it, a hidden mistake looks exactly like working code.
  • The AI sounds sure. It will tell you a feature is complete and secure in the same tone whether it is or not.

So the first real test of the app is its launch, and the testers are your customers.

What "mandatory" looks like in practice

You do not need a large process. You need someone other than the builder, and other than the AI, to try to break the app before the public does. At minimum:

  1. Every form, used badly. Empty, too long, wrong format, pasted, submitted twice.
  2. Every action, as the wrong person. Signed out, signed in as another user, signed in as a normal user on an admin page.
  3. Money, start to finish. Declined cards, refunds, the Back button, a closed tab mid-payment, and whether the amount charged matches the amount shown.
  4. Real devices. At least one iPhone, one Android phone and one desktop browser you did not build on.
  5. Everything again after each change. The old features, not only the new one.

Our QA checklist for apps built with AI tools goes through these one by one.

The cost of skipping it

A bug found before launch costs an afternoon. The same bug found by a customer costs a refund, a support thread, and often the customer. If it involves someone else's data, it can cost a great deal more than that.

AI has made building cheap. It has not made mistakes cheap, and because more apps are being launched by people with no reviewer behind them, there are now more ways for a mistake to reach users.

Where we come in

We test products built with AI tools the same way as any other product: by hand first, as a stranger would use them, including the security checks that AI tools most often miss. With pay-as-you-go pricing you pay for the bugs we find.

If you have built something with an AI tool and are about to share it, tell us what it does and we will say what we would check first.

Chat with us