An automated scan says the app is fine. It never looks where the bug lives.

A common story in AI-built apps goes like this: someone runs one of those automated security scans on a Supabase app — paste the URL, wait a minute, get a score. It comes back looking healthy, and everyone feels good about it for a week.

Then you open the RLS policies and one of them says USING (TRUE). In plain terms, that policy lets any logged-in user read every other user's rows. The scan never caught it. And once you see how those scans work, it makes sense that it couldn't.

A URL scanner is standing outside your house

Here's the part that's easy to miss. A scanner that only has your URL can check whether the doors look locked, or whether you left a key under the mat: exposed secrets, missing headers, open endpoints. Those checks are real and useful. But your row-level security lives inside the database. Whether user A can read user B's data is a decision Postgres makes after a request comes in. From the outside, a correct response and a leaky one look identical. The scanner sees a 200 either way.

So the bugs that matter most in AI-built apps are the ones an outside-in scan can't reach:

A broken RLS policy. USING (TRUE) passes every check because technically RLS is "enabled." The AI turned it on and wrote a policy that does nothing.
An IDOR in an API route. Change one id in the request and you get someone else's record, because the handler never checked who was asking. From outside you'd need to already know a valid id to even notice.

Neither shows up in a response header. You have to read the actual policy or the actual route.

The boring check that works

For RLS, open each policy and ask one question: does the USING clause reference auth.uid()? If it just says TRUE, or compares to a column anyone can set, it isn't protecting anything. "Enabled" is not the same as "scoped to the right user."

For routes, look at every handler that takes an id from the request and ask where it confirms that id belongs to the person making the call. If that line isn't there, that's the bug.

Scans aren't useless — scores just aren't safety

Automated scans aren't useless. They're worth running. They catch the outside stuff fast, and that stuff matters. The trap is reading a green score as "I'm safe," because the score was only ever looking at the outside wall.

AI is genuinely good at writing code that runs and passes tests. It's just not thinking like someone trying to break in, so the security part is still on us to check.