An AI Vibe Coding Horror Story (opens in new tab)

At one point I worked as a customer support agent outsourced to Apple via the company. Apple forced us to us some very outdated browser UIs, basically for filling in forms, across maybe 4-5 different services in some cases. The machines we were given by this outsourcing company of course where Apple computers, fairly locked down.

But one thing they hadn't locked doll wn, was installing extensions in Safari, and given I had some development chops from coding a bunch in my freetime, I saw the opportunity to write a tiny extension that saved me a ton of time by merely copy-pasting stuff into the right forms and so on. Basically making the whole thing more efficient for me.

Everything was great, until the person next to me saw I had something different. Cautiously eager, I let them try the extension too, they loved it, and without thinking about it, spread it to other people in our team. Eventually, the manager and the IT team picked up what was going on, said they'investigate if I could maybe start doing those kind of things full-time instead of being a support agent, and just focus on tooling.

Fast forward two weeks, I get called into a meeting, apparently someone in the company had been "stealing" CC numbers from the customers on the calls, and since they don't think they've found the right person who did it (or something like that), the person who was known for "doing stuff to the computers" was the next possible suspect, and they fired me right there.

Eventually this firing let me find my first actual programming job, so I'm not too mad about it, but it really shows how out of touch lots of companies and people are when it comes to how computers actually work.

AEPD are well known, even in the rest of the world. They have a different strategy compared to other countries. Ireland's DPC are also heavy handed, but focus on large companies mostly.

France's CNIL is also not bad. They are particularly hard against things like "you accidentally sign up for x y z services when only wanting to sign up to service A".

Gdpr in the EU is also miles ahead of what the US has, or at least what it has been enforcing for a long time.

Thta's wonderful! Most of europes GDPR/Data protection autohrities are completely worthless and seem to constantly side with big corps.

Only when they start to side with the people, actually fining business billions and billions will things start to change. I hope we'll see this happen in europe at large, and not only in a few countries.

Ah yes, companies don't have hands. Nothing to burn!

That worked for years & traveled for tens of thousands of kilometers until the disaster. They were also quite aware of the risks associated with a hydrogen airship and did all sorts of mitigations to avoid them.

Compared to that vibe coding has no such qualities.

Nice. Another one I read on HN is “we are building hell on earth.”

The next generation of the user is what I think will be the factor for AI-assisted loss of life, moreso than the models themselves. It's the trust we put in them.

The dinosaurs of the before-LLM age will increasingly struggle to convince young people e.g. to seek competent medical consultation, because ChatGPT can do that much better (it's always supportive, available at any hour, meets you where you are...) -- this aspect alone is bound to have a death toll associated with it.

Or imagine asking ChatGPT to map an Alpine climbing route, -- anything with a bad failure mode.

But even spec-first, using opus4.6 with plan, the output is merely good, and not great. It isn't bad though, and the fixes are often minors, but you _have_ to read the output to keep the quality decent. Notably, I found that LLM dislike removing code that doesn't serve active purpose. Completely dead code, that they remove, but if the dead code have tests that still call it, it stays.

And small quality stuff. Just yesterday it used a static method where a class method was optimal. A lot of very small stuff I used to call my juniors on during reviews.

On another hand, it used an elegant trick to make the code more readable, but failed to use the same trick elsewhere for no reason. I'm not saying it's bad: I probably wouldn't have thought about it by myself, and kept the worse solution. But even when Claude is smarter than I am, I still have to overview it.

(All the discourse around AI did wonder for my imposter syndrome though)

Doesn't require Jira but yes, specification-first is the way to get better (albeit still not reliably good) results out of AI tools. Some people may call this "design-first" or "architecture-first". The point is really to think through what is being built before asking AI to write the implementation (i.e. code), and to review the code to make sure it matches the intended design.

Most people run into problems (with or without AI) when they write code without knowing what they're trying to create. Sometimes that's useful and fun and even necessary, to explore a problem space or toy with ideas. But eventually you have to settle on a design and implement it - or just end up with an unmaintainable mess of code (whether it's pure-human or AI-assisted mess doesn't matter lol).

That's a one-shot, though. Now try making something you want to build on. It falls apart very, very quickly.

Not to mention that these prototypes and samples all turn out more or less the same. Obviously, given how LLMs work. Like LLM prose, LLM-generated web apps have a distinctive, samey look and feel.

This is what I’m noticing. At my workplace, we have 3 or 4 non-devs “writing” code. One was trying to integrate their application with the UPS API.

They got the application right, and began stumbling with the integration - created a developer account, got the API key, but in place of the applications URL, the had input “localhost:5345” and couldn’t get that to work, so they gave up. They never asked the tech team what was wrong, never figured out that they needed to host the application. Some of the fundamental computer literacy is the missing piece here.

I think (maybe hopeful) people will either level up to the point where they understand that stuff, or they will just give up. Also possible that the tools get good enough to explain that stuff, so they don’t have to. But tech is wide and deep and not having an understanding of the basic systems is… IMO making it a non-starter for certain things.

The competence profile of any LLM-based AI is extremely spiky - whether it does a particular task well or not is pretty independent of the (subjective) difficulty of the task. This is very different from our experience with humans.

Maybe this is what's missing in the prompt? We've learned years ago to tell the AI they're the expert principal 100x software developer ninja, but maybe we should also honestly disclose our own level of expertise in the task.

A simple "I'm a professional surgeon, but sadly know nothing about making software" would definitely make the conversation play out differently. How? Needs to be seen. But in an idealized scenario (which could easily become real if models are trained for it), the model would coach the (self-stated) non-expert users on the topics it would ordinarily assume the (implicitly self-stated) expert already knows.

“You’re just holding it wrong”

There’s a reason why many professions have professional bodies and consolidated standards - from medicine to accountancy, actuarial work, civil engineering, aerospace, electronic and electrical engineering, law, surveying, and so many more.

In most of those professions, it is a crime or a civil violation to offer services without the proper qualifications, experience and accreditation from one of the appropriate professional bodies.

We DO NOT have this in software engineering. At all. Anyone can teach themselves a bit of coding and start using it in their professional life.

Analogous to law, you can draft a contract by yourself, but if it goes wrong you have a major headache. You cannot, however, offer services as a solicitor without proper qualifications and accreditation (at least in the UK). Yet in software engineering, not only can we teach ourselves and then write small bits of software for ourselves, we can then offer professional services with no further barriers or steps.

The mishmash of laws we have around data and privacy are not professional standards, nor are they accreditation. We don’t have the framework or laws around this. And I am not aware of the USA (federal level) or Europe (or member states) or China or Russia or India or etc having this.

For example, the BCS in the UK is so weak that although it exists, exceedingly few professional software engineers are even registered with them. They have no teeth. There’s no laws covering any of this stuff. Just good-ol’ GDPR and some sector-specific laws here and there trying to keep people mildly safe.

In some countries you can't call yourself a software engineer if you don't have an engineer degree and a license to practice.

Should be the same everywhere. Anyone can be a coder, but not everyone is an engineer

One of the key functions of a professional body is to ensure all members are aware of existing and new laws, standards and codes of practice. And to ensure different grades of engineer are aware of different levels of the standards. And that sector-specific laws and standards are accredited accordingly.

High profile convictions are not a good way of dealing with this. Not in the short or long term. Sure they have an impact, and laws should be enforced, but that’s not a substitute for managing the industry properly.

Sadly, probably true.

heh, I know that username. I came to the same conclusion. (I hope all is well with you, Yannick)

Okay. If it's real I apologize.

But in any case it's so lacking in detail and so brief as to make it so uninteresting that it might as well be fake.

> Somebody "vibecodes" medical app/system. The app was insecure. Personal info leaked.

Okay cool.

It’s unlikely any LLM tasked with a prompt involving medical records did not automatically address separation of concerns. The type of data involved is worst case scenario. One JS file is also worst case scenario. This is why it may feel manufactured. If it is true, they truly deserve to be put on blast.

The first time I stumbled onto a big security vulnerability (exposed stripe/aws/play store keys. I was poking around an API a web app was using, and instead of hitting /api/v1, if you just hit /api it served them. I wasn't trying to do anything malicious), the very first thing I did was contacted a security researcher friend to ask about covering my ass while performing responsible disclosure.

You hear too much about people being persecuted for trying to point out security vulnerabilities. (Guess they haven't heard about "don't shoot the messenger").

(It turned out fine after finally managing to speak with someone. Had to ring up customer service and say "look, here are the last digits of your stripe private key. Please speak with an engineer". Figuring out how to talk with someone was the difficult thing)

Company should just take down the whole thing. One vuln might be fixed but how many others might be there.

I don't know what to make of the article. First I thought it seems like a made up LinkedIn story, it seems too crazy while talking about it in such a casual manner. Ultimately I don't know, maybe it was vague for a specific reason. I guess one thing I'd find odd is that whoever developed it, that they didn't run and get stuck with CORS issues, if everything was done client side to those services and that they managed to get API keys, subscription stuff everywhere while still making mistakes like this. And no mention of leaked api keys and creds which UI side there must have been, right?

> Everything that could go wrong, did go wrong.

Then this claim seems a bit too much, since what could have gone more wrong is malicious actors discovering it, right? Did they?

Maybe I have trouble believing that a medical professional could be that careless and naive in such a way, but anything could happen.

I guess another thought is... If they built it why would they share the URL to the author? Was author like "Ooh cool, let me check that out", and they just gave the url without auth? Because if it worked as it was supposed to it should have just shown a login screen right? That's the weirdest part to me, I suppose.

> The timelines mentioned are weird - he spoke to them before they built it? Or after? It's not that clear, he mentions they mentioned watching a video.

I took that all to mean she had explained the history of it to the author, but it had already been written and deployed. It is worded a little weird. It's also translated from german, I don't know if that is a factor or not.

The single file thing makes perfect sense if it was built as an artifact in one of the big provider's webui.

I could bet 5 dollars that they used chat and not an agent, and that's also the reason why it's a single html file.

Copypasted and than dropped into hosting folder, sweet web 1.0 style

> The timelines mentioned are weird - he spoke to them before they built it? Or after? It's not that clear, he mentions they mentioned watching a video.

Yeah although I didn't comment I found this weird as well. Chronology was vague and ill-defined. He went to a doctors office and the receptionist mentioned vibe coding their patient records system unprompted?

> A few days later, I started poking around the application.

What!? How... was there even a web-facing component to this system? Did the medical practice grant you access for some reason?

Yeah I'm back to calling bullshit. What a load of crap. Whole post probably written by an LLM.

I don't think you read the article very carefully, the timeline is that he met a person, and that person told him that they made vibe-coded an app after having seen a video. He then investigated the app.

Having experience working with medical software, I call BS on this article as presented, unless it was some minimal support app. When you deal with patient records, there's so much of local law, communication, billing rules and other things baked in that you CANNOT vibe code an app to handle even 1% of that. Your staff would rebel and your records would completely fall apart. Even basic things like appointment bookings have a HISTORY and it's a full blown room scheduling system that multiple people with different roles have to deal with (reception and providers). It takes serious time to even reverse engineer the database of existing apps, and you first have to know how to access the database itself. Then you'll see many magic IDs and will have to reverse engineer what they mean. (yes, LLMs are good at reverse engineering too, but you need some reference data and you can't easily automate that)

I have decompiled database updaters to get the root password for the local SQL Server instance with extremely restricted access rules. (can't tell you which one...) I have also written many applications auto-clicking through medical apps, because there's no other way to achieve some batch changes in reasonable time. I have a lot of collateral knowledge in this area.

Now for the "unless it was some minimal support app" - you'll see lots of them and they existed before LLMs as well. They're definitely not protecting patient data as much as other systems. If the story is true in any way, it's probably this kind of helper that solves one specific usecase that other systems cannot. For example I'm working on an app which handles some large vaccination events and runs on a side of the main clinic management application. But accidentally putting that online, accessible to everyone, and having actual patient data imported would be hard-to-impossible to achieve for a non-dev.

For the recording and transcription, there are many companies doing that at the moment and it would be so much easier to go with any of them. They're really good quality these days.

Because CRMs are very expensive, and they get much more expensive if you need custom development (which you usually need)

Maybe expensive and slow is actually an improvement.

Oh man, I have gone off on rants about software "engineering" here in the past.

My first office job was as an AutoCAD/network admin at a large Civil and Structural engineering firm. I saw how seriously real engineering is taken.

When I brought up your argument to my FAANG employed sibling, he said "well, what would it take to be a real software engineer in your mind!??"

My response was, and always will be: "When there is a path to a software Professional Engineer stamp, with the engineer's name on it, which carries legal liability for gross negligence, then I will call them Software Engineers."

Totally agree - not just medical software either. See replies to my other comment threads. Software engineers really don’t like the idea that they might have to show they can perform at a certain standard to be able to work as a software engineer.

Typically arguments come up:

“that’s gatekeeping” - yes, for good reason!

“Laws already exist” - yeah, and that’s not the same as professional accreditation, standards and codes of practice! Different thing, different purpose. Also the laws are a mishmash and not fit for purpose in most sectors.

Yeah, fair. I am just thinking out loud here. What is a decent solution to this problem? Is there one?

Fully agentic development is neat for scripts and utilities that you wouldn‘t have the time to do otherwise, where you can treat it as intput/output and check both.

In these cases you don’t necessarily care too much about the code itself, as long as it looks reasonable at a glance.

> Vibe coding is just a bad idea, unless you’re willing and able to vet the output, which most people doing it are not.

It says quite a lot about where we are with ai tooling that none of the big players have “no need to review, certified for market X” offerings yet.

It is related to CI and dev practices etc. A experienced developer using AI would add security/data protection, even when vibe coding.

I largely agree, but if a company sold cement explicitly claiming that they will replace every job in the entire construction industry, that the cement is able to plan, verify, and build on its own, without supervision, and that any layperson can now create PhD level bridges with that cement without any input from or verification by professionals, some liability would definitely fall on the company selling that cement under these pretenses.

Switzerland is very liberal in terms of business-oriented regulations to the point that you could crate a new year party in a closed cellar without emergency exists, not to mention anti-fire installation and burn people alive there.

HN should just have a rule that all content should be human-generated. This post is literally an LLM writing about something an LLM did; it's a bot botting about a bot. Aside from how funny and dystopian that is - just ban it? Just make a rule that submissions require a human author.

I don't think solving this is all that complicated, at least for now. It isn't like it's currently difficult to tell what is and isn't LLM word salad, though that will likely change in the future, but by then the argument will involve whether it really matters or not. But for now, when 80% of the submissions are LLM garbage and it really is garbage, it's pretty jarring.

That's what is meant by human negligence. There will always be a hype about something and that is not an excuse to have a devil may care attitude on any work being done

I assume you haven't seen those advertisements where they put duct tapes on everything and present it as a universal solution, also there will always be a hype about something in this world and that is not an excuse to jump on the bandwagon unless you're braindead

> Usually they would just use an off the shelf product and extend it

AI does the same thing an agency or dev would do. Those vibe coding platforms have a template for these things which is usually Vite + React with Supabase for the backend, the same as a dev might use because surprise the LLM trained on the dev's work.

OP's point is that you're not guaranteed a good outcome hiring an agency or solo dev either, in fact I would say you're almost guaranteed a bad outcome either way.

You've got to wonder from where did the "AI" parroted that.

Sure, but you'd hope that the LLM could understand that too.