ChatGPT is now finding bugs in databases (opens in new tab)

I think a lot of overly-eager and under-informed people are jumping on the bandwagon and shouting half-baked proclamations from the rooftops to get clicks and views.

But underneath all the bullshit is something truly useful, so I wouldn't necessarily say that Confidently Incorrect™ is ChatGPT's primary ability.

Coincidentally, over the past few weeks of this ChatGPT craze, I've needed to create a lot of fake data to seed a database. Normally not a big deal, but these records need to have foreign keys pointing to one another. I wondered whether ChatGPT could help me out, so I briefly described the fields I needed, their format and data type, the type of information they contained, etc etc. It did it almost perfectly, and fixing the little errors was trivial. I was dreading having to do this because it's such a pain in the ass.

To me that kind of thing is going to be the most useful application. Everyone's freaking out (in both the "scared" and "excited" sense) over AI's ability to replace creativity, but I'm focused on it's ability to replace toil.

That's almost every reddit thread I've been involved in.

Oratory prowess is way more important than accuracy. If you ever want to "lose a debate" on reddit, pick something that's counter-intuitive or widely misunderstood and use nuance and citations in your defense. You'll get cyberbullied every time - sometimes even banned from the sub.

I’ve been quizzing ChatGPT about it’s knowledge of the oil industry - it has no idea what’s talking about.

For example, I ask it to simulate a discussion between a flow engineer and a geologist, and it doesn’t seem to understand they work together regularly.

Not like there’a a stack overflow for this sort of thing.

In the pre-LLM world we already had rumors/myths/lies/propaganda becoming "facts" after being reinforced through repetition without verification. (Relevant xkcd: Citogenesis https://xkcd.com/978/.)

Now, instead of authoritative sounding humans who are possibly wrong or with an agenda, we have an "infallible", "impartial" oracle capable of inventing whatever it wants to. Yes, I have seen ChatGPT treated as infallible on this very forum: "you are wrong because ChatGPT says <insert nonsense here>". Can't wait to see LLMs taking "post-truth" to a whole new level. The propaganda potential is immense, to point out just one application.

Before the had the internet, back when we still had newspapers, we had a similar phenomenon referred to as Gell-Mann Amnesia.

> Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues.

> In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

https://www.epsilontheory.com/gell-mann-amnesia/

Gell-Mann Amnesia Effect but at scale. I find this all the time when reading the NYT, or Washington Post, or whatever, about something I'm intimately familiar with. ChatGPT just does this as a sort of carnival trick, and people seem to think it's funny for some reason.

https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...

>Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.

>In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

Reddit version of Gell-Man Amnesia: https://www.johndcook.com/blog/2021/01/18/gell-mann-amnesia/

> There's this internet phenomenon (common on Reddit and other such forums) where you find high-quality discussions on a variety of topics and everyone seems so clever and informative, until you come across a thread on a topic that you are actually an expert on and find that every top-voted comment is objectively incorrect.

Sounds like the old days to me, where we would meet up , discuss the world and what have you, without any means to fact check anything online.

Changes there was a real expert on the subject in your circle was at best 20% So we just enjoyed / continue our life. No harm done as long as nobody took the discussion as gospel.

That was also common on TV. Everyone seems credible, until we spot errors when someone in our field starts talking.

edit: minor typo

When I've asked ChatGPT about things I know a little about I find its response is often wrong. So much so that I always respond with "Are you sure that's correct?" it then often responds "I'm sorry you are right, I should have said... " and then says something a little closer to the truth.

I wouldn't rely on ChatGPT or any system that relies on it, except perhaps for summarising/rephrasing a single, known document. It worries me that people are using it for software development - creating stubs, bootstrap code. It's perhaps OK if you know the language/platform well enough to fix issues, but if you then use it on something that your not an expert in, what is the outcome.

For example:

Finding the source of literary quotes - miss attributing them - took 5 or 6 QA responses for it to get close.

Explaining how a p-channel enhancement mode MOSFET works - explained n-channel

Explain regex - often gets the technical parts right but can make very bad suggestions on what the regex aim is.

Name some classic cocktails created in Europe - suggested Manhattan, which it said was acceptable because it was popular in Europe

Suggest some gin based cocktails that don't include citrus, suggested those with lemon juice.

This generalises very broadly.

I have some expertise and knowledge is one or two specialised fields, and when I read Wikipedia, news articles and blogs about those fields, it's maddening how incorrect many people are, whilst sounding perfectly knowledgeable.

Therefor you have to verify the answer or accept that it may be wrong.

I’m finding that it’s fairly good for code snippets. You can test that pretty quickly. But asking it something you don’t know and can’t verify is a huge risk.

> common on Reddit and other such forums

It's especially common on Hacker News.

"mansplaining as a service" is the term I've seen used. Or Artificial Cliff Calvin.

I compared ChatGPT to a middle tech manager that just heard he has to create a last minute presentation. He knows all the current buzz words to sounds smart.

My first recollection of this sentiment pertained to the New York Times, for what it’s worth.

Same effect happens with newspapers. Check your sources.

Sounds like internet version of Gell-Mann effect

it is not limited to the internet. this is also applicable to any news company

I don't think it's an internet phenomenon. It's a human phenomenon. It doesn't matter if it's a conversation or a piece in a newspaper (c.f. the Gell-Mann amnesia effect).

we’ve made the startling discovery that we’re rather predictable

I asked it to generate me documentation for a handful Bicep definitions and include links to the official Microsoft Docs. It generated about 50% erroneous links, but once I updated my prompt and fed it the "root" of the MS docs I wanted it to use, it was able to correct all the urls.

At the time it _seemed_ like it wasn't just doing a mindless string replace, because different parts of the url changed on most of the erroneous links. I figured it wasn't going out and spidering the link I gave, but it led me to believe maybe there was _some_ form of index that it simply hadn't prioritized prior, but was able to do with the additional context.

copy.ai's chat bot can read URLs now

The concern is the increasing number of anecdotes where people are taking ChatGPT's output at face value, even though everyone knows by now that it does sometimes get stuff wrong and does hallucinate. I thought it would take longer.

> OpenAI handed this tool to people with effectively no guidance or instructions so that they can trade on the glow of misconceptions and harvest user data.

When one logs on to Chat GPT, they get the following notices:

> This is a free research preview.

> Our goal is to get external feedback in order to improve our systems and make them safer.

> While we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice.

> Conversations may be reviewed by our AI trainers to improve our systems.

> Please don't share any sensitive information in your conversations.

Effectively no guidance or instructions indeed. Perhaps ChatGPT is sometimes confidently incorrect because it learned from online commenters.

This is baseless speculation

In either scenario, it's hallucinating. Either JSON queries were supported in that snapshot of the documentation, or they weren't.

That's very unlikely. ChatGPT doesn't contain a full copy of everything it saw when it was trained - it has model weights that were updated by that training process.

It's very unlikely the model weights would contain enough detail to let it match content to the URL that the content originally came from.

You’re saying ChatGPT has access to a an older snapshot of the internet with URLs and everything?

Assuming the earlier statements from ChatGPT were correct, the feature did not exist before the training data cutoff, so that's impossible.

I don't think they prefix each document with the URL in the training set, but there might be URLs in the clear text.

It also can be prompted to read the contents of a webpage, I have used it to help me craft a regex expression for removing junk utilizing invoke-webrequest to parse out CPU models compatible with 22H2 upgrades from Microsoft(that page did not exist in 2021).

ChatGPT is also highly familiar with what keywords to search for.

This actually shows perfectly how convincing ChatGTP can be. The author is too eager to believe the output from ChatGTP, even though this should be a clear indicator something else is happening:

> But ChatGPT has rewritten my article in a different style, what a interesting finding…

The URL conveniently contains the topic of the blog-article, meaning ChatGTP can 'hallucinate' the contents, even without internet connection.

To show this, I tried to follow the same steps as @neonforge, but with a non-existing URL. Unfortunately `lynx` 'didn't work', but curl showed some interesting information.

`curl https://medium.com/@peter/learn-how-to-use-the-magic-of-terr...` returned some html. Instead of a 404 and a <title> element containing just 'Medium' it hallucinated the following title tag: `<title>Learn How to Use the Magic of Terraform Locals (Step-by-Step Guide) | by Peter | Medium</title>`

I'm not saying that ChatGTP definitely has no access to internet, but so far I have not seen any proof or indication that it does.

I don't think this shows at all that ChatGPT can access the internet? It looks like ChatGPT was given a URL with a title and hallucinated a blog post based on that title. The author then brushes it off as 'written in a different style', when it just looks like a totally different article overall (or maybe it's testament to the fact that too many Medium articles are low quality and indistinguishable from the output of a LLM!)

If anything, this blog post is a perfect example about how people can put in whatever they want as an input and take the output as truth, without any rigorous approach about what would count as a true fact about the model.

That article is bunkum. There’s not the slightest reason to believe that it “rewrote his article in a different style”: it hallucinated a completely different article based on the URL. The other “proof” is all clearly wildly inconsistent and hallucinated: it’s a desktop machine, a server, inside Google in I think France, in the USA, in AWS, in Germany, and I think the last couple of “tests” don’t make sense anyway.

It convinced him of what he wanted to believe. He didn’t do a very good job of testing.

That was all hallucination.

If you want to prove this to yourself, tail a server log and paste a URL into ChatGPT and see if you get a hit.

Because it has essentially no logic system whatsoever. It doesn't have any way to "know" something is correct because when it produces a word for word copy of a factual statement, it is internally identical to a completely made up (hallucinated) sentence... because both are hallucinated. The deriding "it's fancy autocomplete" remarks are, in a lot of ways, technically correct.

You can still do useful things with it, that much is extremely clear. But it doesn't think.

Told ya

Slightly tangential. After interviewing a candidate, within 20 minutes they sent me a long thank you note that had several generic things that definitely did not apply to what we discussed. Very obviously ChatGPT-generated. Using ChatGPT in and of itself may not be all that bad, but this shows that the candidate did not even take the time to tailor it to our conversation. During the interview I was on the fence about pursuing the candidate but this justifies a clear reject.

I had it write my green card justification. Awesome corp-speak BS.

Creating front ends that integrate different AI APIs is kind of over saturated, that's probably why you're not getting much attention. It's also somewhat hard to understand what's the value proposition of the website, you should focus on putting as much of that kind of information in everyone's face in the landing page, otherwise they'll just bounce. Take everything from your "Learn more" page and put it in your landing page. Simplify, condense and systematize the information in a way that makes it digestible and instantly obvious.

Also, I should note your comment is kind of off-topic, have you thought about creating a Ask HN talking about your issue? It's very possible other HNers could help you get your website and product more interesting. I would also recommend not being bitter about it, Hacker News is all about Silicon Valley and startups, the main rule for these is that 9 times out of 10 you'll fail. It's normal.

At last I should note, if you're doing things as a hobby, do them for yourself, remember to have fun and create NO expectations of what you will or will not be able to do. If you're trying to get a startup or product going, there's a reason why Y Combinator is so popular, if you truly believe in what you're building you should try getting it into an incubator.

Just my two cents.

If you listen to the people who have done this before (e.g. read/listen to indie hackers), Reddit/HN is very hit/miss.

Only thing seems to consistently work is Twitter - and then you should REALLY be building public if you have no audience so that you can build one. Have you tried this? If not, and you've been building in stealth mode, you've missed out on an opportunity but you can still start now.

You are figuring out so much stuff in an area without expert - make a Twitter post every day about what new problem you solved and you will quickly find a following as you establish authority in this new field. So many people are learning from each other about this stuff, it doesn't take much to become an expert.

I've been going through all the Indie Hackers podcasts. Consider listening to Indie hackers #008 or #009, I think they specifically talk about bootstrapping an audience from nothing.

You may even find people are more interested in an educational product from you than your service, and pivot, for instance.

It's not helping that you are constantly hijacking threads which are really only tangentially related to promote your app.

As far as the website premise itself, I feel like this might've been useful 20 years ago when it was more popular to write static vanilla HTML with jquery sprinkled in. These days, if a person wants a relatively no code solution they're gonna go for WordPress or square space, otherwise they're probably going to design something using a framework such as react svelt or vue.

I just watched the demo and personally thought it was amazing. Don't give up! Keep going!

People are always made unnecessary. The smart ones then find other ways in which they are. Not sure how this is any different.

Looking forward to that day.

One of the words it gave me was “Yclept (archaic form of "called" or "named")” which is both silent and invisible!

Humans who don’t understand what they are talking about we should not listen to nor trust. By the same logic it would be stupid to listen to / trust ChatGPT.

CelerData’s StarRocks engine.

Isn't that solved with supervised learning?

The way I see it is that internet content is used to bootstrap the models, then supervision is used to train the models without the risk of a feedback loop causing quality loss.

I'm pretty new to ML so I may be missing something.