this post was submitted on 22 Aug 2026
467 points (97.2% liked)
Technology
87450 readers
3616 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Basically the same conclusion I came to. Which is terrifying when you think of all the devs who depend and all the money thats riding on it.
And syntax is the kind of thing llm's should be awesome at. If they can't even get that right we're all cooked.
In fairness, this was all on a switch, where the commands are very tightly tied to the vendor and their underlying in-house shell. So commands may vary by release greatly. I was already explicitly telling the bot what software version and licenses I had, but ultimately I had to resort to the CLI's help function at times and tell the bot "The command you gave me didn't work. Here's where it broke, and here's the commands it will accept". Given that information, it could (generally) figure it all out.
I dunno if you had this experience but the one I used made up commands if it didn't know them.
It basically hallucinated them.
I've had them do this for various things, nowhere near the 20% of the time they've called "the hallucination rate" but definitely once in a while if it doesn't know it will guess.
Which, to be fair, is what I've been doing for 40+ years in computer programming. When you have the ability to check if you've guessed right or wrong in a couple of seconds or even minutes, but researching it to "be sure" (and sometimes still get it wrong) before trying might take hours it's only natural / efficient to guess a few times before giving up and RTFM.