What if someone is obviously out of their mind and says yes? Random example, say someone has a stroke and has just barely figured out how to use the big red emergency call button on their phone screen. The AI asks if they are calling about blah blah emergency. Maybe the person only understands half of it, thinks they’re being asked if they’re calling about an emergency and say “yes”. Now they’re stuck having a form letter read to them by AI while their brain is dying. A person would catch something was up even if they said yes.
The article even notes this can be a problem without anything medical going on:
It's also possible that AI agents handling 911 triage could fail to understand individuals with stronger accents and dialects, as well as differences in pitch and articulation. This is because AI is programmed with automatic speech recognition systems and, therefore, could potentialy be unreliable when presented with a speech unlike the ones used the AI is trained to recognize.
Responding to people having emergencies is too important a task to leave up to voice recognition slapped on top of an LLM.
“Well I just trust they thought of it” isn’t good enough.