Beware of unfettered automation

In 2018 PayPal sent a letter to Lindsay Durdle that read as follows:

You are in breach of condition 15.4(c) of your agreement with PayPal Credit as we have received notice that you are deceased. In accordance with condition 15.4(c), we are entitled to close your account, terminate your agreement and demand repayment of the full amount outstanding.

This breach is not capable of remedy.

It would be funny if there wasn't a distressed, bereaved husband at the other end of it. PayPal apologised to her widower but didn't go into details about how such a letter came about. It can only be automation, though, because who would be dumb enough to send such a letter to a deceased person?

In 2013 a new, automated system used by the Unemployment Insurance Agency in Michigan ran amok and started accusing people of benefit fraud. It issued 50,000 fraud notices over the next two years causing untold distress, but, several law suits later, it was revealed that only 8% of those were actually fraudulent.

We probably all know about the problems automation caused with the Boeing 777. A system called MCAS (Maneuvering Characteristics Augmentation System) started actively fighting with the pilots over the pitch of the aeroplane, resulting in a couple of notable disasters and the subsequent grounding of the entire fleet.

I'm not presenting these examples as an argument against automation. Automation is good, particularly if it can remove some of the drudgery of life. But care must be taken with critical systems. Automation can help eliminate human error, but we must remember that humans designed and implemented the automation system, and they can make mistakes too. In other words, don't blindly believe what an automated system tells you.

On a lighter note, I've always liked Warren Gamaliel Bennis' quote about automation:

The factory of the future will have only two employees: a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment.