Intellectual debt accrued through machine learning features risks beyond the ones created through old-style trial and error. Because most machine-learning models cannot offer reasons for their ongoing judgments, there is no way to tell when they’ve misfired if one doesn’t already have an independent judgment about the answers they provide. Misfires can be rare in a well-trained system. But they can also be triggered intentionally by someone who knows just what kind of data to feed into that system.

https://www.newyorker.com/tech/annals-of-technology/the-hidden-costs-of-automated-thinking