Posted by Kirsten Gibbs Last updated 23rd April 2021 reading time
The danger of software systems is that because we talk about them as being ‘engineered’, we take them to be infallible, in a way that would be reasonable if we were talking of a bridge, or a train, or a road.
Bridges, trains and roads obey the laws of physics.
There are no such laws behind software systems, only human beings, with prejudices, pressures and sometimes perverse incentives.
We would do well to remember that, especially when the system is accusing a human of being in the wrong.