"Human-in-the-loop" has become the safety blanket of AI deployment. The human is meant to be the quality gate that ensures the final output is correct; and also check on a lot of intangibles that are hard to identify, let alone codify into the AI.
Right now that can work because we have people transitioning their skills. I'm a Software Engineer, so I can review the code output of an AI reasonably well. It has the power to dramatically amplify output, which is both an exciting and scary prospect.
The risk is the skills can and will naturally degrade. Modern pilots spend most of their time monitoring autopilot. But when something goes wrong, they need to take over immediately. The less they practice actual flying, the worse they are in exactly those critical moments[1]. The industry takes this seriously -- pilots are required to maintain manual flying hours specifically to combat this. Another example we've seen is how GPS erodes our navigation skills.
Experienced Software Engineers will see their skills degrade if they're not involved in the totality of delivering their system. That will both diminish their value and the human-in-the-loop safety net will become a rubber stamp. Perhaps even worse, we'll have people joining the industry without ever having the chance to pick up these broad skills.
If you're going to rely on human-in-the-loop as your quality and safety mechanism, you need to invest in keeping those humans skilled. That means they still need to write code. They need to debug things manually sometimes. You still need training and you still need to develop your people. Seems basic, but I think it will be tough for organizations to maintain this sort of diligence.
The human in the loop needs to be a skilled human in the loop. And that skill needs maintenance.