Complacency in automated systems
Automation complacency is a behavioural phenomenon in which human operators of automated or AI-assisted systems reduce their active monitoring, critical scrutiny, and independent verification of system outputs, attributing unwarranted reliability to automated decisions. The concept has been studied across domains including aviation, medical diagnosis, industrial process control, and, more recently, AI-assisted knowledge work. In financial and investment contexts, automation complacency poses specific risks to the quality of due diligence, valuation analysis, and portfolio monitoring when practitioners over-rely on AI-generated outputs without maintaining independent analytical judgment.
Background
The study of automation complacency originates in aviation psychology and human factors research from the 1980s and 1990s. Parasuraman and Manzey (2010) conducted influential work demonstrating that automation use consistently leads to reduced monitoring of automated systems and failure to detect automation errors, a pattern they termed "complacency-related automation misuse".[1] Wickens and Dixon (2007) established that individuals calibrate their trust in automation based on historical reliability, leading to systematic over-trust when automation has performed well across a long run of cases.[2]
The emergence of AI-assisted analytical tools in professional services—legal research, financial analysis, medical diagnosis, and due diligence—extended these concerns to knowledge work contexts where the consequences of undetected errors are severe. High-stakes professional environments are particularly vulnerable because practitioners face competing pressures of speed and accuracy, and AI tools are typically adopted precisely because they accelerate workflows.
Description and Methodology
Automation complacency in AI-assisted knowledge work manifests along a spectrum from vigilance to over-reliance. Wickens' signal detection framework suggests that practitioners become less sensitive to signal-noise distinctions as their baseline expectation of error diminishes with AI use. In due diligence contexts, this translates into reduced scrutiny of AI-extracted financial figures, acceptance of AI-generated risk summaries without source verification, and progressive atrophy of the independent analytical procedures that would historically have caught errors.
Coney (2026) distinguishes verification atrophy as a specific sub-phenomenon: the gradual weakening of a practitioner's verification habits specifically because repeated AI accuracy reinforces the belief that verification is unnecessary. This atrophy is compounded in organisations where AI outputs are presented with high visual confidence—polished formatting, citation appearances, numerical precision—regardless of underlying reliability.[3]
The WorkWise Verification Framework, proposed in that research, specifies cognitive and structural interventions across four domains: (1) interface design features that surface uncertainty signals and require active confirmation steps before acceptance; (2) workflow protocols mandating independent verification at defined confidence thresholds; (3) organisational protocols including randomised audit procedures and verification KPIs; and (4) training programmes designed to maintain analytical calibration even as AI assistance increases.
Research by Dell’Acqua et al. (2023) observed an asymmetric pattern relevant to automation complacency: below-average performers showed quality improvements of approximately 43% when using AI, while above-average performers showed only 17% improvement, partly because skilled practitioners were more likely to critically evaluate AI outputs.[4]
Applications
Automation complacency has been identified as a material risk across several financial services applications.
In due diligence, the primary concern is the acceptance of AI-generated EBITDA normalisations, risk assessments, or legal summaries without independent verification, which can allow material errors to survive into investment committee presentations.
In portfolio monitoring, AI systems that flag covenant breaches or performance anomalies may habituate analysts to respond primarily to AI alerts rather than conducting independent monitoring sweeps, creating blind spots for deterioration patterns the AI model was not trained to detect.
In investment committee processes, AI-generated investment summaries and scoring outputs can anchor committee deliberation, reducing the scope of independent analysis conducted by committee members and concentrating risk assessment around the AI model's training data and embedded assumptions.
In public markets and trading contexts, automation complacency has been linked to over-reliance on algorithmic signals, contributing to the Flash Crash of May 2010 and related incidents in which human operators failed to override automated systems displaying anomalous behaviour.[5]
Challenges
Measuring automation complacency presents methodological difficulties. The phenomenon operates over time and requires longitudinal observation of practitioner behaviour, comparison of AI-on versus AI-off performance, and identification of verification failures that are typically not documented as such.
Interface and workflow countermeasures face adoption resistance: friction deliberately introduced to prompt verification is experienced by practitioners as reduced efficiency, creating pressure to disable or bypass the controls the design intends to enforce.
The relationship between AI reliability and complacency is non-linear. As AI systems become more accurate, the rational Bayesian response is to monitor them less. Interventions that counteract complacency must therefore be calibrated not to the average error rate but to the tail risk of catastrophic failures that rare errors can produce in high-stakes environments.
Organisational cultures that prioritise speed and throughput may structurally disincentivise the verification behaviours that complacency countermeasures require.
See Also
- Skill erosion paradox
- AI-assisted due diligence
- Human-AI workflow design
- AI governance in private equity
- Decision velocity
References
- ↑ Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.
- ↑ Wickens, C. D., & Dixon, S. R. (2007). The benefits of imperfect automation: A synthesis. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 51(4), 212–216.
- ↑ Coney, L. (2026). Combating Automation Complacency in Financial Due Diligence: A Deep Dive into Verification Atrophy, Cognitive Interventions, and Interface Design for Epistemic Humility. SSRN. DOI: 10.2139/ssrn.6111107.
- ↑ Dell’Acqua, F., McFowland, E., Mollick, E., et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013.
- ↑ Kirilenko, A., Kyle, A. S., Samadi, M., & Tuzun, T. (2017). The Flash Crash: High-frequency trading in an electronic market. Journal of Finance, 72(3), 967–998.
- Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
- McKinsey & Company. (2024). AI in Financial Services: Navigating Risk and Reward. McKinsey Global Institute.
- Financial Stability Board. (2024). Artificial Intelligence and Machine Learning in Financial Services: Supervisory Perspectives. FSB.
- Coney, L. (2026). AI Governance Across the Deal Lifecycle: From Sourcing Through Portfolio Monitoring. SSRN. DOI: 10.2139/ssrn.6274559.
