Human Oversight of

AI Systems in Military Contexts

Growing Importance of AI in Combat

AI systems under human supervision might not be as effective as expected, especially in high-pressure combat situations. As AI continues to evolve and gain prominence in warfare, the global community, including governments, tech firms, and international organizations, must prioritize its safety and ethical deployment.

Human supervision is commonly emphasized in most AI safety protocols. In theory, humans serve as a safeguard against AI misuse or false outputs. This might involve reviewing and validating the content AI generates before action is taken.

Flaws in the Concept of Human Oversight

Despite the theory, real-world studies and military use cases suggest fundamental flaws in the notion of humans acting as effective checks on AI systems. Many current regulations advocate for human involvement, such as the European Union’s AI Act. This law requires that high-risk AI applications, including those using biometric data, must be verified by at least two qualified humans.

In February 2024, the UK government highlighted the importance of human oversight in military AI use. It emphasized the need for “meaningful human control” and accountability, making it clear that robots must not be the ones making life-or-death decisions. As of now, human operators still manage drone systems and bear responsibility for their actions. However, AI is steadily enhancing the independence and decision-making capabilities of these military technologies.

Autonomous Targeting and AI Systems

One critical advancement involves AI-powered target acquisition systems. These systems can identify and lock onto enemy targets, awaiting human authorization to strike. Although not widely adopted, the Gaza conflict revealed real-world applications. Israel reportedly used an AI system named Lavender to recommend targets and track their locations in coordination with other automated platforms.

Similarly, the U.S. Department of Defense launched Project Maven in 2017 to integrate AI into weapons systems. Initially designed for analysis, it gradually evolved into a robust AI-powered targeting system, significantly improving strike accuracy.

The Psychology Behind Automation Bias

While human oversight remains part of the operational chain, psychological research raises concerns. Mary Cummings, a U.S. academic, coined the term “automation bias,” referring to the tendency to place excessive trust in computer-generated decisions. This behavior can undermine the human role as a critical checkpoint.

In 1992, researchers Batya Friedman and Peter Kahn argued that reliance on computers could diminish moral responsibility. People may begin viewing decisions as outputs of machines, rather than outcomes of ethical judgment. They might even perceive AI systems as possessing agency, shifting moral accountability away from themselves.

Risks of Diminished Moral Agency

As military personnel become more accustomed to AI-assisted systems, there is a danger they may start viewing real individuals as mere data points on a screen. Such psychological detachment can compromise the ethical safeguards intended by human oversight. Even small algorithmic errors can lead to grave real-life consequences.

Reassessing Human Supervision Protocols

Various laws and global agreements affirm the necessity of human intervention in AI processes. Still, it is essential to ask whether long-term reliance on AI might desensitize human operators, dulling their ethical judgment and perception of responsibility.

To mitigate these risks, AI systems used in combat should include mandatory ethical training for human supervisors. Regular audits and psychological assessments should also be implemented to ensure that humans retain their role as moral agents.

Conclusion

While the involvement of humans in AI decision-making loops is intended to provide safety and accountability, multiple factors challenge its effectiveness. Psychological biases, automation dependency, and desensitization threaten the integrity of oversight mechanisms. As AI grows more autonomous, nations must rethink how to enforce meaningful human control and responsibility.