The United States Department of War (DoW) is rapidly scaling the integration of artificial intelligence (AI) to the warfighter, streamlining routine tasks so operators can focus on what matters most. The successful launch of GenAI.MIL signals both momentum and institutional commitment to embedding AI across operational environments. AI-enabled systems now operate across a spectrum of ever-increasing autonomy that directly shapes how humans engage in decision-making. At the foundational level, these systems function as analytic tools. They aggregate data, detect patterns, and generate recommendations. At this level, human operators remain the decision-makers, and their judgment is active, visible, and attributable. However, as autonomy of the AI component of the human-machine team increases, the relationship begins to shift. The AI systems assume a greater role in prioritizing data options, filtering information, and structuring decision pathways prior to human engagement. Thus, the human can transition from decision-maker to supervisor. Human operators retain decisional authority, but the tempo and complexity of operations compress the time available for reflection, deliberation and choices (of action or inaction, and available options of engagement). Hence, in practice, decisions are partially determined by the focus, scope and tenor of the data and information presented to the human element by the AI system. Such informational ranking, framing, and narrowing constrains independent human judgment.

With even greater AI system autonomy, human authority becomes more focally influential upstream in system design, programming, and authorization, rather than at the point of executive action. Consequently, authority is diffused across designers, commanders, and operators, thereby making attribution for system action and outcomes less clear and command and control less coherent.

This shift produces measurable cognitive effects. As systems demonstrate speed and consistency in desirable performance, human cognitive disposition and behavior adapt and becomes less questioning and more accepting of (if not reliant upon) the tool and technique being used. Over time, this can create a decision environment wherein machine-structured reasoning carries equivalent, or in some cases more weight than independent human analyses.

Concomitant to such cognitive and behavioral shifts, the broader implications of such decisional process become harder to assess. AI systems can translate complex human realities into data, probabilities, and predictive outputs, which can reframe how force is understood and applied. In this light, operational decisions may be increasingly evaluated in terms of efficiency, optimization, and precision, often privileging what can be objectively measured over what can be subjectively experienced or intellectually understood in context. Effects on the ground are filtered through the same data systems that informed the decision, which can create a feedback loop in which outcomes are inferred rather than directly observed. This can make it more difficult to assess unintended effects, or fully account for human costs, particularly in dynamic and contested environments where data are incomplete, delayed, or influenced by the system itself. Recognition of these factors prompts a growing requirement to continuously check and validate both ongoing strategy and its consequences, to ensure that AI-enabled decisions remain grounded in real-world conditions rather than solely in system-generated representations. At face value, this is not problematic, given that the expanding use of any tool or method characteristically requires early to mid-point assessment and re-evaluation to verify and validate system effectiveness in and across operational regimes. Evidently, this creates something of a paradox, at least during the adoption and “fine tuning” phases of technical evaluation in operational use: namely, that the efficiency of time and effort afforded by a tool may be initially burdened by the need to assure its capability in real-world settings and practices.  

Fair enough, but if the imperative is facilitation of the tempo of both 1) the adoption and employment of some technical asset and 2) the operational exigencies and contingencies in which it is deployed, there is temptation to forego such sustained evaluation. This can be problematic. For iteratively autonomous AI system use, absent continuous validation, risk can compound rapidly and systemically. When this occurs, operational decisions can become reinforced by internally generated outputs rather than externally verified conditions, and commanders have a higher probability of acting upon perceived effectiveness as defined by model coherence, versus ground truth. This fosters an illusion of precision and control, wherein operations appear to be optimized by the AI system, while actually increasing the probability for proactively unrecognized second- and third-order effects in the operational environment. In such ways, the gap between intent and outcome tends to widens, attribution can become obscured, and escalation risk increases as actions rely on incomplete or recursively validated data. Under these conditions, the decision loop loses its grounding, becomes self-referential, reinforcing its own assumptions, propagating error, and degrading the integrity of command at the point where judgment and accountability are most important and consequential.

For this reason, meaningful human engagement becomes the decisive variable. This imperative is supported by Department of Defense Directive 3000.09, which mandates human involvement in all operational engagement(s) of AI-based systems; and indeed, at present, human designers build the system, commanders authorize its use, and operators must monitor its performance. Clearly, such human enterprise is creating AI systems that are ever more capable in (1) their diversity, specificity and precision of tasks, and (2) the relative autonomy and agency of such processes and activities. Harkening back to the human cognitive tendency toward diminishing doubt, broadening acceptance, and increasing reliance (if not dependence) upon progressively more able tools, we opine that the iterative expansion of AI and the resultant amenability of human users can create conditions that are ideal for blurring the boundaries, requirements, and parameters of human-AI interaction, and control. So, despite the invocation of Department of Defense Directive 3000.09, yet unanswered questions remain as to if, to what extent and how humans can, and (we post) should retain and define their involvement as AI systems become increasingly autonomous, generative and agentic.

Next week: Part Two: A Proposed System Toward a Solution

Disclaimer

The views and opinions expressed in this essay are those of the authors and do not necessarily reflect those of the United States government, Department of War or the National Defense University.

Dr. Elise Annett is a Research Fellow at in the Program for Disruptive Technology and Future Warfare of the Institute for National Strategic Studies at the National Defense University. 

Dr. James Giordano is Head of the Center for Strategic Deterrence and Study of Weapons of Mass Destruction, and Program Lead for Disruptive Technology and Future Warfare of the Institute for National Strategic Studies at the National Defense University.