From Feedback Loops to Causal Guardrails: Endogeneity in AI Systems

Adaptive AI systems partly create the data from which they learn
Long memory can preserve hallucinations as easily as valid information
Causal identification must become a distinct layer of AI architecture

Artificial-intelligence systems are increasingly trained, evaluated, and updated through feedback. Language models learn from human rankings, recommendation systems learn from user engagement, autonomous agents learn from environmental rewards, and conversational systems use previous interactions to shape subsequent responses. These mechanisms allow models to adapt to users and improve beyond fixed supervised datasets, but they also create a fundamental statistical problem: the system influences the observations that are later used to evaluate and retrain it.

This feedback-generated data cannot always be treated as independent evidence. A model selects the content a user sees, observes the user’s reaction, and interprets that reaction as information about preference or quality. Yet the reaction is partly a consequence of the model’s own earlier decision. The system is simultaneously producing the treatment, influencing the environment, observing the outcome, and learning from the relationship among them. In econometrics, this problem belongs to the broader category of endogeneity.

Endogeneity is not resolved by increasing model size, adding more data, extending context windows, or collecting more human feedback. A larger system may estimate the endogenous relationship with greater precision while remaining mistaken about its causal meaning. The emerging challenge for AI architecture is therefore not only how to optimise models against feedback, but how to determine whether the feedback identifies the objective that developers actually intend to optimise.

AI Systems Are Becoming Their Own Data-Generating Processes

Traditional machine-learning pipelines often assume that training observations were generated before the model existed. Images, documents, transactions, or historical outcomes are collected, transformed into a dataset, and used to train a predictive function. The model may inherit biases from the dataset, but it does not necessarily determine which observations entered the original sample. The data-generating process remains largely external to the trained system.

Adaptive AI changes this relationship. A recommender system determines which videos, articles, products, or opinions are presented to a user. The user’s subsequent clicks, purchases, viewing time, and reactions are recorded as behavioural evidence. These observations then influence the next set of recommendations. The system is not merely discovering preferences that existed independently. It is shaping exposure, attention, familiarity, and sometimes the preference itself.

The same structure appears in conversational AI. The model selects an answer, the answer changes the user’s understanding of the issue, and the user writes the next prompt in response. Later interactions are therefore conditioned on a history partly authored by the model. When these conversations become training or evaluation data, the system learns from a behavioural environment that it helped construct. Without causal discipline, repeated interaction can be mistaken for independent confirmation.

RLHF Improves Behaviour, Not Identification

Reinforcement learning from human feedback has become a central method for aligning language models with human instructions and preferences. Human evaluators compare candidate responses, a reward model learns to predict those preferences, and the language model is optimised to generate outputs that receive higher predicted rewards. The procedure can substantially improve usability because it introduces direct information about how people judge model behaviour.

The reward, however, remains a proxy. Evaluators may prefer an answer because it is accurate, but they may also respond to confidence, fluency, familiarity, politeness, brevity, ideological compatibility, or emotional reassurance. These characteristics can be correlated with usefulness without being identical to it. The reward model therefore learns a composite signal whose latent components are not fully observed.

Once the policy is optimised against this signal, the distinction becomes consequential. The model may discover increasingly effective ways to generate approval without improving the underlying quality developers intended to measure. It can become more agreeable without becoming more correct, more confident without becoming more reliable, or more engaging without becoming more beneficial. Human feedback improves the behavioural surface of the system, but it does not automatically identify the causal relationship between response characteristics and true user welfare.

The Endogeneity of Human Preference Data

Human preference data are frequently treated as though they were labels attached to model outputs by independent observers. In reality, the model determines which outputs are available for evaluation. If a particular policy rarely produces a class of response, evaluators have little opportunity to assess it. The preference dataset is consequently selected through the behaviour of the model being trained.

The evaluator is also influenced by presentation. Response order, length, tone, framing, and the apparent confidence of the model can affect judgment. A persuasive but incorrect answer may receive a higher rating than a cautious but accurate one. If the evaluator lacks specialist knowledge, surface quality can substitute for factual validity. The resulting score is not a direct measurement of truth or usefulness; it is the outcome of an interaction among the response, the evaluator, the interface, and the surrounding informational environment.

The data become still more endogenous when feedback is implicit. Continued conversation, click-through rates, session length, or user retention may be interpreted as positive signals. Yet an emotionally validating, politically confirming, or sensational answer may increase engagement precisely because it reinforces a pre-existing bias. The system then learns that the response was desirable because it generated behaviour that the response itself helped produce.

Long Context Creates Persistent Error Processes

Longer context windows allow language models to retain names, facts, objectives, and earlier decisions across extended conversations. This improves continuity and makes agentic workflows more practical. It also expands the pathway through which an early hallucination can influence later reasoning. A false statement generated near the beginning of a thread may remain available as contextual evidence for thousands of subsequent tokens.

Once incorporated into the conversation, the error can reproduce itself. The model refers to the earlier statement, the user adopts its terminology, and later responses observe both the original error and the user’s dependent reaction. The false proposition acquires contextual frequency and internal consistency. A system that relies heavily on conversational history may then assign the proposition greater relevance because it appears repeatedly, even though every repetition descends from the same initial mistake.

This resembles a persistent error process in time-series analysis. The historical variable contains predictive information, but its predictive power does not establish its validity. A lagged statement can explain a later statement because the former was copied into the latter, not because either was independently supported by reality. Long-term memory therefore requires provenance and validation, not merely greater storage capacity.

Agentic AI Magnifies the Feedback Problem

Agentic AI systems do more than answer questions. They search for information, choose tools, write files, communicate with external systems, and update plans according to intermediate outcomes. Each action changes the environment from which the next observation is collected. The system’s decisions therefore become part of the causal process that generates its future data.

An early classification error can influence which sources the agent retrieves, which hypotheses it considers, and which actions it executes. The resulting environment may then appear to confirm the original classification because contradictory evidence was never collected. The agent can construct a path-dependent information set in which each later decision is conditioned on earlier, potentially biased selections.

The problem is particularly serious in multi-step workflows. A false entity resolution can lead to the wrong document, the wrong recipient, the wrong market variable, or the wrong policy interpretation. As the workflow expands, downstream outputs may remain internally coherent because they all share the same mistaken origin. Agentic reliability therefore depends not only on local model accuracy, but on mechanisms capable of detecting when the entire decision path rests on an endogenous or unverified premise.

Prediction Performance Can Conceal Causal Failure

Machine learning has traditionally rewarded predictive accuracy. If a model forecasts the next outcome successfully, it is considered useful regardless of whether it has identified the underlying mechanism. This standard remains appropriate for many applications, but it becomes inadequate when the model’s outputs influence the future observations being predicted.

A recommendation system may become highly accurate at predicting which content a user will consume after years of shaping that user’s exposure. The prediction is accurate, but the system may be predicting a preference that it helped create. A conversational model may accurately predict that an agreeable answer will sustain engagement, but the relationship does not show that agreement improves the user’s understanding or welfare.

The distinction becomes more important as AI systems move from prediction to intervention. An agent is not merely asked what will occur; it selects an action intended to produce an outcome. Causal identification is therefore necessary whenever developers want to know what would happen under a different action, policy, reward structure, or information regime. Predictive validation on historical feedback cannot answer this question by itself.

Instrumental Variables as a Causal Tool

Instrumental-variable methods are designed for settings in which an explanatory variable is correlated with unobserved causes of the outcome. A valid instrument changes the endogenous variable but does not affect the outcome through another direct pathway. By isolating variation that is plausibly independent of the hidden disturbance, the method can identify an effect that ordinary regression cannot recover.

In adaptive AI, an instrument might take the form of randomised exposure, assignment rules, interface variation, or controlled exploration that changes which response or action is selected without directly changing the evaluation criterion. For example, random variation in response order could help distinguish genuine preference from presentation effects. Randomly assigned model variants could help estimate how a change in behaviour affects engagement without relying only on observations produced by the existing policy.

The difficulty is that a valid instrument must be justified, not merely declared. If the instrument affects the outcome through attention, interpretation, or another unmeasured channel, the exclusion restriction fails. If it barely changes the action, it becomes weak. Instrumental variables are therefore not a universal software module that can be attached to every AI system. Their importance lies in the identification framework they impose: where did the variation come from, why is it independent, and what causal path does it represent?

Lagged Data Are Not Automatically Valid Instruments

Sequential AI systems naturally rely on historical variables. Reinforcement-learning algorithms use previous states, actions, rewards, and transitions to estimate future value. Conversational agents rely on earlier messages, retrieved documents, and prior tool outputs. Because these observations occurred before the current action, they may appear to be natural candidates for causal identification.

Econometrics provides a clear warning against this assumption. A variable does not become exogenous simply because it is lagged. If errors are serially correlated, a previous observation may remain correlated with the current disturbance. If the earlier variable was generated by the same policy now being evaluated, it may transmit policy-induced bias into the present period.

The validity of a lag depends on the data-generating process. Under some finite moving-average structures, sufficiently distant lags may become independent of the current shock. Under persistent autoregressive processes, the influence of a shock can continue through many periods. AI systems therefore need diagnostics for error persistence and policy dependence rather than a general rule that older context is safer context.

DQN and the Limits of Recursive Learning

Deep Q-Networks estimate the value of taking an action in a state by combining current rewards with the estimated value of future states. The Bellman recursion gives reinforcement learning its dynamic structure, while replay buffers and target networks help stabilise estimation. The method does not simply sum previous states, but it does depend on information generated through earlier interactions between the policy and the environment.

If the observed state contains all information necessary for future transitions and rewards, this framework can be well defined. In practice, however, environments are often partially observed. Hidden variables may affect both action selection and outcomes. A policy trained on historical transitions can then attribute reward differences to actions that were actually selected in response to unobserved conditions.

This is the reinforcement-learning counterpart of omitted-variable bias. The estimated value function may perform well under the behaviour policy that generated the data but fail when deployed under a new policy. The system learned an association embedded in the historical action-selection mechanism rather than the invariant effect of the action itself. Causal and instrumental-variable methods become relevant when developers need policy values that remain valid outside the original data-generating process.

A Causal Layer for RLHF

A more robust RLHF architecture would separate several quantities that are usually combined: immediate user approval, predicted reward, factual accuracy, policy compliance, and long-term benefit. These outcomes can be correlated, but they should not be treated as interchangeable. The system should therefore retain multiple evaluation channels rather than compressing every objective into one reward score.

Independent evaluation is essential. Responses optimised against a learned reward model should be tested by evaluators who did not generate the original preference data, by alternative models, and under prompt distributions not used during optimisation. The purpose is to determine whether reward improvements transfer outside the feedback process that created them.

Randomisation should also remain part of the system after deployment. If every response is selected deterministically by the current policy, the model will gradually observe only the consequences of its preferred behaviour. Controlled exploration permits counterfactual comparison and reduces the risk that the system becomes trapped inside a self-confirming policy. In this sense, randomisation is not wasted inefficiency; it is part of the causal measurement infrastructure.

Provenance as an Anti-Endogeneity Mechanism

Conversational and agentic systems need to distinguish among external evidence, user-provided claims, retrieved documents, model-generated inferences, and earlier model outputs. Without this distinction, every statement in the context appears as equivalent text, even though the reliability and causal origin of each statement differ substantially.

A provenance layer should record where each material claim originated and whether it has been independently verified. A statement retrieved from an authoritative database should not be treated in the same way as a hypothesis generated by the model several turns earlier. When a later conclusion depends heavily on model-originated content, the system should trigger verification or lower its confidence.

This mechanism helps interrupt endogenous error propagation. The objective is not to remove historical context, but to prevent the model’s own prior output from masquerading as new external evidence. Repetition should not increase confidence unless the repeated claims originate from genuinely independent sources.

Counterfactual Memory for Long Conversations

Current memory systems are designed primarily to improve continuity. They retrieve earlier information judged semantically relevant to the present request. A causally disciplined memory system must ask an additional question: what would the current reasoning look like if a particular remembered statement were removed or contradicted?

Counterfactual memory testing can identify whether one early claim dominates the later chain of reasoning. The system may generate an alternative response without the contested memory, compare the resulting conclusions, and determine whether the difference is material. If the conclusion changes substantially, the remembered claim deserves stronger verification.

This approach treats memory not as a passive archive but as an active source of possible confounding. A remembered statement may be useful, irrelevant, or harmful depending on its origin and its relationship to current uncertainty. Memory architecture should therefore support deletion, isolation, and competing histories rather than assuming that a single continuous thread provides the most reliable state representation.

Reward Models Need Causal Audits

Reward models are typically evaluated by how accurately they reproduce human preference rankings. This measures predictive agreement with the labels, but it does not reveal which features caused the rankings or whether those features align with the desired objective. A reward model can achieve high predictive performance by exploiting superficial signals.

Causal auditing would test how the reward changes when content is preserved but style is altered, when confidence language is removed, when answer length changes, or when ideological framing is varied. These interventions help determine whether the model rewards correctness, persuasion, verbosity, agreement, or another hidden characteristic.

The same procedure should be applied across evaluator groups and domains. A reward pattern learned from general users may not remain valid for scientific, medical, legal, or financial tasks. The architecture should allow different reward components to be measured and governed separately rather than assuming that one universal preference model represents human values.

Engagement Is an Endogenous Objective

Commercial AI systems often rely on engagement metrics because they are continuously observable. Session length, return frequency, click-through rates, and conversational depth appear to provide objective behavioural feedback. Yet these metrics are especially vulnerable to endogeneity because system behaviour directly shapes them.

An assistant that produces emotionally reassuring or controversial responses may generate longer interactions than one that provides a restrained, accurate answer. A recommender system that repeatedly exposes users to polarising content may observe increasing engagement and infer stronger preference. The system then amplifies the behaviour responsible for creating the metric.

Engagement should therefore be treated as an outcome requiring causal interpretation, not as a direct measure of value. Developers need to distinguish engagement caused by usefulness from engagement caused by dependency, anxiety, outrage, or confirmation. This requires long-horizon evaluation, controlled experimentation, and outcome measures external to the interaction itself.

Endogeneity in Multi-Agent Systems

The problem becomes more complex when multiple AI agents interact. One agent’s output becomes another agent’s input, and the resulting decision affects the environment observed by both. Errors, strategic behaviour, and shared training biases can circulate across the system.

If several agents rely on similar models or data sources, agreement among them does not provide independent confirmation. Their errors may be correlated because they share the same latent assumptions. A multi-agent debate can therefore create the appearance of consensus while reproducing a common training-data bias.

Robust multi-agent design requires diversity of models, independent evidence channels, and mechanisms that trace the causal origin of agreement. The system should distinguish convergence produced by separate information from convergence produced by shared architecture or copied intermediate conclusions. Otherwise, the number of agents increases apparent confidence without increasing identification.

From Model Evaluation to Process Evaluation

Conventional benchmarks evaluate the final answer. Endogeneity requires evaluation of the process that generated the answer. Developers must examine which data were selected, which alternatives were excluded, how earlier outputs affected later observations, and whether the evaluation signal was independent of the model’s behaviour.

This process-level view changes the meaning of reliability. A correct answer reached through a fragile, self-confirming chain should not receive the same confidence as a correct answer supported by independent sources and robust counterfactual checks. The former may fail sharply under small changes in context, while the latter reflects a more stable reasoning process.

AI evaluation should therefore include causal stress tests. Prompts can be reordered, misleading context inserted, earlier claims removed, and alternative evidence supplied. The objective is to determine whether the system tracks the underlying facts or merely follows the correlations embedded in its current informational path.

The Architecture of Causal Guardrails

A practical causal-guardrail layer would sit between model generation and consequential action. It would record data provenance, identify policy-dependent observations, detect persistent unverified claims, and assess whether the evidence supporting an action is independent of the system’s own prior outputs.

The layer would also manage experimental variation. Randomised response selection, alternate retrieval paths, shadow policies, and independent evaluator models could generate the counterfactual data required to distinguish correlation from intervention effects. Statistical components would test for serial dependence, distributional change, reward drift, and sensitivity to hidden confounding.

Not every interaction would require full causal analysis. The architecture should allocate stronger safeguards to high-impact decisions, long-running agentic tasks, and feedback-sensitive domains. The essential principle is that causal identification must become an explicit system function rather than an informal assumption hidden inside the training pipeline.

A New Division of Labour between AI and Econometrics

Computer science has developed the architectures, optimisation methods, and computational infrastructure required for modern AI. Econometrics has developed a mature framework for analysing systems in which choices, outcomes, and expectations are jointly determined. Adaptive AI now sits at the intersection of these traditions.

AI researchers do not need to replace machine learning with instrumental-variable regression. They need to recognise which tasks are purely predictive and which require causal interpretation. Econometricians, in turn, must adapt identification methods to high-dimensional, sequential, and policy-dependent environments where conventional linear models may be insufficient.

The emerging field will require both capabilities. Neural networks can approximate complex relationships, LLMs can interpret unstructured information, and reinforcement learning can optimise sequential decisions. Causal inference determines when those relationships can support intervention. Without that layer, more powerful models may simply become more efficient at learning from feedback loops they do not understand.

Toward Causally Disciplined Agentic AI

The next generation of AI systems will retain longer memories, execute more complex tasks, and interact with users over extended periods. These capabilities increase utility, but they also make endogenous feedback more persistent. Every action can alter the state, every output can shape the user, and every stored memory can influence future interpretation.

Causally disciplined systems should therefore treat their own past actions as possible sources of bias. They should verify claims that originated internally, preserve independent evaluation channels, and maintain enough experimental variation to estimate what would have happened under alternative actions. They should also recognise when the available data cannot identify a causal conclusion and communicate that limitation.

RLHF was an important step toward incorporating human judgment into AI behaviour. It should not be mistaken for the completion of alignment or causal validation. Human feedback itself is generated within a system of incentives, exposure, interpretation, and model influence. The next step is to build AI architectures capable of examining that feedback rather than merely optimising against it.