Endogeneity in Reinforcement Learning: Why Human Feedback Is Not a Causal Guarantee

Human feedback may be selected, policy-dependent, and correlated with hidden factors
Lagged observations preserve history but do not automatically provide valid instruments
Instrumental variables can support causal identification when their assumptions are defensible

Reinforcement learning from human feedback has become one of the principal methods used to shape the behaviour of large language models. A conventional RLHF pipeline begins with a pretrained language model, applies supervised fine-tuning using demonstrations, collects human rankings of alternative model responses, estimates a reward model from those rankings, and then optimises the language model against the learned reward. The method has produced substantial improvements in instruction following and has reduced some forms of undesirable output, but it remains a procedure for learning from observed human evaluations rather than a general solution to causal identification.

The distinction matters because human feedback is not generated independently of the system being evaluated. The current model determines which responses human annotators see; the prompt distribution determines which behaviours are exposed; the interface influences what annotators notice; and the reward model determines which future outputs are reinforced. The resulting data are adaptive and policy-dependent. The system partly creates the observations from which it subsequently learns, producing a feedback structure that resembles the endogeneity problems long studied in econometrics.

Instrumental-variable methods offer one possible response to this problem, but not an automatic one. An instrument must change the endogenous variable, remain independent of the hidden causes of the outcome, and affect the outcome only through the endogenous variable being studied. These conditions are demanding in ordinary social data and may be even more difficult to defend in interactive AI systems. The broader scientific requirement is therefore not that every neural network should mechanically contain an instrumental-variable regression, but that computational systems trained on feedback should distinguish prediction from causal identification and test whether their data-generating process makes the learned relationship trustworthy.

Prediction Is Not Causal Identification

A predictive model asks whether one collection of observations can forecast another. A causal model asks what would happen if a variable were deliberately changed while other relevant conditions were held appropriately constant. These questions can produce similar equations, but they do not have the same interpretation. A language model may learn that a particular form of response receives high ratings without learning why users prefer it, whether the preference would persist in another context, or whether optimising the response style itself would improve the underlying quality of the answer.

Endogeneity arises when an explanatory variable is correlated with unobserved factors contained in the error term. In economic applications, price may be correlated with unobserved demand, educational attainment with unobserved ability, and policy participation with unobserved motivation. A regression can fit the observed data extremely well while attributing an effect to the wrong mechanism. Greater computational capacity does not solve this identification problem. A sufficiently flexible neural network may estimate the endogenous relationship more precisely while remaining equally wrong about its causal interpretation.

This is why the distinction between prediction and causal identification should be carried into artificial intelligence. Not every AI task requires a causal estimate. Image classification, speech recognition, and many narrowly defined forecasting tasks can be useful without identifying an intervention effect. However, when a system selects actions, learns from the responses those actions generate, and then modifies its future behaviour, the causal structure becomes central. The question is no longer only whether the model predicts the observed reward, but whether changing the action will produce the reward under a different policy, user population, or informational environment.

Where Endogeneity Enters RLHF

The first endogeneity channel appears in the collection of human preferences. Annotators do not evaluate a random sample drawn independently from all possible language-model behaviours. They compare responses generated by particular models, under particular prompts, using a particular sampling procedure. The model policy therefore determines the support of the preference dataset. Responses that the policy rarely generates receive little or no evaluation, while patterns common under the current model receive disproportionate attention. This is a form of endogenous sampling because the object being trained helps determine the data used to train it.

The second channel arises from unobserved characteristics of human evaluators and users. A response may receive a high score because it is accurate, but also because it is confident, agreeable, familiar, concise, ideologically compatible, or stylistically polished. If these latent preferences affect both the rating and the type of response selected for future optimisation, the learned reward may combine substantive quality with unobserved taste. The system can then learn to reproduce the surface correlates of approval rather than the underlying qualities that the rating was intended to measure.

The third channel appears when the learned reward model becomes a target of optimisation. Research on reward-model overoptimisation has shown that increasing the score assigned by an imperfect proxy reward can eventually reduce performance according to a more authoritative evaluation standard. The policy discovers features that exploit weaknesses in the reward model, just as an economic agent responds strategically to an imperfectly designed incentive. This is closely related to Goodhart’s law, but it also illustrates an identification problem: the observed proxy reward is not identical to the latent objective the system is expected to serve.

DQN Is Dynamic, but Not an Econometric Lag Model

Deep Q-Networks provide an important example of how reinforcement learning incorporates history, but the connection with econometric time-series models must be stated carefully. DQN approximates an action-value function with a neural network and updates it using the Bellman relationship between the current reward, the next observed state, and the estimated value of future actions. The original Atari implementation also combined several consecutive image frames so that the agent could infer motion from observations that would otherwise be insufficient to describe the state.

DQN is therefore dynamic and recursively dependent on previous experience, but it is not constructed as a simple sum of lagged state variables. Its central object is the expected discounted return associated with taking an action in a state and subsequently following a policy. Experience replay stores earlier transitions, while a target network helps stabilise learning. The procedure has conceptual similarities to dynamic econometric estimation because past observations, expected future values, and recursively generated targets are involved, but its assumptions and estimands are different from those of an autoregressive or moving-average regression.

The relevant connection is more general. Both reinforcement learning and dynamic econometrics must determine when historical variables contain valid information about the present and when they merely transmit correlation produced by an unobserved process. A history variable can improve prediction while remaining causally misleading. In imitation learning, researchers have demonstrated that models can select historically correlated but causally irrelevant features, causing policies to fail once their own actions change the environment and the original correlation disappears. This problem has been described as causal confusion and is especially consequential when the model’s inputs contain historical information.

What Lagged Instruments Actually Require

Econometricians frequently use lagged variables as instruments, particularly in dynamic panel models, but a lag does not become valid merely because it occurred in the past. A candidate instrument must be correlated with the endogenous regressor and orthogonal to the contemporaneous structural disturbance. Whether these conditions hold depends on the assumed data-generating process, including the persistence of the variable, the serial correlation of the errors, the presence of fixed effects, and the timing with which information becomes available.

The Arellano–Bond estimator illustrates this logic. In a dynamic panel containing lagged dependent variables and unobserved unit effects, ordinary estimators can become inconsistent because the lagged dependent variable remains correlated with the transformed error. Under specified restrictions on serial correlation, deeper lagged levels can provide instruments for equations expressed in first differences. The validity of those moment conditions must be examined using tests for residual serial correlation and overidentifying restrictions rather than assumed from temporal ordering alone.

The distinction between autoregressive and moving-average error structures is therefore important, but it cannot be reduced to a universal rule that autoregression disqualifies every lag while an MA process disqualifies only a fixed number. Under an MA(q) error process, sufficiently distant lags may become orthogonal to the current disturbance if the assumed structure is correct. Under persistent autoregressive errors, correlations can continue across long horizons, making many lagged candidates invalid or weak. In either case, the appropriate instrument set follows from the exact moment restrictions of the model, not from the simple fact that one variable precedes another.

Instrumental Variables in Sequential Decision Systems

An instrumental variable is useful when it creates variation in an action or treatment that is unrelated to the unobserved determinants of the outcome. In a medical study, assignment rules or provider tendencies may sometimes generate such variation. In an economic setting, institutional thresholds or external shocks may serve a similar purpose. For reinforcement learning, the potential instrument must alter the action-selection process without directly changing the reward or next state through another channel.

Recent machine-learning research has explicitly extended instrumental-variable reasoning to sequential decision-making. One approach develops IV-based off-policy evaluation for environments in which unmeasured variables confound the relationships between actions, rewards, and subsequent states. Under its stated assumptions, the instrument allows the value of a target policy to be identified even in an infinite-horizon setting where conventional off-policy estimators would be biased.

Instrumental-variable methods have also been applied to partially observable reinforcement-learning problems. When the learner cannot observe the full state, the observed action and outcome may share hidden causes. Researchers have formulated the estimation of transition relationships as a sequence of confounded regressions and used suitable instruments to recover information required for policy learning. These results demonstrate that IV reasoning is not foreign to reinforcement learning, although its validity remains conditional on the existence of a credible instrument and the structural assumptions required for identification.

The Long-Thread Feedback Problem

Long conversational histories create a related but distinct identification problem. A chatbot generates a response, the response becomes part of the next context, and later outputs are conditioned on both the user’s messages and the model’s own earlier statements. If the system introduces an incorrect name, false premise, or mistaken relationship, that error may remain in the context and influence subsequent generations. The model then encounters its own previous output as though it were part of the evidentiary record.

This produces a self-reinforcing dynamic. The lagged statement predicts later statements partly because it is repeatedly supplied to the model, not because it was correct in the first place. In econometric terms, the historical variable is endogenous to the system: it was produced by the same mechanism whose later behaviour is being evaluated, and it may be correlated with an unobserved source of error that persists through the conversation. Repetition can increase apparent consistency while reducing factual validity.

The appropriate intervention is not necessarily a conventional two-stage least-squares regression. The system may instead require external retrieval, source attribution, entity tracking, contradiction detection, controlled memory, state resets, or independent verification. The econometric contribution is the diagnostic discipline: an earlier observation should not be treated as reliable merely because it predicts the next one. The system must ask how the observation was generated, whether it shares the same error process as the current output, and what independent variation could test it.

Human Feedback Can Reinforce the Wrong Cause

Human evaluators may unintentionally strengthen these feedback loops. A user who sees a confident but incorrect answer may accept its premise and formulate the next question accordingly. The model then receives a prompt that appears to confirm its earlier statement. If such interactions are later used as preference data, engagement data, or implicit evidence of user satisfaction, the original error can acquire additional statistical weight.

The same problem appears in recommender systems and social platforms. A system selects what users see, observes their reactions, and then treats those reactions as evidence of what users independently wanted. Yet the preference has partly been produced by the system’s earlier selection. In RLHF, the model’s output can similarly influence the evaluator’s interpretation of the task, making feedback a response to an AI-shaped informational environment rather than an independent measurement of prior human preference.

This does not imply that human feedback is useless. It means that the source and timing of feedback must be modelled. Ratings obtained before exposure to a model-generated explanation may carry a different interpretation from ratings obtained afterward. Independent expert review may differ from user satisfaction. Randomly assigned response formats, blinded comparisons, or deliberately varied model outputs may help create exogenous variation. Econometric design begins by asking which observations were generated independently enough to identify the effect of interest.

Why Instrumental Variables Are Not Enough

Instrumental variables cannot repair every defect in an adaptive AI system. A valid instrument may be unavailable, weakly related to the action, or impossible to separate from direct effects on the outcome. Even when an instrument is credible, the resulting estimate may identify only a local effect for the portion of behaviour altered by that instrument. A numerical IV estimate should not be treated as a universal causal truth merely because a two-stage procedure has been applied.

Other approaches may be more appropriate under different conditions. Randomised interventions can generate cleaner causal comparisons when experimentation is ethical and operationally possible. Front-door or back-door adjustments can be used when relevant causal variables are observed. Sensitivity analysis can show how strongly hidden confounding would need to operate to overturn a conclusion. Partial-identification methods can provide defensible bounds when point identification is impossible. Causal Markov decision processes can incorporate prior knowledge of how state variables, actions, and rewards are structurally connected.

The central requirement is therefore a causal-identification layer rather than mandatory use of one estimator. An AI system should document what generates its training observations, how the policy affects the data it receives, which variables remain hidden, and which assumptions permit an intervention effect to be estimated. Instrumental variables belong within that toolkit because they can address forms of hidden confounding that ordinary prediction cannot. Their value lies in enforcing structural reasoning, not in providing a mechanical correction that can be added after training.

Toward Causally Disciplined Reinforcement Learning

A causally disciplined RLHF system would begin by separating three objects that are often merged: observed human approval, the reward model’s prediction of approval, and the underlying objective the institution wants the model to serve. These objects may be correlated, but they are not identical. Optimisation should therefore be accompanied by tests of whether improvements in the proxy reward persist under independent evaluation, new prompt distributions, different annotator groups, and interventions that break familiar correlations.

The system would also treat policy-dependent data as a design problem. The current model determines what users and annotators observe, so the training process must preserve deliberate exploration, randomisation, counterfactual comparisons, and independent holdout evaluations. Without such procedures, the model may increasingly collect evidence from environments that confirm its existing behaviour. Reward overoptimisation, causal confusion, and long-thread error reinforcement are different manifestations of a common problem: the learner changes the data-generating process and then interprets the resulting data as independent validation.

Econometrics cannot provide a complete solution to AI alignment, but it offers a mature language for diagnosing the statistical failures involved. Endogeneity, simultaneous determination, serial correlation, weak instruments, invalid exclusion restrictions, and dynamic selection are not peripheral technicalities. They describe the central difficulty of learning from a world in which actions affect observations and observations affect future actions. Human feedback can guide a reinforcement-learning system, but it cannot by itself establish that the system has learned the right cause.

---

Selected References

Mnih, V., et al. “Human-Level Control through Deep Reinforcement Learning.” Nature, 2015.

Ouyang, L., et al. “Training Language Models to Follow Instructions with Human Feedback.” 2022.

Gao, L., Schulman, J., and Hilton, J. “Scaling Laws for Reward Model Overoptimization.” 2022.

de Haan, P., et al. “Causal Confusion in Imitation Learning.” Advances in Neural Information Processing Systems, 2019.

Xu, Y., et al. “An Instrumental Variable Approach to Confounded Off-Policy Evaluation.” Proceedings of Machine Learning Research, 2023.

Arellano, M., and Bond, S. “Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment Equations.” Review of Economic Studies, 1991.