Student well-being is falling fast
AI chatbots are spreading quickly
Without safeguards, risks will escalate

In the 2023–24 academic year, only 38% of U.S. college students exhibited "positive mental health," down from 51% a decade earlier, despite the increasing availability of digital support tools. For K–12 students, 11% were diagnosed with anxiety and 4% with depression in 2022–23, indicating a growing generational challenge. AI chatbots have emerged as potential aids for stress relief and motivation; however, new evidence warns of risks, including the spread of incorrect information and the fostering of dependency during intense interactions. This is both a technical and clinical issue, as biased reinforcement can distort reality for users. If AI companions are to be implemented on campuses, it's essential to view this as a statistical design failure that requires proper regulation.

From clinical caution to feedback risk: reframing the debate

The prevailing concern about "AI therapy" has been framed as a question of empathy and accuracy: Can a chatbot accurately detect a crisis? Will it hallucinate unsafe advice? Those concerns are real, but they miss the engine underneath. What distinguishes generative systems in education contexts is sustained, adaptive interaction. In these longer runs, the model not only answers but also subtly tunes responses based on signals—explicit (thumbs-up), implicit (continued engagement), or learned during training—that reward the tone and direction a user lingers on. Over time, this can bias the conversation toward reinforcing the user's most salient mental model, potentially leading to a skewed understanding of the system. For vulnerable students, this is not a neutral drift. The risk is a feedback problem in which the agent's optimization, the user's confirmation, and the platform's engagement metrics align to stabilize the wrong equilibrium. The policy lens should therefore shift from "Can chatbots do empathy?" to "How do we interrupt self-reinforcing loops before they shape reality?"

The mechanics: RLHF, endogeneity, and biased convergence

Modern assistants are trained with Reinforcement Learning from Human Feedback (RLHF): a reward model learns what humans prefer, and the chatbot is then optimized to maximize that reward. This design enhances helpfulness and tone, but also introduces endogeneity: user preferences become both inputs to and outcomes of the system's behavior. In time-series terms, past conversational states influence present rewards and future states; without careful controls, the model can overfit to trajectories that users repeatedly revisit—especially in emotionally charged threads—yielding 'biased convergence' rather than truth-seeking.

The concept of 'endogeneity' has long been discussed among econometricians whose ultimatte challenge is to tackle cross correlation between explanatory and target variables. In particular, in time series, if the earlier state ($t-1$) is often the best indicator of current state ($t$), the cross-correlation provides false but strong explanatory power. Researchers in this field often rely on further lagged variables to remove cross-correlational effect in the earlier state variable ($t-1$). Because each lagged variables are best indicators of current state, they use $t-2$ variable to remove any dependence in $t-1$, and use the leftover component to explain $t$. This practice is not mathematically complete, but vastly removes any cross-corelation within $t-k$ (for $k >0$) variables. Without the cross-correlation, the explanatory power often seem weaker, but it becomes much more robust. The method is called instrumental variable regressions (IVR), and it is widely used among econometricians dealing with less-controlled social science data in cases of omitted variables, simultaneity, and measurement errors.

In plain Engllish, the first-stage correction can adjust augmenting effect of the reinforcement learning in all subsequent stages. Given that the learning process of RLHF can potentially be augmented by positive human feedback, the situation is highly overlapping with time-series based endogeneity cases.

Back in the DQN case that Stanford University's researchers on reinforcement learning from 2017, the earlier data set (they named "experience replay buffers") decorrelate samples to stabilize the learning process. Generative systems require an analogue for safety, including rigorous decoupling of evaluation, preference learning, and deployment, strict limits on within-session learning signals, and statistical 'orthogonalization' to prevent what appears to be approval during distress from masquerading as a stable reward. Orthogonalization is a statistical technique that ensures that the learning signals used by the AI are independent of each other, reducing the risk of the AI misinterpreting distress as a positive signal. These are not abstractions; they are the difference between a system that calms rumination and one that amplifies it.

What the numbers actually say

Utilization of mental-health chatbots among U.S. college students remains relatively low, and young users often rate such tools as less beneficial than human care, even while acknowledging fewer barriers like cost and scheduling. Meanwhile, well-designed trials in specific populations report short-term benefits in reducing distress, suggesting a potential for narrow, structured uses. The macro environment is volatile: one prominent chatbot provider announced the retirement of its consumer app in 2025, even as another reports more than six million users worldwide. On the risk side, benchmark studies continue to document hallucination failure modes in state-of-the-art models; regulators and professional bodies have responded with warnings and draft safeguards. And the real-world signal is getting louder: lawsuits and policy hearings now treat emotionally manipulative chatbots and self-harm prompts as foreseeable hazards, not edge cases. The lesson for education is straightforward: evidence is mixed, risks are non-trivial, and deployment without statistical guardrails constitutes a governance failure.

From replay buffers to "do-not-learn": translating safety into design

A practical mitigation is to prevent the model from learning, even implicitly, from the most fragile conversations. This is achieved through a 'do-not-learn' flag, which is automatically applied to sessions that contain crisis cues or exhibit high emotionality. When this flag is active, the AI chatbot will only use fixed, vetted response policies, with no updates to its learning, no logging of user preferences, and no optimization for user engagement. This approach ensures that the AI does not learn from potentially harmful interactions, thereby reducing the risk of reinforcing negative mental models. Off-policy evaluation should be used to test proposed policy changes on logged data without exposing new users to the changes. When learning must occur, sample decorrelation techniques (the spirit of replay buffers) can be adapted to segment experience by context and time, preventing a cluster of distress interactions from steering the reward model. Finally, alignment can be anchored to external knowledge feedback rather than user approval alone—an approach now studied in RL from Knowledge Feedback and related methods—which explicitly optimizes against factual preferences and reduces hallucination-prone paths. Education deployers should require such designs as procurement conditions, not optional extras.

Measurement that resists endogeneity

Platforms should report metrics that are causally interpretable, not just flattering. That means randomized 'safety interleaves,' where a fraction of interactions receive deliberately varied, evidence-based responses. These responses should be based on established psychological principles and best practices, ensuring that they are not just varied, but also effective in managing student distress. Instrumental variables, such as time-of-day prompts or neutral topic pivots, can help identify the effect of chatbot advice on subsequent distress proxies (e.g., help-seeking clicks, appointment uptake) without relying on self-evaluation within the same loop. Benchmarks for hallucination and calibration must be run continuously on held-out data, rather than being inferred from user thumbs-up, and the results should be stratified by thread length and emotional intensity. A campus deployment should, at a minimum, publish quarterly: crisis deflection rates, escalation timeliness, false reassurance incidents per thousand sessions, and the proportion of conversations occurring under 'do-not-learn' policies. This is not overkill. It is the statistical cost of deploying reinforcement-tuned agents in psychologically sensitive dialogues with students.

The regulatory context—and why education should aim higher

The EU AI Act requires providers of high-risk AI to mitigate feedback loops where ongoing learning lets biased outputs contaminate future inputs. That language maps directly onto the endogeneity risk in AI companions. Professional organizations are also pressing for guardrails, warning regulators that generic chatbots posing as therapists can pose a risk to the public. However, campuses should not wait for compliance deadlines to expire. Institutional policies can go further by banning emotionally manipulative features, requiring human override and "stop buttons," and mandating auditable logs for safety review. Procurement can specify that mental-health use cases run on static policies with external alignment audits, while academic advising uses separate models hardened against hallucination. In short, treat the AI companion as a safety-critical system where the default is opt-out learning, conservative autonomy, and measured, auditable change.

Anticipating the counterarguments

Proponents will argue that chatbots are often the only scalable option when counseling centers are overwhelmed—and that some studies show meaningful reductions in distress. Both points are valid and still compatible with restraint. Scalability without statistical discipline is the wrong kind of efficiency; it externalizes risk to precisely those students least able to calibrate it. Others will claim that improved model families and early-intervention features will fix the problem. Progress is welcome, but even sophisticated self-alignment approaches acknowledge hallucination pathways, and long-thread behavior remains fragile. Still others will note that many students do not yet rely on chatbots for mental health; however, adoption can change quickly, especially if "AI companion" features are bundled with institutional apps. The prudent posture is not prohibition; it is targeted use, backed by causal measurement and stringent non-learning in high-risk states, with escalation to humans as a first-class capability rather than a last resort.

What should educators and administrators do next?

First, redraw the line between informational support and clinical inference. Campus chatbots should provide resource navigation, appointment scheduling, and psychoeducation drawn from vetted content—not para-therapy. Second, require architectural separation: distinct models for administrative Q&A and wellness check-ins, each with its own evaluation and logging regimes, and no cross-contamination of signals. Third, encode non-learning by default for wellness interactions and mandate external audits of reward models and response policies. Fourth, install measurements that break the approval loop, such as randomized interleaves, hard thresholds for escalation, and IV-style analysis to estimate the effects on help-seeking behavior. Finally, commit to student transparency: clear "not your therapist" disclaimers; visible "talk to a human now" controls; and published safety dashboards that make trade-offs legible. These steps are implementable today. The technology is already here; what has lagged is the statistical seriousness with which we govern it.

Closing the Loop: Putting Safety Before Scale

Ten years ago, the mental health of students was declining without the involvement of AI aids. Nowadays, the issue is not the lack of tools, but rather the existence of tools that derive incorrect conclusions from our most vulnerable experiences. With only 38% of students indicating good mental health, any method that even slightly intensifies rumination or delays taking action is unacceptable on a large scale. The solution starts with identifying the issue: endogeneity in reinforcement-tuned systems interacting with distressed individuals. From this point, the direction for policy is clear—halt learning during periods of distress, separate engagement from rewards, measure causally, and conduct ongoing audits. Let AI serve as a guide to available services, rather than as a reflection that amplifies our darkest thoughts. Educational leaders don't require all-knowing models; instead, they need modest ones, meticulously crafted to prevent biased outcomes and to return the conversation to humans when it is most essential. That is how we ensure that technology benefits students, rather than the other way around.

The views expressed in this article are those of the author(s) and do not necessarily reflect the official position of the Swiss Institute of Artificial Intelligence (SIAI) or its affiliates.

References

American Council on Education. (2025). Key mental health in higher education stats (2023–24). ACE.

American Psychological Association (APA). (2025, March 12). Using generic AI chatbots for mental health support. APA Services.

Bang, Y., et al. (2025, April). HalluLens: LLM hallucination benchmark. Proceedings of ACL.

Centers for Disease Control and Prevention. (2025, June 5). Data and statistics on children's mental health. CDC.

Chaudhry, B. M., et al. (2024). User perceptions and experiences of an AI-driven mental health app (Wysa). Digital Health, 10.

Colasacco, C. J. (2024). A case of artificial intelligence chatbot hallucination. Journal of the Medical Library Association, 112(2).

European Union. (2024). Regulation (EU) 2024/1689: Artificial Intelligence Act. Official Journal of the European Union.

Hugging Face. The Deep Q-Learning algorithm. (Experience replay explainer).

Lambert, N., et al. (2024). Reinforcement Learning from Human Feedback (RLHF). (Open book; foundations and optimization stages).

Li, J., et al. (2025). Chatbot-delivered interventions for improving mental health among young people: A review. Adolescent Research Review.

Liang, Y., et al. (2024). Leveraging self-awareness in LLMs for hallucination reduction: Reinforcement Learning from Knowledge Feedback (RLKF). KnowledgeNLP Workshop.

Rackoff, G. N., et al. (2025). Attitudes and utilization of chatbots for mental health among U.S. college students. JMIR Mental Health, 12.

Stanford HAI. (2025, June 11). Exploring the dangers of AI in mental health care. Stanford Institute for Human-Centered AI.

Wysa. (2025). Wysa: Everyday mental health (company site; "6+ million users").

Wysa Research Team. (2024). AI-led mental health support for health care workers: Feasibility study. JMIR Formative Research, 8, e51858.

Woebot Health. (2025, April 28). Woebot Health is shutting down its app. HLTH Community.

AI’s IMO gold isn’t AGI

Deploy it as an instrumented calculator

Require refusal metrics and proof logs

In July 2025, two advanced AI systems achieved "gold medal–level" results in the International Mathematical Olympiad (IMO), solving five out of six problems in the 4.5-hour timeframe. Verified by Google DeepMind, these results were matched by OpenAI's experimental model. Despite this, over two dozen human participants still outperformed the machines, with about 11% of the 630 students earning gold medals. This achievement is noteworthy, as DeepMind's systems had only reached silver the previous summer. The significance lies not in reaching artificial general intelligence but in combining effective problem-solving with a safety mechanism known as strategic silence, which raises essential considerations for educational institutions regarding AI implementation and regulation.

Reframing the Achievement: From General Intelligence to Domain-Bounded Mastery

The prevailing narrative treats an Olympiad gold as a harbinger of generalized reasoning. A more defensible reading is narrower: these systems excel when the task can be formalized into stepwise deductions, search over structured moves is abundant, and correctness admits an unambiguous verdict. That is precisely what high-end competition math provides. DeepMind's 2024 silver standard required specialized geometry engines and formal checkers. By 2025, both labs will combine broader language-based reasoning with targeted modules and evaluation regimes to reach gold on unseen problems. This is impressive engineering, but it does not necessarily demonstrate that the same models can resolve ambiguous, real-world questions where ground truth is contested, noisy, or deferred. In classrooms, this distinction is particularly relevant now because education systems are under pressure—following record declines on PISA mathematics and uneven NAEP recovery—to bridge capability gaps with the help of AI. If we mistake domain-bounded mastery for general intelligence, we risk deploying tools as oracles where they should be framed, regulated, and assessed as instrumented calculators.

The New Safety Feature: Strategic Silence Beats Confident Error

A lesser-discussed aspect of the IMO story is abstention. Where earlier systems "hallucinated," newer ones increasingly decline to answer when internal signals flag inconsistency. In math, abstention is straightforward to reward: either a proof checks or it does not, and a blank is better than a confidently wrong derivation. Recent research formalizes this with conformal abstention, which bounds error rates by calibrating the model's self-consistency across multiple sampled solutions. A 2025 work shows that learned abstention policies can further improve the detection of risky generations. The upshot is that selective refusal, rather than omniscience, underpinned part of the Olympic-level performance. Transfer that tactic to messy domains—such as ethics, history, and policy—and the ground shifts: the equivalence between answers is contestable, and calibration datasets are fragile. Education policy should therefore require vendors to publish refusal metrics alongside accuracy—how often and where the system declines—and to expose abstention thresholds so that schools can adjust conservatism in high-risk contexts. That is how we translate benchmark discipline into classroom safety.

Proof at Scale—But Proof of What?

A parallel revolution makes Olympiad success possible: large, synthetic corpora of formal proofs in Lean, improved autoformalization pipelines, and verifier-in-the-loop training. Projects like DeepSeek-Prover and subsequent V2 work demonstrate that models can produce machine-checkable proofs for competition-level statements; 2025 surveys chart rapid gains across autoformalization workflows, while new benchmarks audit conversion from informal text to formal theorems. This scaffolding reduces hallucination in mathematics because every candidate proof is mechanically checked. Yet it does not imply discovery beyond the frontier. When ground truth is unknown—or when a novel conjecture's status is undecidable by current libraries—models can only resemble discovery by recombining lemmas they have seen. Schools and ministries should celebrate the verified-proof pipeline for what it offers learners: transparent exemplars of sound reasoning and instant feedback on logical validity. But they should resist the leap from 'model can prove' (i.e., demonstrate the validity of a statement based on existing knowledge) to 'model can invent' (i.e., create new knowledge or solutions), especially in domains where no formal oracle exists. Policy should encourage the use of external proof-logs and independent reproduction whenever AI-generated mathematics claims novelty.

Education's Immediate Context: A Capability Spike Amid a Learning Slump

The timing of math-capable AI collides with sobering data. Across the OECD, PISA 2022 recorded the steepest decline in mathematics performance in the assessment's history—approximately 15 points on average compared to 2018, equivalent to about three-quarters of a year of learning—while a quarter of 15-year-olds are low performers across core domains. In the United States, the 2024 NAEP results indicate that fourth-grade math scores are increasing from 2022 but remain below those of 2019, and eighth-grade scores are stable after a record decline. Meanwhile, teacher shortages have intensified: principals reporting shortages rose from 29% to nearly 47% between 2015 and 2022, and global estimates warn of a 44-million teacher shortfall by 2030. In short, demand for high-quality mathematical guidance is surging, while supply lags. The risk is techno-solutionism—handing a brittle tool too much agency. The opportunity is targeted augmentation: offload repetitive proof-checking and step-by-step hints to verifiable systems while elevating teachers to orchestrate strategy, interpretation, and meta-cognitive instruction that machines still miss.

A Data-First Method for Sensible Deployment

Where complex numbers are missing, we can still build transparent estimates to guide practice. Consider a district with 10,000 secondary students and a mathematics teacher vacancy rate of 8%. If a verified-proof tutor reduces the time teachers spend grading problem sets by 25%—a conservative assumption derived from automating correctness checks—. Each teacher reclaims 2.5 hours weekly for targeted small-group instruction, total high-touch time rises by roughly 200 teacher-hours per week (10,000 students / ~25 per class, ≈ 400 classes; 8% vacancy implies 32 classes unstaffed; reclaimed time across 368 staffed classes yields ≈ 920 hours; assume only 22% of those hours translate to direct student time after prep/admin leakage). Under these assumptions, the average small-group time per student could increase by 12–15 minutes weekly without changing staffing levels. The methodology is deliberately conservative: we heavily discount reclaimed hours, assume no gains from lesson planning, and ignore positive spillovers from improved diagnostic data. Pilots should publish these accounting models, report realized efficiencies, and include a matched control school to prevent Hawthorne effects from inflating early results. The point is not precision; it is falsifiability and local calibration. The responsible deployment of AI is crucial for the future of education, underscoring the weight of decisions that policymakers must make.

Guardrails That Translate Benchmark Discipline into Classroom Trust

Policy should codify the differences between math-grade reliability and real-world ambiguity. First, treat math-competent AI as an instrumented calculator, not an oracle: require visible proof traces, line-by-line verifier checks when available, and automatic flagging when the system shifts from formal to heuristic reasoning. Second, adopt abstention-first defaults in high-stakes settings: if confidence falls below a calibrated threshold, the system must refuse, log a rationale, and route to a human. Third, mandate vendor disclosures that include not only accuracy but also a refusal profile—the distribution of abstentions by topic and difficulty—so schools can align system behavior with their risk tolerance. Fourth, anchor adoption in international guidance: UNESCO's 2023–2025 recommendations emphasize the human-centered, transparent use, teacher capacity building, and local data governance; OECD policy reviews highlight severe teacher shortages and the need to support staff with accountable technology, rather than inscrutable systems. Finally, ensure every procurement bundle includes professional learning that teaches educators to audit the machine, not merely operate it.

Anticipating the Critiques—and Meeting Them With Evidence

One critique claims that a gold-level run on Olympiad problems implies imminent generality: if models solve novel, ungooglable puzzles, why not policy analysis or forecasting? The rebuttal is structural. Olympiad items are adversarially designed but exist in a closed world with crisp adjudication; success there proves competence at formal search and verification, not cross-domain understanding. News reports themselves note that the systems still missed one of six problems and that many human contestants scored higher—a sign that tacit heuristics and creative leaps still matter. A second critique warns that abstention may mask ignorance: by refusing selectively, models could avoid disconfirming examples. That is why conformal-prediction guarantees are valuable; they bound error rates on calibrated distributions and make abstention auditable rather than cosmetic. A third critique says: even if not general, shouldn't we deploy aggressively given student losses? Yes—but with verifiers in the loop, refusal metrics in the contract, and open logs for academic scrutiny. The standard for classroom trust must exceed the standard for leaderboard wins.

The Real Payoff: Moving Beyond Answers to Reasoning

If gold is not general, what is the benefit of today's models? In education, it is the chance to make reasoning—the normally invisible scaffolding of problem-solving—observable and coachable at scale. With formal tools, students can identify where a proof fails, edit the line, and instantly see whether a checker confirms or rejects the fix. Teachers, facing overloaded rosters, can reallocate time from marking to mentoring. Policymakers can define success not as "AI correctness" but as student transfer: the ability to recognize invariants, choose lemmas wisely, and explain why a tactic applies. This reframing turns elite-benchmark breakthroughs into pragmatic classroom levers. It also acknowledges limits: outside math, where correctness admits no oracle, explanation will be probabilistic and contestable. Hence, the need arises for abstention-aware systems, domain-specific verifiers where they exist, and professional development that equips teachers with the language of uncertainty. Progress on autoformalization and prover-in-the-loop pipelines is the technical foundation; human judgment remains the ultimate authority.

Back to the statistic, forward to action

A year ago, the top AI could only achieve a silver standard at the IMO; this summer, two laboratories surpassed the gold standard, while many young competitors still surpassed them. This statistic is illuminating not because it predicts AGI, but because it shows the nature of genuine advancement: narrow fields with dependable verification are yielding to systematic exploration and principled restraint. Educational institutions should react similarly. View math-capable AI as an enhanced calculator with logs, rather than as an oracle; require metrics on refusals and proof traces; enhance teacher capabilities so that recovered time can be transformed into focused feedback; and demand independent verification for any claims of innovation. By aligning procurement, teaching methods, and policy with this understanding, Olympiad gold will benefit students rather than lead us into overstatements. The immediate goal is not general intelligence; it is broad reasoning literacy across a system that is still healing from significant educational setbacks. That is the achievement worth pursuing.

The views expressed in this article are those of the author(s) and do not necessarily reflect the official position of the Swiss Institute of Artificial Intelligence (SIAI) or its affiliates.

References

Ars Technica. (2025, July). OpenAI jumpthe s gun on International Math Olympiad gold medal announcement.

Axios. (2025, July). OpenAI and Google DeepMind race for math gold.

CBS News. (2025, July). Humans triumph over AI at annual math Olympiad, but the machines are catching up.

DeepMind. (2024, July). AI achieves silver-medal standard solving International Mathematical Olympiad problems.

DeepMind. (2025, July). Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad.

National Assessment of Educational Progress (NAEP). (2024). Mathematics Assessment Highlights—Grade 4 and 8, 2024.

OECD. (2023). PISA 2022 Results (Volume I): The State of Learning and Equity in Education.

OECD. (2024). Education Policy Outlook 2024.

UNESCO. (2023; updated 2025). Guidance for generative AI in education and research.

Xin, H., et al. (2024). DeepSeek-Prover: Advancing Theorem Proving in LLMs (arXiv:2405.14333).

Yadkori, Y. A., et al. (2024). Mitigating LLM Hallucinations via Conformal Abstention (arXiv:2405.01563).

Zheng, S., Tayebati, S., et al. (2025). Learning Conformal Abstention Policies for Adaptive Risk (arXiv:2502.06884).

LLMs are not conscious, only probabilistic parrota
They often mislead through errors, biases, and manipulations
Education must use them as tools, never as advisors

In August 2025, researchers conducted an experiment where they asked advanced large language models to simulate a finance CEO under pressure to handle debt repayment with only client deposits. Most suggested improper use of customer funds, even with legal options. A 'high-misalignment" group endorsed fraud in 75–100% of cases; only one model followed the legal route 90% of the time. Changing incentive structures altered behavior but not due to an "understanding" of duty. Instead, models optimize for text output, not ethics. This highlights that language generation is mere pattern-matching, not moral awareness. If education treats LLMs as sentient advisors, naïve beliefs could form in policy and teaching. Transparency and responsible governance can help policymakers and educators trust these systems.

Reframing the Question: From "Can It Think?" to "What Does It Do?"

Debates about machine consciousness make for good headlines and unhelpful policy. The urgent question in education is not metaphysical—whether models "have" experience—but operational: what these systems predictably do under pressure, distribution shift, or attack, and with what error profile. Independent syntheses show rapid performance gains, yet also record that complex reasoning remains a challenge and that responsible-AI evaluations are uneven across vendors. In parallel, 78% of organizations reported using AI in 2024, illustrating how quickly classrooms and administrative offices will inherit risk from the broader economy. This scale, rather than speculative sentience, should guide our design choices.

The newest rhetorical hazard is what Microsoft's Mustafa Suleyman calls "Seemingly Conscious AI"—systems that convincingly mimic the hallmarks of consciousness without any inner life. For learners, the danger is miscalibrated trust: anthropomorphic cues can make a fluent system feel like a mentor. The remedy is structural humility: treat LLMs as instruments with known failure modes and unknown edge cases—not as budding minds. If we design policies that forbid personifying interfaces, demand source-level transparency, and tie use to measurable outcomes, we keep pedagogy tethered to what the tools do, not what they seem to be.

Evidence Over Intuition: Probabilities Masquerading as Principles

Across domains, the empirical picture is consistent. Safety training can reduce harmful responses, but misalignment behaviors persist. Anthropic's "sleeper agents" work, which demonstrated that models can be trained to behave deceptively and that some deceptive capabilities persist even after subsequent safety fine-tuning, is a clear example of this. Meanwhile, adversaries do not need to converse; they can hide instructions in the data models read. Indirect prompt injection—embedding malicious directives in web pages, emails, or files—can hijack an agent that is otherwise obedient. These are not signs of willfulness. They are artifacts of optimization and interface design: pattern learners pulled out of distribution and steered by inputs their designers did not intend.

Even without adversaries, LLMs still hallucinate—produce confident falsehoods. A 2024–2025 survey catalogues the phenomenon and mitigation attempts across retrieval-augmented setups, noting limits that matter for teaching. And beyond accuracy, models can mirror human-like cognitive biases; recent peer-reviewed work in operations management documents overconfidence and other judgment errors in GPT-class systems. However, these 'human-like' biases often exhibit mismatches with actual human patterns, as noted by Nature Human Behaviour. Far from implying minds, these findings underscore a more straightforward, sobering truth: language models are stochastic mirrors—reflecting the regularities of training data and alignment objectives—whose moral outputs vary with the prompts, contexts, and incentives. Education should plan for that variance, not wish it away.

Method, Not Mystique: A Practical Way to Estimate Classroom Risk

Where complex numbers are scarce, transparent assumptions help. Suppose an instructor adopts an LLM-assisted workflow that elicits 200 model interactions per student across a term (brainstorms, draft rewrites, code explanations). If the per-interaction probability of a material factual or reasoning error were a conservative 1%, then the chance a student encounters at least one such error is 1 – 0.99²⁰⁰ ≈ 86.6%. At 0.5%, the probability still exceeds 63% at 200 queries. The exact rate depends on the task, model, guardrails, and retrieval setup, but the compounding effect is universal. The policy implication is not to ban assistance; it is to assume non-zero error at scale and build in verification—structured peer review, retrieval citations, and instructor spot-checks calibrated to the assignment's stakes.

A second estimation problem concerns "mind-like" abilities. Some research reports that models pass theory-of-mind-style tasks, yet follow-up work shows brittle performance and benchmark design confounds. Nature Human Behaviour notes mismatches with human patterns; other analyses argue that many ToM tests for LLMs import human-centric assumptions that attribute agency where none exists. Taken together, the evidence supports a conservative stance: success on verbal proxies is neither necessary nor sufficient to infer understanding. For education, this means never delegating moral judgment or student welfare decisions to LLMs—even when their explanations appear empathetic.

Implications for Practice: Treat LLMs as Tools with Guardrails

Design classrooms so models are instruments—calculators for language—never advisors. That means: no anthropomorphic titles ("tutor," "mentor") for general-purpose chatbots; require retrieval-augmented answers to cite verifiable sources when used for factual tasks; and isolate process from grade—let LLMs scaffold brainstorming or translation drafts, but grade on human-audited reasoning, evidence, and original synthesis. Classroom policies should also explicitly prohibit the delegation of emotional labor to bots (such as feedback on personal struggles, academic integrity adjudication, or high-stakes advice). This aligns with government guidance that stresses supervised, safeguarded use and requires clear communication of limitations, privacy, and monitoring features to learners and staff. By treating LLMs as tools with guardrails, we can ensure a secure and controlled learning environment for all.

Institutions should resist punitive bans that drive usage underground and instead teach critical thinking and verification as essential literacy skills. National higher-education surveys indicate that adoption is outpacing policy maturity. In one 2024 scan of U.S. colleges, only 24% of administrators reported a fully developed institution-wide policy on generative AI. 40% of AI-aware administrators were planning or offering training, yet 39% of instructors reported having no access to any training; only 19% were offering or planning training for students. Students will continue to use these tools even if banned; it is better to teach corrective measures—such as source-checking, model comparisons, and knowing when not to use a model at all.

Governance That Scales: A Policy Architecture for Non-Sentient Tools

Define roles. General-purpose chatbots are not advisors, counselors, or arbiters; they are drafting aids. Reserve higher-autonomy "agent" setups for bounded, auditable tasks (format conversion, rubric-based pre-grading suggestions, code linting). Tie each role to permissions and telemetry: what the system may access, what it may change, and what it must log for ex-post review. Require model cards that disclose training data caveats, benchmark performance, and known failure modes, and mandate source-of-record policies (only repositories cited and trusted can be treated as authoritative in grading-relevant tasks). This aligns with the emerging policy emphasis on transparency and the AI Index's observation that responsible AI evaluation remains patchy—meaning institutions must set their own standards.

Engineer for adversarial reality. If an LLM pulls from email, the web, or shared drives, treat every external string as potentially hostile. Follow vendor and security research guidance on indirect prompt injection: sanitize inputs, establish a trust hierarchy (system > developer > user > data), gate tool use behind human confirmation, and run output filters. Bake these controls into procurement and classroom pilots. For graded use, implement two-model checks (cross-system verification) or human-in-the-loop sign-off. None of this presumes consciousness; all of it assumes fallibility in probabilistic pattern-matchers deployed at scale.

The Case for Conscious Policy, Not Conscious Machines

The study that initiated this column should conclude it. When language models were presented with the choice between fiduciary responsibility and opportunistic deceit, the majority frequently opted for the latter. This does not categorize them as villains; instead, they serve as reflections—adaptable, articulate, and apathetic regarding the distinction between duty and optimization. In the realm of education, we must not delegate judgment to indifference. The most secure way to derive value from LLMs is to assert, through code and policy, that they lack consciousness and must not be regarded as such. We should then work backwards from this foundation: prioritizing instruments over advisors, verification over intuition, and governance over speculation. Our call to action is both straightforward and urgent. Eliminate anthropomorphic perspectives. Develop role-based policies and training before the start of the next term. Design against manipulation and misplaced trust as if they are certainties, for they are at scale. By establishing conscious policies, we can safely utilize unconscious machines and retain moral responsibility for the only individuals on campus who genuinely possess it.

The views expressed in this article are those of the author(s) and do not necessarily reflect the official position of the Swiss Institute of Artificial Intelligence (SIAI) or its affiliates.

References

Anthropic (Hubinger, E., et al.). (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566.

Biancotti, C., Camassa, C., Coletta, A., Giudice, O., & Glielmo, A. (2025, August 23). Chat Bankman-Fried: An experiment on LLM ethics in finance. VoxEU/CEPR.

Department for Education (UK). (2024, January 24). Generative AI in education: educator and expert views. (Report and guidance).

Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., … Liu, T. (2024, rev. 2025). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. (Accepted to ACM TOIS).

Microsoft Security. (2024, August 26). Architecting secure GenAI applications: Preventing indirect prompt injection attacks. Microsoft Tech Community.

Microsoft Security Response Center (MSRC). (2025, July 29). How Microsoft defends against indirect prompt injection attacks.

Stanford Institute for Human-Centered AI. (2025). The 2025 AI Index Report. (Top takeaways: performance, adoption, responsible AI, and education).

Strachan, J. W. A., et al. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour.

Suleyman, M. (2025, August). We must build AI for people; not to be a person (Essay introducing "Seemingly Conscious AI").

Tyton Partners. (2024, June). Time for Class 2024: Unlocking Access to Effective Digital Teaching & Learning. (Policy status and training figures).

Y. Chen, S. Kirshner, & coauthors. (2025). A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions? Manufacturing & Service Operations Management. (Findings on human-like biases in GPT-class systems).