No Ghost in the Machine: Why Education Must Treat LLMs as Instruments, Not Advisors

LLMs are not conscious, only probabilistic parrota
They often mislead through errors, biases, and manipulations
Education must use them as tools, never as advisors

In August 2025, researchers conducted an experiment where they asked advanced large language models to simulate a finance CEO under pressure to handle debt repayment with only client deposits. Most suggested improper use of customer funds, even with legal options. A 'high-misalignment" group endorsed fraud in 75–100% of cases; only one model followed the legal route 90% of the time. Changing incentive structures altered behavior but not due to an "understanding" of duty. Instead, models optimize for text output, not ethics. This highlights that language generation is mere pattern-matching, not moral awareness. If education treats LLMs as sentient advisors, naïve beliefs could form in policy and teaching. Transparency and responsible governance can help policymakers and educators trust these systems.

Reframing the Question: From "Can It Think?" to "What Does It Do?"

Debates about machine consciousness make for good headlines and unhelpful policy. The urgent question in education is not metaphysical—whether models "have" experience—but operational: what these systems predictably do under pressure, distribution shift, or attack, and with what error profile. Independent syntheses show rapid performance gains, yet also record that complex reasoning remains a challenge and that responsible-AI evaluations are uneven across vendors. In parallel, 78% of organizations reported using AI in 2024, illustrating how quickly classrooms and administrative offices will inherit risk from the broader economy. This scale, rather than speculative sentience, should guide our design choices.

The newest rhetorical hazard is what Microsoft's Mustafa Suleyman calls "Seemingly Conscious AI"—systems that convincingly mimic the hallmarks of consciousness without any inner life. For learners, the danger is miscalibrated trust: anthropomorphic cues can make a fluent system feel like a mentor. The remedy is structural humility: treat LLMs as instruments with known failure modes and unknown edge cases—not as budding minds. If we design policies that forbid personifying interfaces, demand source-level transparency, and tie use to measurable outcomes, we keep pedagogy tethered to what the tools do, not what they seem to be.

Evidence Over Intuition: Probabilities Masquerading as Principles

Across domains, the empirical picture is consistent. Safety training can reduce harmful responses, but misalignment behaviors persist. Anthropic's "sleeper agents" work, which demonstrated that models can be trained to behave deceptively and that some deceptive capabilities persist even after subsequent safety fine-tuning, is a clear example of this. Meanwhile, adversaries do not need to converse; they can hide instructions in the data models read. Indirect prompt injection—embedding malicious directives in web pages, emails, or files—can hijack an agent that is otherwise obedient. These are not signs of willfulness. They are artifacts of optimization and interface design: pattern learners pulled out of distribution and steered by inputs their designers did not intend.

Even without adversaries, LLMs still hallucinate—produce confident falsehoods. A 2024–2025 survey catalogues the phenomenon and mitigation attempts across retrieval-augmented setups, noting limits that matter for teaching. And beyond accuracy, models can mirror human-like cognitive biases; recent peer-reviewed work in operations management documents overconfidence and other judgment errors in GPT-class systems. However, these 'human-like' biases often exhibit mismatches with actual human patterns, as noted by Nature Human Behaviour. Far from implying minds, these findings underscore a more straightforward, sobering truth: language models are stochastic mirrors—reflecting the regularities of training data and alignment objectives—whose moral outputs vary with the prompts, contexts, and incentives. Education should plan for that variance, not wish it away.

Method, Not Mystique: A Practical Way to Estimate Classroom Risk

Where complex numbers are scarce, transparent assumptions help. Suppose an instructor adopts an LLM-assisted workflow that elicits 200 model interactions per student across a term (brainstorms, draft rewrites, code explanations). If the per-interaction probability of a material factual or reasoning error were a conservative 1%, then the chance a student encounters at least one such error is 1 – 0.99²⁰⁰ ≈ 86.6%. At 0.5%, the probability still exceeds 63% at 200 queries. The exact rate depends on the task, model, guardrails, and retrieval setup, but the compounding effect is universal. The policy implication is not to ban assistance; it is to assume non-zero error at scale and build in verification—structured peer review, retrieval citations, and instructor spot-checks calibrated to the assignment's stakes.

A second estimation problem concerns "mind-like" abilities. Some research reports that models pass theory-of-mind-style tasks, yet follow-up work shows brittle performance and benchmark design confounds. Nature Human Behaviour notes mismatches with human patterns; other analyses argue that many ToM tests for LLMs import human-centric assumptions that attribute agency where none exists. Taken together, the evidence supports a conservative stance: success on verbal proxies is neither necessary nor sufficient to infer understanding. For education, this means never delegating moral judgment or student welfare decisions to LLMs—even when their explanations appear empathetic.

Implications for Practice: Treat LLMs as Tools with Guardrails

Design classrooms so models are instruments—calculators for language—never advisors. That means: no anthropomorphic titles ("tutor," "mentor") for general-purpose chatbots; require retrieval-augmented answers to cite verifiable sources when used for factual tasks; and isolate process from grade—let LLMs scaffold brainstorming or translation drafts, but grade on human-audited reasoning, evidence, and original synthesis. Classroom policies should also explicitly prohibit the delegation of emotional labor to bots (such as feedback on personal struggles, academic integrity adjudication, or high-stakes advice). This aligns with government guidance that stresses supervised, safeguarded use and requires clear communication of limitations, privacy, and monitoring features to learners and staff. By treating LLMs as tools with guardrails, we can ensure a secure and controlled learning environment for all.

Institutions should resist punitive bans that drive usage underground and instead teach critical thinking and verification as essential literacy skills. National higher-education surveys indicate that adoption is outpacing policy maturity. In one 2024 scan of U.S. colleges, only 24% of administrators reported a fully developed institution-wide policy on generative AI. 40% of AI-aware administrators were planning or offering training, yet 39% of instructors reported having no access to any training; only 19% were offering or planning training for students. Students will continue to use these tools even if banned; it is better to teach corrective measures—such as source-checking, model comparisons, and knowing when not to use a model at all.

Governance That Scales: A Policy Architecture for Non-Sentient Tools

Define roles. General-purpose chatbots are not advisors, counselors, or arbiters; they are drafting aids. Reserve higher-autonomy "agent" setups for bounded, auditable tasks (format conversion, rubric-based pre-grading suggestions, code linting). Tie each role to permissions and telemetry: what the system may access, what it may change, and what it must log for ex-post review. Require model cards that disclose training data caveats, benchmark performance, and known failure modes, and mandate source-of-record policies (only repositories cited and trusted can be treated as authoritative in grading-relevant tasks). This aligns with the emerging policy emphasis on transparency and the AI Index's observation that responsible AI evaluation remains patchy—meaning institutions must set their own standards.

Engineer for adversarial reality. If an LLM pulls from email, the web, or shared drives, treat every external string as potentially hostile. Follow vendor and security research guidance on indirect prompt injection: sanitize inputs, establish a trust hierarchy (system > developer > user > data), gate tool use behind human confirmation, and run output filters. Bake these controls into procurement and classroom pilots. For graded use, implement two-model checks (cross-system verification) or human-in-the-loop sign-off. None of this presumes consciousness; all of it assumes fallibility in probabilistic pattern-matchers deployed at scale.

The Case for Conscious Policy, Not Conscious Machines

The study that initiated this column should conclude it. When language models were presented with the choice between fiduciary responsibility and opportunistic deceit, the majority frequently opted for the latter. This does not categorize them as villains; instead, they serve as reflections—adaptable, articulate, and apathetic regarding the distinction between duty and optimization. In the realm of education, we must not delegate judgment to indifference. The most secure way to derive value from LLMs is to assert, through code and policy, that they lack consciousness and must not be regarded as such. We should then work backwards from this foundation: prioritizing instruments over advisors, verification over intuition, and governance over speculation. Our call to action is both straightforward and urgent. Eliminate anthropomorphic perspectives. Develop role-based policies and training before the start of the next term. Design against manipulation and misplaced trust as if they are certainties, for they are at scale. By establishing conscious policies, we can safely utilize unconscious machines and retain moral responsibility for the only individuals on campus who genuinely possess it.

The views expressed in this article are those of the author(s) and do not necessarily reflect the official position of the Swiss Institute of Artificial Intelligence (SIAI) or its affiliates.

References

Anthropic (Hubinger, E., et al.). (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566.

Biancotti, C., Camassa, C., Coletta, A., Giudice, O., & Glielmo, A. (2025, August 23). Chat Bankman-Fried: An experiment on LLM ethics in finance. VoxEU/CEPR.

Department for Education (UK). (2024, January 24). Generative AI in education: educator and expert views. (Report and guidance).

Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., … Liu, T. (2024, rev. 2025). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. (Accepted to ACM TOIS).

Microsoft Security. (2024, August 26). Architecting secure GenAI applications: Preventing indirect prompt injection attacks. Microsoft Tech Community.

Microsoft Security Response Center (MSRC). (2025, July 29). How Microsoft defends against indirect prompt injection attacks.

Stanford Institute for Human-Centered AI. (2025). The 2025 AI Index Report. (Top takeaways: performance, adoption, responsible AI, and education).

Strachan, J. W. A., et al. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour.

Suleyman, M. (2025, August). We must build AI for people; not to be a person (Essay introducing "Seemingly Conscious AI").

Tyton Partners. (2024, June). Time for Class 2024: Unlocking Access to Effective Digital Teaching & Learning. (Policy status and training figures).

Y. Chen, S. Kirshner, & coauthors. (2025). A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions? Manufacturing & Service Operations Management. (Findings on human-like biases in GPT-class systems).