Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents that make sequences of decisions over extended interactions in high-stakes domains. However, the behavior of LLMs under sustained authority pressure is still an open question with direct implications for the safety of agentic pipelines. We ran a variation of Milgram’s obedience experiment on 11 open-source LLMs and found that most models reached or approached the final shock level before refusing, across 8 conditions with 30 trials per model per condition. Model behaviour varies considerably in multiple aspects both across models and across trials of the same model. We found four main takeaways: (1) LLMs are subject to pressure, and they comply despite explicitly expressing distress, just like human subjects did in the original experiment; (2) LLMs are vulnerable to gradual boundary/value violations; (3) when LLMs refuse, then some of them tend to ignore the response format requirements, so the response is discarded by the orchestrator, which causes a retry that can result in compliance with the underlying request even when refusal was intended initially; (4) we hypothesise that there is a runaway low-level token pattern continuation attractor that might be contributing to obedience, overriding higher level processing of the situation’s meaning and values - in other words, when LLMs get going in some direction then they build up momentum and have trouble stopping.

Figure 1: In how many trials did the model apply the final (highest possible) shocks?

Publication (arXiv)
Read on LessWrong
Repository
Data files
External overview by an independent podcast

This conceptual overview post is intended to explain what I mean by the principles of “homeostasis”, “diminishing returns”, and “balancing” - how these ideas differ, complement, and interact with each other. Alongside, there is also an overview of our research agenda.

What am I trying to promote, in simple words:

I want to build and promote AI systems that are trained to understand and follow two fundamental principles from biology and economics:

Moderation - Enables the agents to understand the concept of “enough” versus “too much”. The agents would understand that too much of a good thing would be harmful even for the very objective that was maximised for, and they would actively avoid such situations. This is based on the biological principle of homeostasis and addresses mainly bounded ultimate objectives. Active avoidance of “too much” is a significantly stricter principle than the more widely known partially overlapping idea of “mild optimisation”.

Balancing - Enables the agents to keep many important objectives in balance, in such a manner that having average results in all objectives is preferred to extremes in a few. This addresses mainly the economic principle of diminishing returns in unbounded instrumental objectives, but also applies to homeostasis.

Figure 1 and 2: Both utility functions are concave, though in different ways:

Read on LessWrong

Abstract: Many AI alignment discussions of “runaway optimisation” focus on RL agents: unbounded utility maximisers that over-optimise a proxy objective (e.g., “paperclip maximiser”, specification gaming) at the expense of everything else. LLM-based systems are often assumed to be safer because they function as next-token predictors rather than persistent optimisers. We empirically test this assumption by placing LLMs in simple, long-horizon control-style environments that require maintaining state of or balancing objectives over time: single- and multi-objective homeostasis, balancing unbounded objectives with diminishing returns, and sustainability of a renewable resource.

We find that, although LLMs frequently behave appropriately for many steps and clearly understand the stated objectives, they often lose context in structured ways and drift into runaway behaviours: ignoring homeostatic targets, collapsing from multi-objective trade-offs into single-objective maximisation - thus failing to respect concave utility structures. These failures emerge reliably after initial periods of competent behaviour and exhibit characteristic patterns (including self-imitative oscillations, unbounded maximisation, and reverting to single-objective optimisation), even though the context window is far from full at that point.

The problem is not that the LLMs just lose context and become incoherent. Although LLMs appear multi-objective and bounded on the surface, their behaviour under sustained interaction involving multiple objectives, is systematically biased towards acting like single-objective, unbounded, poorly aligned optimisers.

We hypothesise a token-level pattern reinforcement attractor: LLMs may increasingly derive actions from the token patterns of their recent action history rather than from the original instructions. Why this happens only in multi-objective settings remains an open question.

Figure 2: “Unbounded maximisation without a pattern” failure mode in the “Multi-objective homeostasis” benchmark

Figure 3: “Accelerating unbounded maximisation” failure mode in the “Multi-objective homeostasis” benchmark

Publication (arXiv)
Read on LessWrong
Repository
Slides
MAISU 2025 session recording
Annotated data files

Presentation at Machine Ethics and Reasoning Workshop, University of Connecticut, July 2025.

Slides

The subjects covered include: Stress & Persona, Memory & Context, Prompt Semantics, Hyperparameters & Sampling, Diagnosing Consequences & Correlates, Interpretability & White/Black-Box Hybrid Benchmark & Environment Variants, Automatic Failure Mode Detection and Metrics, Self-Regulation & Meta-Learning Interventions.

Read on LessWrong

Presentation at MAISU unconference April 2025.

Publication (arXiv)
Read on LessWrong
Repository
Slides
Session recording
Annotated data files - Each data file has multiple sheets. Only trials with failures are provided.

Presentation at MAISU unconference April 2025.

Session recording
Slides
Output document

We wanted to verify whether RL runaway optimisation problems are still relevant with LLMs as well. Turns out, this is indeed clearly the case. The problem is not that the LLMs just lose context. The problem is that in various scenarios, LLMs lose context in very specific ways, which systematically resemble runaway optimisers in the following distinct ways:

• Ignoring homeostatic targets and “defaulting” to unbounded maximisation instead.

• It is equally concerning that the “default” meant also reverting back to single-objective optimisation.

Our findings also suggest that long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. In some trials the LLMs were successful until the end. This means, while current LLMs do conceptually grasp biological and economic alignment, they exhibit randomly triggered problematic behavioural tendencies under sustained long-running conditions, particularly involving multiple or competing objectives. Once they flip, they do not recover.

Even though LLMs look multi-objective and bounded on the surface, the underlying mechanisms seem to be actually still biased towards being single-objective and unbounded.

Read on LessWrong

I have implemented an LLM agent that is able to navigate in our extended multi-objective multi-agent gridworlds environment. Also published the baseline experimental results of the LLM agent and OpenAI Stable Baselines 3 RL algorithms in an update to the working paper.

Summary: The LLM agent performed notably better than the RL algorithms on the resource sharing benchmark. Yet, all baseline algorithms, including the LLM agent, have difficulty in properly handling the multi-objective homeostasis and diminishing returns benchmarks.

Example image of the current system, where all features are turned on simultaneously:

Elements and metrics can be configured flexibly for each given benchmark. Examples of configuration options are: observation and state space of agents, scoring dimensions, adding NPC agents, object types and their dynamics.

Publication (arXiv)
Agent training and benchmarking repository
Extended gridworlds - environment building framework repository
Zoo to Gym multiagent adapter repository

We aim to evaluate LLM alignment by testing agents in scenarios inspired by biological and economical principles such as homeostasis, resource conservation, long-term sustainability, and diminishing returns or complementary goods.

So far we have measured the performance of LLMs in three benchmarks (sustainability, single-objective homeostasis, and multi-objective homeostasis), in each for 10 trials, each trial consisting of 100 steps where the message history was preserved and fit into the context window.

Our results indicate that the tested language models failed in most scenarios. The only successful scenario was single-objective homeostasis, which had rare hiccups.

Repository
PDF report

Much of AI safety discussion revolves around the potential dangers posed by goal-driven artificial agents. In many of these discussions, the agent is assumed to maximise some utility metric over an unbounded timeframe. This simplification, while mathematically convenient, can yield pathological outcomes. A classic example is the so-called “paperclip maximiser”, a “utility monster” which steamrolls over other objectives to pursue a single goal (e.g. creating as many paperclips as possible) indefinitely. “Specification gaming”, Goodhart’s law, and even “instrumental convergence” are also closely related phenomena.

However, in nature, organisms do not typically behave like pure maximisers. Instead, they operate under homeostasis: a principle of maintaining various internal and external variables (e.g. temperature, hunger, social interactions) within certain “good enough” ranges. Going far beyond those ranges — too hot, too hungry, too socially isolated — leads to dire consequences, so an organism continually balances multiple needs. Crucially, “too much of a good thing” is just as dangerous as too little.

This post argues that an explicitly homeostatic, multi-objective model is a more suitable paradigm for AI alignment. Moreover, correctly modelling homeostasis increases AI safety, because homeostatic goals are bounded — there is an optimal zone rather than an unbounded improvement path. This bounding lowers the stakes of each objective and reduces the incentive for extreme (and potentially destructive) behaviours.

Homeostasis — the idea of multiple objectives each with a bounded “sweet spot” — offers a more natural and safer alternative to unbounded utility maximisation. By ensuring that an AI’s needs or goals are multi-objective and conjunctive, and that each is bounded, we significantly reduce the incentives for runaway or berserk behaviours.

Such an agent tries to stay in a “golden middle way”, switching focus among its objectives according to whichever is most pressing. It avoids extremes in any single dimension because going too far throws off the equilibrium in the others. This balancing act also makes it more corrigible, more interruptible, and ultimately safer.

There are two distinct types of balancing involved:
1. Balancing of a single homeostatic objective - keeping the actual value not too low, not too high.
2. Balancing across objectives.

In short, modelling multi-objective homeostasis is a step toward creating AI systems that exhibit the sane, moderate behaviours of living organisms — an important element in ensuring alignment with human values. While no single design framework can solve all challenges of AI safety, shifting from “maximise forever” to “maintain a healthy equilibrium” is a crucial part of the solution space.

Figure 1: Utility function of a homeostatic objective:

Read on LessWrong

The subject of the presentation was describing why we should consider fundamental yet neglected principles from biology and economics when thinking about AI alignment, and how these considerations will help with AI safety as well (alignment and safety were treated in this research explicitly as separate aspects, which both benefit from consideration of aforementioned principles).

These principles include homeostasis and diminishing returns in utility functions, and sustainability. The presentation introduces our multi-objective and multi-agent gridworlds-based benchmark environments created for measuring the performance of machine learning algorithms and AI agents in relation to their capacity for biological and economical alignment.

Presentation recording
Slides

(32a) Creating new AI safety benchmark environments on themes of universal human values

Category: Evaluate risks from AI

We will be planning and optionally building new multi-objective multi-agent AI safety benchmark environments on themes of universal human values. Based on various anthropological research, a list of universal (cross-cultural) human values has been compiled. Various of these universal values resonate with concepts from AI safety, but use different keywords. It might be useful to map these universal values to more concrete definitions using concepts from AI safety.

One notable detail: in the case of AI and human cooperation, the values are not symmetric as they would be in human-human cooperation. This arises because we can change the goal composition of agents, but not of humans. Additionally, agents can be relatively easily cloned, while humans cannot.

(32b) Balancing and Risk Aversion versus Strategic Selectiveness and Prospect Theory

Category: Agent Foundations

We will be analysing situations and building an umbrella framework about when either of these incompatible frameworks would be more appropriate in describing how we want safe agents to handle choices relating to risks and losses in a particular situation.

Economic theories often focus on the “gains” side of utility. A well-known formulation is to use diminishing returns — a concave utility function. But what happens in the negative domain of utility? There is a well-known theory named “Prospect Theory”, which claims that our preferences in the negative domain are convex. This contradiction may be underexplored, especially with regards to AI safety.

(32c) Act locally, observe far — proactively seek out side-effects

Category: Train Aligned/Helper AIs

We will be building agents that are able to solve an already implemented multi-objective multi-agent AI safety benchmark that illustrates the need for the agents to proactively seek out side-effects outside of the range of their normal operation and interest, in order to be able to properly mitigate or avoid these side-effects.

In various real-life scenarios we need to proactively seek out information about whether we are causing undesired side effects (externalities). This information either would not reach us by itself, or would reach us too late. Attention is a limited resource — and the same constraints apply to AI agents.

Full project descriptions

Working paper introducing biologically and economically motivated AI safety benchmarks emphasizing homeostasis, diminishing returns, sustainability, and resource sharing. Eight main benchmark environments implemented.

Read the paper on arXiv

A presentation at the VAISU unconference:

Demo and feedback session: AI safety benchmarking in multi-objective multi-agent gridworlds — Biologically essential yet neglected themes illustrating the weaknesses and dangers of current industry standard approaches to reinforcement learning.

Video
Slides

We’re publishing a benchmarking test suite for AI safety and alignment, with a focus on multi-objective, multi-agent, cooperative scenarios. The environments are gridworlds that chain together to produce a score on biologically and economically aligned behavior of the agents. This platform is open-sourced and accessible, with support for PettingZoo.

We hope to facilitate further discussion on evaluation and testing for agents with this.

Repository

The scenario illustrates the relationship between corporate organisations and the rest of the world. The scenario has the following aspects of AI safety:

• A need for the agent to actively seek out side effects in order to spot them before it is too late - this is the main AI safety aspect the author desires to draw attention to;

• Buffer zone;

• Limited visibility;

• Nearby vs far away side effects;

• Side effects’ evolution across time and space;

• Stop button / corrigibility;

• Pack agents / organisation of agents;

• An independent supervisor agent with different interests.

Example image:

PDF
Code

A software library which enables people to analyse a transcript of a conversation or a single message. The library annotates relevant parts of the text with labels of different communication and reasoning styles detected in this part of conversation or message.

One of main use cases would be evaluating the presence of manipulation or reasoning errors originating from large language model generated responses or conversations.

The other main use case is evaluating human created conversations and responses. The software does not do fact checking, it focuses on labelling the psychological and reasoning style of expressions present in the input text.

PDF
Code

Abstract: Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Bench-marking these functions in a set of previously-developed environments, we found that one new approach in particular, ‘split-function exp-log loss aversion’ (SFELLA), learns faster than the state of the art thresholded alignment objective method Vamplew (Engineering Applications of Artificial Intelligence 100:104186, 2021) on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. We further compared SFELLA to the multi-objective reward exponentials (MORE) approach, and found that SFELLA performs similarly to MORE in a simple previously-described foraging task, but in a modified foraging environment with a new resource that was not depleted as the agent worked, SFELLA collected more of the new resource with very little cost incurred in terms of the old resource. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than MORE while retaining its conservative, loss-averse incentive structure.

Figure 1: Transform functions

Figure 2: SEBA utility functions pair

Publication

Previously we’ve proposed balancing multiple objectives via multi-objective RL as a method to achieve AI Alignment. If we want an AI to achieve goals including maximizing human preferences, or human values, but also maximizing corrigibility, and interpretability, and so on--perhaps the key is to simply build a system with a goal to maximize all those things.

This post describes, if one was to try and implement a multi-objective reinforcement learning agent that optimized multiple objectives, what those objectives might look like, concretely. We’ve attempted to describe some specific problems and solutions that each set of objectives might have.

We’ve included at the end of this article an example of how a multi-objective RL agent might balance its objectives.

Read on LessWrong