@phdthesis {pub6395,
	title = {Implicitly Cooperative Agents Through Impact Aware Learning},
	author = {David Rother},
	year = {2026},
	month = {April},
	abstract = {In recent years, the increasing integration of autonomous agents in human-centric environments has emphasized the importance of advanced interaction methodologies. This dissertation studies the challenges and solutions that arise in the deployment of robots and AI assistants, particularly focusing on the intricacies of human-agent interaction.
Despite the advancements in machine learning and neural networks, the scalability and
adaptability of interaction models in dynamic, human-populated environments remain an
open challenge.
Because predefining every possible interaction behavior is infeasible, one research direction,
Reinforcement Learning (RL), has evolved through leveraging computational advance-
ments and enables AI agents to learn from experience, an aspect that is lucrative to use in
human-robot interaction (HRI). A key aspect of HRI is the balance between autonomous
robotic capabilities and human oversight. This balance is intricate, as it must account for
the unpredictable nature of human behavior, akin to dealing with a partially observable
Markov decision process in RL. However, unlike RL where computational limits are often
the constraining factor, HRI{\textquoteright}s complexities are augmented by the need for intuitive and
adaptable interfaces, as well as ethical considerations and safety protocols. Robots that
operate around people must act under partial observability, under nonstationary dynamics
that arise from human behavior, and under strict safety and ethical requirements. In
these conditions, agents are expected to achieve coexistence by avoiding undue negative
impact on others, to coordinate by resolving interference among concurrent tasks, and
to cooperate by pursuing joint goals when appropriate. Reinforcement learning is well
suited for these settings because it allows agents to learn from experience, adapt online,
and reason about uncertainty without enumerating all interaction possibilities. Innovative
examples, such as collaborative robots in manufacturing provide a concrete illustration.
Such systems improve through continued interaction with human partners, learning to
anticipate needs and to adapt to diverse behaviors through algorithms that infer human
intent and that support safe and intuitive interfaces.
The primary objective of this dissertation is to enhance the decision-making capabilities of
autonomous agents in diverse and unpredictable interactions with humans. Traditional
planning and standard reinforcement learning often struggle to scale to many human participants and to generalize across novel interaction scenarios. To address these limita-
tions, the dissertation introduces frameworks and methodologies that improve interaction
dynamics between humans and artificial agents while maintaining tractability and safety.
Chapter 1 provides an introduction to reinforcement learning, particularly in mixed-motive
environments. It revisits theoretical principles, contrasts single-agent and multi-agent RL,
and presents case studies to highlight real-world applications. Significant challenges, such
as modeling other agents{\textquoteright} behaviors and ethical considerations, are discussed, concluding
with future research directions.
Chapter 2 addresses coexistence, which is the need to act effectively while accounting for
the impact of one agent on others who pursue their own tasks. The chapter develops an
entropy-weighted policy blending framework that explicitly disentangles task learning
from interaction learning. Separate policies are trained and are recombined at deploy-
ment through an entropy-weighted mechanism that trades off competence and caution
under task uncertainty. Empirical evaluations in a new multi-agent cooking environment
demonstrate improved coordination, robustness, and scalability in comparison to joint
learners that entangle task and interaction learning.
Chapter 3 extends these ideas and addresses open-ended coordination with dynamic
agent populations and changing task configurations. The chapter proposes Modular
Open Policies within the Double Open Bayesian Stochastic Game framework. This design
enables agents to adapt during execution to varying numbers of partners and to new
combinations of goals without retraining. Through online goal inference and modular
blending at execution time, the method establishes a foundation for generalizable multi-
agent coordination and improves sample efficiency. Validation in diverse experimental
scenarios shows that the approach adapts across partners and tasks while maintaining
performance.
Chapter 4 investigates frequent problems that occur when using one of the most prominent
approaches in RL, Soft Actor Critic (SAC), when applying it to settings with different
sparsity or absolute value of reward, as seen in the scenarios considered here. The chapter
presents a theoretical analysis that interprets SAC as KL regularized policy search rather
than the entropy maximization through the entropy bonus in the reward function. This
interpretation explains failure modes under sparse feedback and under approximation
error. Building on this analysis, the chapter derives the new algorithm Kullback-Leibler
Actor Critic (KLAC), which restores near-optimal guarantees while maintaining practical
exploration benefits, thereby providing a more stable basis for the design of impact-aware
policies.
In summary, this dissertation{\textquoteright}s contributions advance the design of reinforcement learning
agents that can coexist, coordinate, and cooperate implicitly in dynamic and uncertain
environments. These contributions add to the field of artificial intelligence by presenting novel methodologies for enhancing the interaction capabilities of autonomous agents in
human-centric environments. Through theoretical exploration and practical application,
this dissertation offers a significant advancement in the understanding and implementation
of multi-agent reinforcement learning to facilitate coexisting robots.},
	publisher = {Technische Universit{\"a}t Darmstadt},
	url = {https://tubiblio.ulb.tu-darmstadt.de/160882/},
	booktitle = {Technische Universit{\"a}t Darmstadt},
	city = {Darmstadt, Germany},
	pages = {129},
	institution = {Technische Universit{\"a}t Darmstadt}
}