Foundational challenges in assuring alignment and safety of large language models
This work identifies 18 foundational challenges in assuring the alignment and safety of large
language models (LLMs). These challenges are organized into three different categories …
language models (LLMs). These challenges are organized into three different categories …
Language instructed reinforcement learning for human-ai coordination
One of the fundamental quests of AI is to produce agents that coordinate well with humans.
This problem is challenging, especially in domains that lack high quality human behavioral …
This problem is challenging, especially in domains that lack high quality human behavioral …
Learning to influence human behavior with offline reinforcement learning
When interacting with people, AI agents do not just influence the state of the world--they also
influence the actions people take in response to the agent, and even their underlying …
influence the actions people take in response to the agent, and even their underlying …
Causal confusion and reward misidentification in preference-based reward learning
Learning policies via preference-based reward learning is an increasingly popular method
for customizing agent behavior, but has been shown anecdotally to be prone to spurious …
for customizing agent behavior, but has been shown anecdotally to be prone to spurious …
(Ir) rationality in AI: State of the Art, Research Challenges and Open Questions
O Macmillan-Scott, M Musolesi - arXiv preprint arXiv:2311.17165, 2023 - arxiv.org
The concept of rationality is central to the field of artificial intelligence. Whether we are
seeking to simulate human reasoning, or the goal is to achieve bounded optimality, we …
seeking to simulate human reasoning, or the goal is to achieve bounded optimality, we …
Learning zero-shot cooperation with humans, assuming humans are biased
There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an
agent that can cooperate with humans in a zero-shot fashion without using any human data …
agent that can cooperate with humans in a zero-shot fashion without using any human data …
Personalizing reinforcement learning from human feedback with variational preference learning
Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning
foundation models to human values and preferences. However, current RLHF techniques …
foundation models to human values and preferences. However, current RLHF techniques …
Beyond preferences in ai alignment
The dominant practice of AI alignment assumes (1) that preferences are an adequate
representation of human values,(2) that human rationality can be understood in terms of …
representation of human values,(2) that human rationality can be understood in terms of …
Learning to assist humans without inferring rewards
Assistive agents should make humans' lives easier. Classically, such assistance is studied
through the lens of inverse reinforcement learning, where an assistive agent (eg, a chatbot …
through the lens of inverse reinforcement learning, where an assistive agent (eg, a chatbot …
Learning to make adherence-aware advice
As artificial intelligence (AI) systems play an increasingly prominent role in human decision-
making, challenges surface in the realm of human-AI interactions. One challenge arises from …
making, challenges surface in the realm of human-AI interactions. One challenge arises from …