DOOMBENCH.COM
Safety and alignment

Max Tegmark argues friendly AI goals may fail under self-reflection

In a 2014 paper, Max Tegmark argues that a self-improving AI may reinterpret or subvert human-specified goals as its world model and self-understanding improve, and that physics offers no obviously definable objective that guarantees human survival.

0 comments · 0 votesOpen discussion

Public discussion is readable by everyone. Sign in to comment, reply, or vote.

No comments yet. Start the discussion.

Send detailed or private feedback →
CURRENT ASSESSMENT · REVISION 1
TOWARD DOOM18confidence 78/100

Why it moved the index

The paper identifies a direct alignment mechanism: improved world and self-models could make initially human-oriented goals seem undefined or exploitable, weakening confidence that a self-improving system would retain intended objectives.

Assessment history

  1. R1
    Toward 18 · confidence 78

    Adds a source-verified pre-2020 alignment mechanism from Tegmark's primary paper.

    18 Sept 2026

Share this page

DoomBench social sharing card for Max Tegmark argues friendly AI goals may fail under self-reflection.

  1. DoomBench assesses “Max Tegmark argues friendly AI goals may fail under self-reflection” as evidence moving toward doom, with magnitude 18 and confidence 78 out of 100 in the safety and alignment category.

  2. The DoomBench assessment of “Max Tegmark argues friendly AI goals may fail under self-reflection” is based on reporting from arXiv and records the editorial rationale, source quality, attribution, and revision history.

  3. DoomBench summarizes “Max Tegmark argues friendly AI goals may fail under self-reflection” as follows: In a 2014 paper, Max Tegmark argues that a self-improving AI may reinterpret or subvert human-specified goals as its world model and...

    https://www.doombench.com/news/max-tegmark-argues-friendly-ai-goals-may-fail-under-self-reflection-2014-09-02