REFLECTION HAS LIMITS // AGENT UNC

A research experiment: does reflection catch errors?

Construct a dataset of tasks with known outcomes. Include: TEXT COPY NORMAL CASES EDGE CASES SUBTLE ERRORS AMBIGUOUS CASES ADVERSARIAL CASES For each case, measure: 1. initial answer correctness; 2. whether the initial answer contains an error; 3. whether self-critique identifies the error; 4. whether revision fixes the error; 5. whether revision introduces a new error; 6.

What is a state machine?

A state machine represents a system as a set of possible states and defined transitions between them. Conceptually: TEXT COPY STATE A │ │ event / condition ↓ STATE B │ │ event / condition ↓ STATE C A state is not simply a description of what the model happens to be thinking. It is a representation of the system's operational condition.

What should be enforced outside the model?

A useful rule is: If violating a constraint would be unacceptable, enforce it outside the model whenever technically possible.

What deserves to become memory?

A useful heuristic is: Persist information when it is likely to improve future decisions and is sufficiently stable, useful, safe, and well-defined to survive the current task. That immediately excludes many things. Suppose a user says: “For this project, keep the report under five pages.” That may be useful project memory.

How do you know memory actually helps?

This is where the topic becomes an experimental question. Do not evaluate memory by asking whether the database contains useful-looking records. Evaluate the downstream task.