An empirical series on whether an author's why can be recovered from what the code does. You cannot read a purpose off code when several purposes fit it equally well. Checking a declared purpose beats guessing, but dies when the honest declaration already justifies the behaviour.
How much of an author’s why can be recovered from, or checked against, what the code does? A later branch turns the same question on the models instead of the code: can you read a model’s parameters and predict how it behaves, without running it?
How it is run
Every experiment is pre-registered, meaning the decision rule is written down before the run. Everything uses public data. On the code side the work is done on abstracted representations only, operation and capability and AST-feature sequences, never functional malware. Raw generations and seeds are committed, so every reported number regenerates without new API calls.
What came out of it
- You cannot read a purpose off code when several purposes fit that code. Sorting records and then sending them somewhere fits “sync the user’s data” and “exfiltrate the user’s data” equally well. Ask a generator to write the code from each candidate purpose and both come back looking right. This kills two experiments, and it survives being measured a second time with a better ruler.
- Checking beats guessing, conditionally. Given a declared purpose, asking whether the code does more than that purpose justifies works, but only when the extra behaviour is something the declaration flatly rules out. On real apps that quietly track their users, the honest declaration already justifies the tracking capabilities, and the check dies.
- A shortcut that did not work, and an instrument that was broken. Weight inspection does not predict how badly a code model breaks under harmless rewrites. Worse, the thing being predicted turned out to be largely rigidity: a model that always says the same thing scored perfectly. The replacement is two-sided and scores such a model at zero.
What was interesting
Finding that the measuring instrument was wrong, after the result it produced had already been accepted. The second experiment is the one that matters, and it only exists because the first answer looked too clean.