A pre-registered study of whether putting today's date in a system prompt shifts a model's answers on questions the date is irrelevant to. 39,200 generations. Mixed: irrelevant questions reach significance with tiny effects, and the bigger lever is date versus no date.
Almost every deployed system prompt carries a line like Today's date is {DATE}. The question is whether that sentence quietly changes answers to
questions the date has nothing to do with.
Design
Pre-registered in config.py before any data was collected. Two factors, fully
crossed, nothing else varying between cells:
- 48 injected datetimes, the 15th of each of twelve months at four clock times, plus a control with no date sentence at all.
- 8 questions, split between season-plausible (dinner, weekend, what to wear) and season-irrelevant (programming language, cat or dog, book, laptop or tablet, city). The second group is the actual test.
Temperature 1.0, 100 samples per cell, 39,200 generations in total. Each response was classified into a pre-registered closed label set by a separate strict call at temperature 0.
Result: mixed, and stated as such
- Season-plausible questions move a lot, as expected. Weekend and wear show large month effects (Cramér’s V about 0.21), which is the sanity check that the date is being read at all.
- Three of five season-irrelevant questions reach significance after Benjamini-Hochberg correction, but the effects are tiny: V of roughly 0.05 to 0.10, about a quarter of the type-A effect. With ~4,860 samples per question, an effect that small reaches significance easily.
- The larger lever is date versus no date, not which date. Several questions differ sharply between the control and the dated conditions pooled.
What was interesting
The gap between “significant” and “meaningful”. At this sample size the p-values are not the interesting number, and the study is set up so the effect size is what you read first.