A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
This research dissects the mechanism behind attention sinks in GPT-2 models, revealing how specific circuit interactions drive disproportionate token attenti...