This research introduces a framework for constructing fact-storing MLPs in transformers that achieve asymptotic parameter efficiency aligned with information...
Level: expert
By Owen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré
Category: research