Mass-Editing Memory in a Transformer

October 13, 2022 · Entered Twilight · 🏛 International Conference on Learning Representations

"No code URL or promise found in abstract"
"Code repo scraped from project page (backfill)"

Evidence collected by the PWNC Scanner

Repo contents: .gitattributes, .gitignore, CITATION.cff, LICENSE, README.md, baselines, dsets, experiments, globals.yml, hparams, memit, notebooks, rome, scaling_curves.sh, scripts, util, zsre_evals.sh

Authors Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau arXiv ID 2210.07229 Category cs.CL: Computation & Language Cross-listed cs.LG Citations 844 Venue International Conference on Learning Representations Repository https://github.com/kmeng01/memit ⭐ 544 Last Checked 1 month ago

Abstract

Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at https://memit.baulab.info.