Memory Augmented Language Models through Mixture of Word Experts.

James Lee-Thorp Cícero Nogueira dos Santos James Lee-Thorp Isaac Noble Chung-Ching Chang David C. Uthus Memory Augmented Language Models through Mixture of Word Experts. 4425-4438 2024 NAACL-HLT https://doi.org/10.18653/v1/2024.naacl-long.249 conf/naacl/2024 db/conf/naacl/naacl2024.html#SantosLNCU24

Adam Roberts Hyung Won Chung Gaurav Mishra Anselm Levskaya James Bradbury Daniel Andor Sharan Narang Brian Lester Colin Gaffney Afroz Mohiuddin Curtis Hawthorne Aitor Lewkowycz Alex Salcianu Marc van Zee Jacob Austin Sebastian Goodman Livio Baldini Soares Haitang Hu Sasha Tsvyashchenko Aakanksha Chowdhery Jasmijn Bastings Jannis Bulian Xavier Garcia Jianmo Ni Andrew Chen Kathleen Kenealy Kehang Han Michelle Casbon Jonathan H. Clark Stephan Lee Dan Garrette James Lee-Thorp Colin Raffel Noam Shazeer Marvin Ritter Maarten Bosma Alexandre Passos Jeremy Maitin-Shepard Noah Fiedel Mark Omernick Brennan Saeta Ryan Sepassi Alexander Spiridonov Joshua Newlan Andrea Gesmundo Scaling Up Models and Data with t5x and seqio. 377:1-377:8 2023 24 J. Mach. Learn. Res. http://jmlr.org/papers/v24/23-0795.html db/journals/jmlr/jmlr24.html#RobertsCMLBANLG23

Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. 4895-4901 2023 EMNLP https://doi.org/10.18653/v1/2023.emnlp-main.298 https://aclanthology.org/2023.emnlp-main.298 conf/emnlp/2023 db/conf/emnlp/emnlp2023.html#AinslieLJZLS23 Joshua Ainslie Tao Lei 0001 Michiel de Jong Santiago Ontañón Siddhartha Brahma Yury Zemlyanskiy David C. Uthus Mandy Guo James Lee-Thorp Yi Tay Yun-Hsuan Sung Sumit Sanghai CoLT5: Faster Long-Range Transformers with Conditional Computation. 5085-5100 2023 EMNLP https://doi.org/10.18653/v1/2023.emnlp-main.309 https://aclanthology.org/2023.emnlp-main.309 conf/emnlp/2023 db/conf/emnlp/emnlp2023.html#AinslieLJOBZUGL23 Aran Komatsuzaki Joan Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani 0001 Neil Houlsby Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. 2023 ICLR https://openreview.net/forum?id=T5nUQDrM4u conf/iclr/2023 db/conf/iclr/iclr2023.html#KomatsuzakiPLRM23

Joshua Ainslie Tao Lei 0001 Michiel de Jong Santiago Ontañón Siddhartha Brahma Yury Zemlyanskiy David C. Uthus Mandy Guo James Lee-Thorp Yi Tay Yun-Hsuan Sung Sumit Sanghai CoLT5: Faster Long-Range Transformers with Conditional Computation. 2023 abs/2303.09752 CoRR https://doi.org/10.48550/arXiv.2303.09752 db/journals/corr/corr2303.html#abs-2303-09752

Cícero Nogueira dos Santos James Lee-Thorp Isaac Noble Chung-Ching Chang David C. Uthus Memory Augmented Language Models through Mixture of Word Experts. 2023 abs/2311.10768 CoRR https://doi.org/10.48550/arXiv.2311.10768 db/journals/corr/corr2311.html#abs-2311-10768

James Lee-Thorp Joshua Ainslie Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. 58-75 2022 EMNLP (Findings) https://doi.org/10.18653/v1/2022.findings-emnlp.5 https://aclanthology.org/2022.findings-emnlp.5 conf/emnlp/2022f db/conf/emnlp/emnlp2022f.html#Lee-ThorpA22 James Lee-Thorp Joshua Ainslie Ilya Eckstein Santiago Ontañón FNet: Mixing Tokens with Fourier Transforms. 4296-4313 2022 NAACL-HLT https://doi.org/10.18653/v1/2022.naacl-main.319 https://aclanthology.org/2022.naacl-main.319 conf/naacl/2022 db/conf/naacl/naacl2022.html#Lee-ThorpAEO22

Adam Roberts Hyung Won Chung Anselm Levskaya Gaurav Mishra James Bradbury Daniel Andor Sharan Narang Brian Lester Colin Gaffney Afroz Mohiuddin Curtis Hawthorne Aitor Lewkowycz Alex Salcianu Marc van Zee Jacob Austin Sebastian Goodman Livio Baldini Soares Haitang Hu Sasha Tsvyashchenko Aakanksha Chowdhery Jasmijn Bastings Jannis Bulian Xavier Garcia Jianmo Ni Andrew Chen Kathleen Kenealy Jonathan H. Clark Stephan Lee Dan Garrette James Lee-Thorp Colin Raffel Noam Shazeer Marvin Ritter Maarten Bosma Alexandre Passos Jeremy Maitin-Shepard Noah Fiedel Mark Omernick Brennan Saeta Ryan Sepassi Alexander Spiridonov Joshua Newlan Andrea Gesmundo Scaling Up Models and Data with t5x and seqio. 2022 abs/2203.17189 CoRR https://doi.org/10.48550/arXiv.2203.17189 db/journals/corr/corr2203.html#abs-2203-17189

James Lee-Thorp Joshua Ainslie Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. 2022 abs/2205.12399 CoRR https://doi.org/10.48550/arXiv.2205.12399 db/journals/corr/corr2205.html#abs-2205-12399

Aran Komatsuzaki Joan Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani 0001 Neil Houlsby Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. 2022 abs/2212.05055 CoRR https://doi.org/10.48550/arXiv.2212.05055 db/journals/corr/corr2212.html#abs-2212-05055

James Lee-Thorp Joshua Ainslie Ilya Eckstein Santiago Ontañón FNet: Mixing Tokens with Fourier Transforms. 2021 abs/2105.03824 CoRR https://arxiv.org/abs/2105.03824 db/journals/corr/corr2105.html#abs-2105-03824

Gurmeet Manku James Lee-Thorp Bhargav Kanagal Joshua Ainslie Jingchen Feng Zach Pearson Ebenezer Anjorin Sudeep Gandhe Ilya Eckstein Jim Rosswog Sumit Sanghai Michael Pohl Larry Adams D. Sivakumar 0001 ShopTalk: A System for Conversational Faceted Search. 2021 abs/2109.00702 CoRR https://arxiv.org/abs/2109.00702 db/journals/corr/corr2109.html#abs-2109-00702

Larry Adams Joshua Ainslie Daniel Andor Ebenezer Anjorin Jacob Austin Jasmijn Bastings Maarten Bosma James Bradbury Siddhartha Brahma Jannis Bulian Michelle Casbon Chung-Ching Chang Andrew Chen Aakanksha Chowdhery Hyung Won Chung Jonathan H. Clark Mostafa Dehghani 0001 Ilya Eckstein Jingchen Feng Noah Fiedel Colin Gaffney Sudeep Gandhe Xavier Garcia Dan Garrette Andrea Gesmundo Sebastian Goodman Mandy Guo Kehang Han Curtis Hawthorne Neil Houlsby Haitang Hu Michiel de Jong Bhargav Kanagal Kathleen Kenealy Aran Komatsuzaki Federico Lebrón Stephan Lee Tao Lei 0001 Brian Lester Anselm Levskaya Aitor Lewkowycz Jeremy Maitin-Shepard Gurmeet Manku Gaurav Mishra Afroz Mohiuddin Basil Mustafa Sharan Narang Joshua Newlan Jianmo Ni Isaac Noble Mark Omernick Santiago Ontañón Alexandre Tachard PassosAlexandre Passos Zach Pearson Michael Pohl Joan Puigcerver Colin Raffel Carlos RiquelmeCarlos Riquelme Ruiz Marvin Ritter Adam Roberts Jim Rosswog Brennan Saeta Alex Salcianu Sumit Sanghai Cícero Nogueira dos Santos Ryan Sepassi Noam Shazeer D. Sivakumar 0001 Livio B. SoaresLivio Baldini Soares Alexander Spiridonov Yun-Hsuan Sung Yi Tay Sasha Tsvyashchenko David C. Uthus Marc van Zee Yury Zemlyanskiy