Functional Interpolation for Relative Positions improves Long Context Transformers.

Sumit Sanghai Shanda Li Chong You Guru Guruganesh Joshua Ainslie Santiago Ontañón Manzil Zaheer Sumit Sanghai Yiming Yang 0002 Sanjiv Kumar Srinadh Bhojanapalli Functional Interpolation for Relative Positions improves Long Context Transformers. 2024 ICLR https://openreview.net/forum?id=rR03qFesqk conf/iclr/2024 db/conf/iclr/iclr2024.html#LiYGAOZSYKB24 Yury Zemlyanskiy Michiel de Jong Luke Vilnis Santiago Ontañón William W. Cohen Sumit Sanghai Joshua Ainslie MEMORY-VQ: Compression for Tractable Internet-Scale Memory. 737-744 2024 NAACL (Short Papers) https://doi.org/10.18653/v1/2024.naacl-short.64 conf/naacl/2024-2 db/conf/naacl/naacl2024-2.html#ZemlyanskiyJVOCSA24 streams/conf/naacl

Yi Liang You Wu 0001 Honglei Zhuang Li Chen Jiaming Shen Yiling Jia Zhen Qin 0001 Sumit Sanghai Xuanhui Wang Carl Yang 0001 Michael Bendersky Integrating Planning into Single-Turn Long-Form Text Generation. 2024 abs/2410.06203 CoRR https://doi.org/10.48550/arXiv.2410.06203 db/journals/corr/corr2410.html#abs-2410-06203 streams/journals/corr

Michiel de Jong Yury Zemlyanskiy Joshua Ainslie Nicholas FitzGerald Sumit Sanghai Fei Sha William W. Cohen FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference. 11534-11547 2023 ACL (Findings) https://doi.org/10.18653/v1/2023.findings-acl.732 https://aclanthology.org/2023.findings-acl.732 conf/acl/2023f db/conf/acl/acl2023f.html#JongZAFSSC23 Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. 4895-4901 2023 EMNLP https://doi.org/10.18653/v1/2023.emnlp-main.298 https://aclanthology.org/2023.emnlp-main.298 conf/emnlp/2023 db/conf/emnlp/emnlp2023.html#AinslieLJZLS23 Joshua Ainslie Tao Lei 0001 Michiel de Jong Santiago Ontañón Siddhartha Brahma Yury Zemlyanskiy David C. Uthus Mandy Guo James Lee-Thorp Yi Tay Yun-Hsuan Sung Sumit Sanghai CoLT5: Faster Long-Range Transformers with Conditional Computation. 5085-5100 2023 EMNLP https://doi.org/10.18653/v1/2023.emnlp-main.309 https://aclanthology.org/2023.emnlp-main.309 conf/emnlp/2023 db/conf/emnlp/emnlp2023.html#AinslieLJOBZUGL23 Michiel de Jong Yury Zemlyanskiy Nicholas FitzGerald Joshua Ainslie Sumit Sanghai Fei Sha William W. Cohen Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute. 7329-7342 2023 ICML https://proceedings.mlr.press/v202/de-jong23a.html conf/icml/2023 db/conf/icml/icml2023.html#JongZFASSC23 Luke Vilnis Yury Zemlyanskiy Patrick Murray Alexandre Tachard Passos Sumit Sanghai Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models. 35120-35136 2023 ICML https://proceedings.mlr.press/v202/vilnis23a.html conf/icml/2023 db/conf/icml/icml2023.html#VilnisZMPS23

Michiel de Jong Yury Zemlyanskiy Nicholas FitzGerald Joshua Ainslie Sumit Sanghai Fei Sha William W. Cohen Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute. 2023 abs/2301.10448 CoRR https://doi.org/10.48550/arXiv.2301.10448 db/journals/corr/corr2301.html#abs-2301-10448

Joshua Ainslie Tao Lei 0001 Michiel de Jong Santiago Ontañón Siddhartha Brahma Yury Zemlyanskiy David C. Uthus Mandy Guo James Lee-Thorp Yi Tay Yun-Hsuan Sung Sumit Sanghai CoLT5: Faster Long-Range Transformers with Conditional Computation. 2023 abs/2303.09752 CoRR https://doi.org/10.48550/arXiv.2303.09752 db/journals/corr/corr2303.html#abs-2303-09752

Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. 2023 abs/2305.13245 CoRR https://doi.org/10.48550/arXiv.2305.13245 db/journals/corr/corr2305.html#abs-2305-13245

Michiel de Jong Yury Zemlyanskiy Nicholas FitzGerald Sumit Sanghai William W. Cohen Joshua Ainslie GLIMMER: generalized late-interaction memory reranker. 2023 abs/2306.10231 CoRR https://doi.org/10.48550/arXiv.2306.10231 db/journals/corr/corr2306.html#abs-2306-10231

Yury Zemlyanskiy Michiel de Jong Luke Vilnis Santiago Ontañón William W. Cohen Sumit Sanghai Joshua Ainslie MEMORY-VQ: Compression for Tractable Internet-Scale Memory. 2023 abs/2308.14903 CoRR https://doi.org/10.48550/arXiv.2308.14903 db/journals/corr/corr2308.html#abs-2308-14903

Shanda Li Chong You Guru Guruganesh Joshua Ainslie Santiago Ontañón Manzil Zaheer Sumit Sanghai Yiming Yang 0002 Sanjiv Kumar Srinadh Bhojanapalli Functional Interpolation for Relative Positions Improves Long Context Transformers. 2023 abs/2310.04418 CoRR https://doi.org/10.48550/arXiv.2310.04418 db/journals/corr/corr2310.html#abs-2310-04418

Yury Zemlyanskiy Michiel de Jong Joshua Ainslie Panupong Pasupat Peter Shaw 0004 Linlu Qiu Sumit Sanghai Fei Sha Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing. 4946-4951 2022 COLING https://aclanthology.org/2022.coling-1.438 conf/coling/2022 db/conf/coling/coling2022.html#ZemlyanskiyJAPS22 Li Yang Qifan Wang 0001 Zac Yu Anand Kulkarni Sumit Sanghai Bin Shu Jon Elsas Bhargav Kanagal MAVE: A Product Dataset for Multi-source Attribute Value Extraction. 1256-1265 2022 WSDM https://doi.org/10.1145/3488560.3498377 conf/wsdm/2022 db/conf/wsdm/wsdm2022.html#YangWYKSSEK22

Yury Zemlyanskiy Michiel de Jong Joshua Ainslie Panupong Pasupat Peter Shaw 0004 Linlu Qiu Sumit Sanghai Fei Sha Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing. 2022 abs/2209.14899 CoRR https://doi.org/10.48550/arXiv.2209.14899 db/journals/corr/corr2209.html#abs-2209-14899

Luke Vilnis Yury Zemlyanskiy Patrick Murray Alexandre Passos Sumit Sanghai Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models. 2022 abs/2210.15458 CoRR https://doi.org/10.48550/arXiv.2210.15458 db/journals/corr/corr2210.html#abs-2210-15458

Michiel de Jong Yury Zemlyanskiy Joshua Ainslie Nicholas FitzGerald Sumit Sanghai Fei Sha William W. Cohen FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference. 2022 abs/2212.08153 CoRR https://doi.org/10.48550/arXiv.2212.08153 db/journals/corr/corr2212.html#abs-2212-08153

Luke Vilnis Zachary Fisher Bhargav Kanagal Patrick Murray Sumit Sanghai ImPaKT: A Dataset for Open-Schema Knowledge Base Construction. 2022 abs/2212.10770 CoRR https://doi.org/10.48550/arXiv.2212.10770 db/journals/corr/corr2212.html#abs-2212-10770

Gurmeet Manku James Lee-Thorp Bhargav Kanagal Joshua Ainslie Jingchen Feng Zach Pearson Ebenezer Anjorin Sudeep Gandhe Ilya Eckstein Jim Rosswog Sumit Sanghai Michael Pohl Larry Adams D. Sivakumar 0001 ShopTalk: A System for Conversational Faceted Search. 2021 abs/2109.00702 CoRR https://arxiv.org/abs/2109.00702 db/journals/corr/corr2109.html#abs-2109-00702

Li Yang Qifan Wang 0001 Zac Yu Anand Kulkarni Sumit Sanghai Bin Shu Jon Elsas Bhargav Kanagal MAVE: A Product Dataset for Multi-source Attribute Value Extraction. 2021 abs/2112.08663 CoRR https://arxiv.org/abs/2112.08663 db/journals/corr/corr2112.html#abs-2112-08663

Joshua Ainslie Santiago Ontañón Chris Alberti Vaclav Cvicek Zachary Fisher Philip Pham Anirudh Ravula Sumit Sanghai Qifan Wang 0001 Li Yang ETC: Encoding Long and Structured Inputs in Transformers. 268-284 2020 EMNLP (1) https://doi.org/10.18653/v1/2020.emnlp-main.19 https://aclanthology.org/2020.emnlp-main.19/ https://orkg.org/orkg/resource/R216277 conf/emnlp/2020-1 db/conf/emnlp/emnlp2020-1.html#AinslieOACFPRSW20 Qifan Wang 0001 Li Yang Bhargav Kanagal Sumit Sanghai D. Sivakumar 0001 Bin Shu Zac Yu Jon Elsas Learning to Extract Attribute Value from Product via Question Answering: A Multi-task Approach. 47-55 2020 KDD https://doi.org/10.1145/3394486.3403047 conf/kdd/2020 db/conf/kdd/kdd2020.html#WangYKSSSYE20

Joshua Ainslie Santiago Ontañón Chris Alberti Philip Pham Anirudh Ravula Sumit Sanghai ETC: Encoding Long and Structured Data in Transformers. 2020 abs/2004.08483 CoRR https://arxiv.org/abs/2004.08483 db/journals/corr/corr2004.html#abs-2004-08483

Larry Adams Joshua Ainslie Christopher AlbertiChris Alberti Ebenezer Anjorin Michael Bendersky Srinadh Bhojanapalli Siddhartha Brahma Li Chen William W. Cohen Vaclav Cvicek Ilya Eckstein Jon Elsas Jingchen Feng Zachary Fisher Nicholas FitzGerald Sudeep Gandhe Mandy Guo Guru Guruganesh Yiling Jia Michiel de Jong Bhargav Kanagal Anand Kulkarni Sanjiv Kumar Federico Lebrón James Lee-Thorp Tao Lei 0001 Shanda Li Yi Liang Gurmeet Manku Patrick Murray Santiago Ontañón Alexandre Tachard PassosAlexandre Passos Panupong Pasupat Zach Pearson Philip Pham Michael Pohl Zhen Qin 0001 Linlu Qiu Anirudh Ravula Jim Rosswog Fei Sha Peter Shaw 0004 Jiaming Shen Bin Shu D. Sivakumar 0001 Yun-Hsuan Sung Yi Tay David C. Uthus Luke Vilnis Qifan Wang 0001 Xuanhui Wang You Wu 0001 Carl Yang 0001 Li Yang Yiming Yang 0002 Chong You Zac Yu Manzil Zaheer Yury Zemlyanskiy Honglei Zhuang