Doc2LoRA enables language model representations for scientific paper ideas
Doc2LoRA Provides Decodable Representations of Scientific Ideas
Computation and LanguageDigital LibrariesInformation RetrievalMachine Learning
Summary
It can be hard to understand and explore scientific papers by just reading them. The authors created a way to turn each paper into a special language model adapter that can talk about the paper's ideas. By mixing these adapters, people can generate explanations for new combined ideas that don't exist yet. This method also helps find similar papers and describe scientific fields more accurately than some current methods.
What this means in practice
- •For academic search engines: Create paper representations that enable natural language Q&A and generation for improved search and exploration of scientific literature.
- •For ai content generation teams: Generate abstracts and descriptions for novel research ideas formed by combining existing scientific concepts.
Authors
Chand Sahil Mansuri, Joel Zachariah, Sadamori Kojaku
Abstract
Representing scientific papers as points in a space lets us search for similar papers and inquire about how fields relate to one another and drive innovation. Beyond search, the vector space of papers invites generation: mixing papers through simple vector operations creates new points, mirroring combinatorial novelty, the recombination of existing ideas into new ones. However, a mixed point often represents an idea no paper has yet realized, with no papers nearby to identify the idea. We propose representing each paper by a LoRA adapter generated by the Doc-to-LoRA hypernetwork. Every point in the space, including mixtures, thus represents a large language model (LLM) open to questions and instructions in natural language. On papers from the American Physical Society (APS), we instruct the LLM at the average of each subfield to name the field in a few words and obtain labels closer to the official names than the labels of five baselines, as judged by word overlap and a panel of five LLM judges. We also ask the LLMs at points between two APS papers to write an abstract and obtain descriptions shifting from one paper to the other in step with the mixing weight. While Doc-to-LoRA is trained for generation, a small invertible transform makes the embeddings competitive for search, on par with SPECTER2 and EmbeddingGemma and close to SBERT. Because the transform is invertible, every point in the transformed space still maps back to an LLM. The embeddings thus serve both search and generation, enabling researchers to question the idea at any point in the space as a starting point for generating new ideas.