Llm-assisted Literature Reviews Address 3 Key Gaps, Enabling Trusted Collaboration and Reducing Verification Burden

Large language models are rapidly becoming integral to academic research, yet a comprehensive understanding of how these tools impact the crucial process of literature review remains surprisingly limited. Brenda Nogueira from University of Notre Dame, alongside Werner Geyer and Andrew Anderson from IBM Research, and their colleagues, investigate how researchers currently employ these models and identify significant challenges. Their user study reveals three key issues, a lack of trust in generated outputs, a substantial burden of verifying information, and the need to juggle multiple different tools. This work addresses these pain points by proposing six design goals and a framework that prioritises verifiable results, improved visualisation of related papers, and alignment with human feedback, ultimately fostering a more trustworthy and collaborative relationship between researchers and artificial intelligence. Researchers’ LLM Usage and Perceptions This research investigates how researchers currently use Large Language Models (LLMs) like ChatGPT, their perceptions of these tools, and the challenges and opportunities they present for academic research. The study reveals widespread adoption of LLMs across many disciplines, with around 54% of researchers already experimenting with them. Researchers commonly use LLMs for literature reviews, summarizing papers and identifying relevant sources, as well as for writing assistance, including proofreading, editing, and generating drafts. LLMs also support idea generation and brainstorming, and, to a lesser extent, basic data analysis. Researchers generally view LLMs as potentially valuable tools, but express caution regarding accuracy, plagiarism, and the need for critical evaluation of generated content. A key benefit cited is the potential for time savings on tedious tasks, allowing researchers to focus on higher-level thinking. However, concerns exist regarding the possibility of LLMs generating factually incorrect information, requiring careful verification of all outputs. The study highlights challenges related to accuracy, plagiarism, potential bias in generated content, and a lack of transparency in how LLMs arrive at their conclusions. Future work should focus on developing guidelines for responsible use, improving the reliability of LLM-generated content, promoting critical thinking, and further investigating the long-term impact of LLMs on the research process. Overall, the research paints a picture of LLMs as a rapidly evolving technology with the potential to transform academic research, but one that requires careful consideration and responsible implementation. LLM Integration, Trust, and Researcher Practices This study pioneers a detailed examination of how researchers integrate large language models (LLMs) into the literature review process, uncovering deeper challenges in maintaining trust and accuracy. The research team conducted a user study with eight participants from diverse disciplines, including Computer Science, Biology, English, and Political Science, and compensated them for their participation. Each participant engaged in a semi-structured, virtual interview lasting approximately 40 minutes, allowing researchers to explore their workflows and experiences with LLMs. Researchers employed carefully constructed prompts to guide the conversation and ensure comprehensive data collection. The study prioritized capturing nuanced insights into how researchers actually utilize these tools, rather than relying on hypothetical scenarios. Data analysis involved a rigorous thematic analysis to identify recurring patterns and key themes within the transcribed interview data. This meticulous approach enabled the team to move beyond surface-level observations and uncover deeper, process-level gaps that shape how trust, accuracy, and conceptual structure are maintained across LLM-assisted review tasks. The resulting findings highlight areas where current practices fall short and where future agent design could substantially improve reliability, reasoning transparency, and reproducibility. Researchers Repurpose LLMs as Knowledge Organizers This research details a comprehensive user study examining how researchers currently integrate large language models (LLMs) into their workflows, revealing critical gaps and opportunities for improvement. The study, conducted with eight participants across diverse disciplines including computer science, biology, and political science, involved approximately 40-minute virtual interviews. Participants received compensation for their time. The findings demonstrate that researchers are repurposing LLMs not simply as text generators, but as “meta-organizers” that assist in structuring scholarly knowledge, particularly during the early stages of research. Several participants described using LLMs daily as a “toolkit” for brainstorming, drafting, and summarizing, with ongoing chat sessions serving as evolving workspaces. Others adopted a more selective approach, engaging LLMs for specific writing challenges, such as clarifying ideas or improving readability. This varied frequency of use reflects differences in disciplinary practices and levels of trust in LLMs for complex tasks. Researchers consistently favored structured information from LLMs, such as frameworks, taxonomies, and standardized formats like tables and bullet points, over narrative summaries. Participants specifically requested systems that could extract hypotheses or results in these formats to facilitate comparison and synthesis. The study also revealed a diverse ecosystem of LLM usage, with participants frequently employing both general-purpose models like ChatGPT and Gemini, alongside domain-oriented platforms like SciSpace and NotebookLM. Researchers routinely paired LLMs with verification tools such as LitMaps, Scite. ai, and Google Scholar to mitigate issues like misattribution or hallucination, often verifying LLM outputs by manually searching for citations on Google Scholar.

This research highlights an unmet opportunity: integrating the fluency and synthesis capabilities of LLMs with the citation grounding and reliability of research-specific platforms. The findings underscore the need for future agent design to substantially improve reliability, reasoning transparency, and reproducibility in LLM-assisted research workflows.

Knowledge Graph Curation for Trustworthy Reviews This research investigates how academics currently use large language models (LLMs) when conducting literature reviews, identifying key challenges and proposing a new framework to improve the process. Through a user study, the team found that researchers often lack trust in LLM outputs, face a significant burden in verifying information, and require multiple tools to complete their work. These findings motivated the development of a framework centred around improved visualization of related papers, continuous verification of generated content, and alignment with human feedback through clear explanations. The proposed framework reframes literature review as a collaborative process of knowledge graph construction and curation, rather than simple text generation. By grounding generated claims in verified source passages and offering interactive controls to expose the system’s reasoning, the team aims to position LLMs as collaborative evaluators that assist researchers in maintaining accuracy and scholarly rigor. The researchers acknowledge that their study focused on early adopters of LLMs and that broader sampling may reveal additional needs across different disciplines. They also highlight potential risks of reinforcing existing biases through relevance signals, such as venue or author reputation, and emphasize the need for transparent weighting, user controls, and regular audits to mitigate these issues. Future work includes prototyping and evaluating this framework with researchers, assessing its impact on faithfulness, edit stability, and user trust, and exploring extensions involving community ontologies and cross-domain generalization. 👉 More information 🗞 From Verification Burden to Trusted Collaboration: Design Goals for LLM-Assisted Literature Reviews 🧠 ArXiv: https://arxiv.org/abs/2512.11661 Tags: