A Study of LLM-Powered Student Query Support
DOI:
https://doi.org/10.47756/aihc.y9i1.141Keywords:
Learning, Large Language Models, User StudyAbstract
In this paper, we explore the use of Large Language Models (LLMs) to help students improve their information-seeking skills while encouraging the use of references to aid library literacy efforts. This study aims to expand the reach of library support by introducing an approach that leverages the capabilities of LLMs and well-structured prompts. Our approach begins with surveying the current changes students have faced in the last two years concerning their study habits and how they search for information. We subsequently propose a multi-step system prompt, referred as prompting architecture, for foundational and instructed LLMs. The proposed prompt architecture powers a web application named LibRef. We explore the adaptability of the prompting architecture to different information retrieval needs by refining search prompts and providing academic references. A field experiment is conducted using LibRef in academic settings. Our results suggest that the use of LibRef enhances students’ academic information-seeking experience. Our research underscores the potential of prompting architectures in procedural refinement of academic queries from students. We believe our findings can provide valuable insights on the current capabilities of LLMs for instructing students to provide more targeted prompts as well as incentivize the use of references.
Downloads
References
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623. DOI: https://doi.org/10.1145/3442188.3445922
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
Kun Huang, Xiaoyu Wang, Shichao Luo, Qiuping Su, and Lei Li. 2022. What difficulties did the college students encountered in information seeking during the COVID-19 pandemic? Data and Information Management 6, 2 (2022), 100005. DOI: https://doi.org/10.1016/j.dim.2022.100005
James R Lewis. 1995. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 7, 1 (1995), 57–78. DOI: https://doi.org/10.1080/10447319509526110
Isaiah Michael Omame and Juliet C Alex-Nmecha. 2020. Artificial intelligence in libraries. In Managing and adapting library information services for future users. IGI Global, 120–144. DOI: https://doi.org/10.4018/978-1-7998-1116-9.ch008
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7. DOI: https://doi.org/10.1145/3411763.3451760
Sei-Ching Joanna Sin and Kyung-Sun Kim. 2008. Use and non-use of public libraries in the information age: A logistic regression analysis of household characteristics and library services variables. Library & information science research 30, 3 (2008), 207–215. DOI: https://doi.org/10.1016/j.lisr.2007.11.008
JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. DOI: https://doi.org/10.1145/3544548.3581388
Downloads
Published
How to Cite
Issue
Section
License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
AMexIHC make every effort to ensure the accuracy and rigour of all the information (the "Content") contained in out publications. However, AMexIHC and our representatives make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by AMexIHC. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information.