Papers for
university it support teams
Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.
Graduate workers manage risks in ongoing use of large language models
"If You're Not Doing It, Somebody Else Is": Active Negotiation and the Invisible Labor of Sustained LLM Use
Abstract: Large language models (LLMs) have become fixtures of academic work even as their users describe them as degrading their writing, thinking, and skills. Dominant adoption frameworks read continued use as evidence of satisfaction, and cannot explain continued use of a distrusted tool. We interviewed 36 graduate student workers, balanced between English-as-a-foreign-language (EFL) and non-EFL speakers, and introduce the Active Negotiation framework: a model of sustained LLM use as a recurring cycle of risk, mitigation, and justification. A failure surfaces a risk, mitigation labor addresses it, and a justification renders the residual risk tolerable until the next failure reopens the cycle. The cycle runs across three dimensions: practical, auditing output; internal, auditing one's own cognition and identity; and social, managing how peers and institutions perceive use. EFL participants invoke linguistic parity as a further justification. We reframe continued adoption as compliance sustained by invisible labor.
Retrieval systems perform differently on synthetic versus real user questions
Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation
Abstract: RAG systems are routinely evaluated using synthetic question sets generated from the target document corpus. While this practice provides a useful check on overall retrieval capability, relying exclusively on synthetic benchmarks can mislead under distribution shift and overstate deployment readiness. Synthetic generation spreads questions evenly across the corpus, formulating long, detailed queries; real users put most of their traffic on a few administrative and procedural topics in short queries, while also asking about matters the generator never covers at all. We demonstrate this gap on a university faculty information system, comparing 1,851 synthetic questions generated via Gemini Notebook against 322 authentic queries collected via a student survey. The synthetic and authentic query sets differ significantly: authentic queries average 6.8 words versus 15.7 for the synthetic ones, and draw from only 53 unique sources compared to 165. Consequently, configurations that appear highly effective on synthetic benchmarks experience a substantial performance drop on authentic queries. Importantly, optimizing on synthetic queries selected a higher-latency hybrid retriever. In our setting the sparse retrieval component benefited long synthetic questions but not short authentic ones, costing up to $8\times$ the latency of the fastest configuration we tested. We propose treating synthetic and authentic query sets as complementary extremes of the query-quality spectrum: synthetic data verifies maximum retrieval capacity under idealized conditions, while authentic queries test system robustness to the imprecise, underspecified inputs of real users.