software product managers

Abstract: Recent years have seen a surge in tools that support collaborative ideation through shared canvases. While these canvases offer flexible workspaces, how collaborators leverage them to iteratively develop ideas by externalizing thoughts, organizing artifacts, and coordinating tasks remains underexplored. To address this gap, we conducted an exploratory study with seven triads planning a travel itinerary using an infinite-canvas technology probe. Drawing on participant interviews and video analysis of their canvas interactions, we observed that collaborators contributed in parallel and arranged artifacts freely without a predefined structure. This spatial organization allowed collaborators to coordinate implicitly, made less vocal participants' contributions visible, helped distinguish private exploratory regions from shared working areas, and supported the emergence of distinct roles through repeated patterns of contribution. We discuss design implications for canvas-based ideation tools, including surfacing unnoticed contributions, supporting emergent roles without formalizing assignments, and keeping spatial cues legible for AI integration in collaborative canvases.

Abstract: Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US\$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at https://github.com/upwork/SWE-Clar.

Abstract: Between 2023 and 2026, AI changed how software is priced. Traditional SaaS charged per seat because serving one more user cost almost nothing. Model inference has non-zero marginal cost, and usage varies by orders of magnitude across customers, producing prepaid credits, metered usage, hybrid seat-plus-usage models, and outcome pricing that supplement, and in rare cases replace, seat-based pricing. These pricing models have not been systematically characterized. This paper presents a reproducible study of public pricing artifacts - pricing pages and documentation - across three cohorts: the full Forbes AI 50 (2026) as the AI-native core sample, cloud platforms selling AI services included by a documented rule, and a frozen purposive cohort of 9 distinctive designs. It contributes (1) a dataset coding each offering's pricing primitive and credit mechanics with dated primary sources, prepared for public release with the paper, (2) a taxonomy of the primitives in use and their distribution across cohorts, and (3) empirically grounded requirements for automated, configurable billing platforms, including ledger state for credits, attribution pipelines for outcome pricing, and configuration support for high pricing-change velocity. Subscription with usage quota is the dominant observable model among AI-native companies (55% of observable AI-native rows; 45-61% under sensitivity analysis of hybrid boundary cases). 23 of the 52 AI-native rows publish no pricing. Hybrid products are better understood as compositions of packaging anchor, allowance mechanism, and billable unit than as mutually exclusive categories. All artifacts are public; my employer is excluded from the sample.

Abstract: Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English, Estonian and Finnish about three software categories, four of twelve candidate brands per question, three repeats) were sent to gemini-3.1-pro-preview in three waves within about ten hours on 22 and 23 September 2026, and again on 13 October 2026, 20.3 days after the third wave, with identical request bodies. The plan and its decision rule were frozen, hashed and registered on OSF before the first call. The outcome is a lexical mention label per answer and brand, 648 per wave. The primary forecast, each brand-and-question cell's mention rate over the nine earlier answers, scored Brier 0.0281 and log loss 0.126, against Brier 0.0259 for the six-answer forecast of the 8-hour wave and a registered ceiling of 0.0559. It beat brand history (0.104), the overall rate (0.234) and a constant 0.5 (0.250) on both scores and in all nine question families (exact sign-flip p = 0.0039), so durability holds under the registered rule. A forecast frozen from the first wave alone scored 0.0599, 0.0587 and 0.0604 at horizons of 1.5 hours, 9.6 hours and 20.7 days. Of the 202 cells unanimous in the 8-hour wave, 194 were unchanged at D1 and none reversed. The latest wave alone was nearly as accurate as the pooled history (Brier 0.0292; pooled lower in 5 of 9 families, p = 0.28). Every answer reported the requested model version, and none issued a search query although search was enabled. A blind human check of 96 labels agreed with the rule on 94; both disagreements were adjudicated as rater omissions, so after adjudication the rule matched on 96 of 96. The test covers one model, one interval and twelve known brands, and estimates no decay rate.

Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Abstract: Evidence shows that humans and AI systems perform better together, by collaborating, than alone. This paper examines two key design dimensions of human-AI collaboration (autonomy and initiative) and explores the collaboration patterns that they generate. Documenting these patterns starts with identifying the underlying problems and solutions, followed by examining the internal tensions within the problems. The paper uses a paradox perspective to analyze those tensions. It describes a process for surfacing the tensions and mapping the underlying paradoxes. It also illustrates how the pattern descriptions can be derived from mapping these paradoxes. Finally, the paper documents four human-AI collaboration patterns: Instruction, Delegation, Assistance, and Co-creation.