Auditing responsibility in complex multi-route recommender systems accurately
FunnelAudit: Responsibility Auditing in Multi-Route Recommender Systems
Information Retrieval
Summary
Recommender systems often combine many steps and routes to decide what to show users, making it hard to figure out which part is responsible for a given recommendation. The authors present FunnelAudit, a method to carefully check responsibility by simulating different control setups and identifying the smallest factors that truly matter. They tested this on real data and found that simple methods miss most responsibility cases, showing that detailed replay and clear rules are important to fairly assign accountability. Their approach also comes with proofs that can be independently verified, ensuring trust in these judgments.
recommender systemsmulti-route funnelsresponsibility auditingablation studiesranking algorithmsinteraction datasetsaccountabilitySASRec rankingexecutable frameworks
Authors
Jie Li, Dudu Luo, Jiayang Niu, Ke Deng, Yongli Ren
Abstract
Multi-route recommender systems combine retrieval, allocation, fusion, and ranking, making individual inclusions and exclusions difficult to audit. Route overlap can hide effects from one-at-a-time ablations, while freezing downstream stages produces counterfactuals inconsistent with serving behavior. We introduce FunnelAudit, an executable framework for incident-level responsibility auditing. An accountability contract specifies the disputed Top-K event, controls and owners, permitted reference actions, and replay semantics. FunnelAudit evaluates every permitted control configuration and applies graded actual responsibility to find the smallest outcome-preserving contingency that makes each control pivotal. Its certificate records the contingency and paired serving executions needed to verify the judgment. We instantiate the framework in two-stage, nine-route funnels using fixed union, weighted quota allocation, or weighted reciprocal-rank fusion, followed by SASRec ranking. Across 258,809 user-target incidents from three real interaction datasets, 4.24-16.24% admit a responsible control. Among responsible incident-control pairs, 92.55-99.64% require a nonempty contingency, so single-control ablation recovers only 0.36-7.45%. Policies differing in factual outcomes on only 0.31-2.39% of incidents yield 21.44-54.05% Jaccard distance between responsible-route sets on matched exclusions. Independent replay reproduces all 9,121,792 checked target-world outcomes; exhaustive search and a generic mixed-integer linear program agree with every sampled judgment. These findings demonstrate the importance of explicit serving semantics and checkable witnesses for recommender accountability.