FunnelAudit enables clear accountability in complex recommender systems
FunnelAudit: Responsibility Auditing in Multi-Route Recommender Systems
Information Retrieval
Summary
Recommender systems often use many different paths to suggest items, making it hard to tell who or what is responsible for what users see. The authors present FunnelAudit, a tool that carefully replays decisions in these systems to find which parts truly affect recommendations. This helps uncover hidden influences that simpler methods miss and provides a clear record to prove responsibility. Their tests on large real datasets show this method works reliably and reveals new insights about accountability.
What this means in practice
- •For recommender system engineers: Audit which components of multi-route recommenders are responsible for specific recommendations to improve transparency and debugging.
- •For advertising platform operators: Provide verifiable proofs of responsibility for recommendations influencing ad delivery to ensure accountability and compliance.$Commercial implications: Enables sale of accountability reports and compliance auditing tools to advertising platforms needing transparent recommendation auditing.
Authors
Jie Li, Dudu Luo, Jiayang Niu, Ke Deng, Yongli Ren
Abstract
Multi-route recommender systems combine retrieval, allocation, fusion, and ranking, making individual inclusions and exclusions difficult to audit. Route overlap can hide effects from one-at-a-time ablations, while freezing downstream stages produces counterfactuals inconsistent with serving behavior. We introduce FunnelAudit, an executable framework for incident-level responsibility auditing. An accountability contract specifies the disputed Top-K event, controls and owners, permitted reference actions, and replay semantics. FunnelAudit evaluates every permitted control configuration and applies graded actual responsibility to find the smallest outcome-preserving contingency that makes each control pivotal. Its certificate records the contingency and paired serving executions needed to verify the judgment. We instantiate the framework in two-stage, nine-route funnels using fixed union, weighted quota allocation, or weighted reciprocal-rank fusion, followed by SASRec ranking. Across 258,809 user-target incidents from three real interaction datasets, 4.24-16.24% admit a responsible control. Among responsible incident-control pairs, 92.55-99.64% require a nonempty contingency, so single-control ablation recovers only 0.36-7.45%. Policies differing in factual outcomes on only 0.31-2.39% of incidents yield 21.44-54.05% Jaccard distance between responsible-route sets on matched exclusions. Independent replay reproduces all 9,121,792 checked target-world outcomes; exhaustive search and a generic mixed-integer linear program agree with every sampled judgment. These findings demonstrate the importance of explicit serving semantics and checkable witnesses for recommender accountability.