Summary
When software teams make decisions without much past data, they can use expert-based Bayesian networks to help. But creating these networks needs many precise probability estimates, which is hard. The authors compared two semi-automatic methods, WSA and RNM, in a real software company for tasks like choosing features and design options. They found that while both methods ranked choices similarly, they differed in how certain or uncertain their probability estimates were. This means picking a method depends on details like how experts think about probabilities and how the results will be used.
What this means in practice
- •For software project managers: Improve decision support tools by selecting Bayesian network methods that better reflect expert uncertainty for feature and design choices.
- •For risk analysts in finance: Adapt semi-automatic Bayesian quantification methods like WSA and RNM to compare their impact on representing uncertainty in financial decision models.
Tested on one dataset.
Authors
Mirko Perkusich, João Nunes, Emilia Mendes, Emanuel Dantas, Ademar Sousa, Danyllo Albuquerque, Kyller C. Gorgônio, Angelo Perkusich
Abstract
Expert-driven Bayesian networks can support recurring software decisions when historical data are limited, but quantifying their conditional probability tables (CPTs) requires many probability judgments. Semi-automatic methods reduce direct elicitation, yet practitioners have little comparative evidence from operational decision processes. We report an embedded case study comparing the Weighted Sum Algorithm (WSA) and Ranked Nodes Method (RNM) as complete elicitation-and-quantification pipelines in two contexts within a single software research and development (R&D) organization: feature selection for Internet of Things projects and user interface design selection. Within each context, the pipelines shared the graph, root priors, and decision evidence. We evaluated them through 15 expert-defined model-walkthrough scenarios and retrospective reconstructions of alternatives recorded in decision meetings. WSA matched 7/7 and 6/8 walkthrough expectations, whereas RNM matched 4/7 and 4/8. Both pipelines placed the selected alternatives near the top of the retrospective rankings. Their complete distributions nevertheless differed: the median total variation distance was 0.304 across four distinct feature patterns and 0.340 across seven distinct design patterns, rising to 0.890 for one design pattern. These results concern two ordinal value-estimation models in one organization. They show that similar shortlist behavior does not imply equivalent representations of uncertainty. For comparable software decision-support models, method selection and validation should consider node semantics, the judgments experts can provide, calibration requirements, complete output distributions, and the intended downstream use of the probabilities.