Featured Publications

Please see my CV for a full list.

FARE: Diagnostics for Fair Ranking using Pairwise Error Metrics

Published in The Web Conference, 2019

In this work we propose to broaden the scope of fairness assessment, which heretofore has largely been limited to classification tasks, to include error-based fairness criteria for rankings. Our approach supports three criteria: Rank Equality, Rank Calibration, and Rank Parity, which cover a broad spectrum of fairness considerations from proportional group representation to error rate similarity. The underlying error metrics are formulated to be rank-appropriate, using pairwise discordance to measure prediction error in a model agnostic fashion. Based on this foundation, we then design a fairness auditing mechanism which captures group differences throughout the entire ranking, generating in-depth, nuanced diagnostics. We demonstrate the efficacy of our error metrics using real-world scenarios, exposing trade-offs among fairness criteria and providing guidance in the selection of fair-ranking algorithms.

Recommended citation: Caitlin Kuhlman, MaryAnn VanValkenburg, Elke Rundensteiner. FARE: Diagnostics for Fair Ranking using Pairwise Error Metrics. The Web Conference (WWW) Web and Society track 2019. http://web.cs.wpi.edu/~cakuhlman/publications/fare.pdf

Evaluating Preference Collection Methods for Interactive Ranking Analytics

Published in ACM Conference on Human Factors in Computing Systems, 2019

Rankings distill a large number of factors into simple comparative models to facilitate complex decision making. Yet key questions remain in the design of mixed-initiative systems for ranking, in particular how best to collect users' preferences to produce high-quality rankings that users trust and employ in the real world. To address this challenge we evaluate the relative merits of three preference collection methods for ranking in a crowdsourced study. We find that with a categorical binning technique, users interact with a large amount of data quickly, organizing information using broad strokes. Alternative interaction modes using pairwise comparisons or sub-lists result in smaller, targeted input from users. We consider how well each interaction mode addresses design goals for interactive ranking systems. Our study indicates that the categorical approach provides the best value-added benefit to users, requiring minimal effort to create sufficient training data for the underlying ranking algorithm.

Recommended citation: Caitlin Kuhlman, Paul-Henry Schoenhagen, MaryAnn VanValkenburg, Diana Doherty, Malika Nurbekova, Goutham Deva, Zarni Phyo, Elke Rundensteiner, Lane Harrison. Evaluating Preference Collection Methods for Interactive Ranking Analytics ACM Conference on Human Factors in Computing Systems (CHI) 2019. https://dl.acm.org/authorize?N673729