How Much is Performance Worth to Users? A Quantitative Approach

Architects and systems designers artfully balance multiple competing design constraints during the design process but are unable to translate between system metrics and end user experience. This work presents three methodologies to fill in this gap. The first is an incentive-compatible methodology that determines a "ground truth" measurement of users' value of speed in terms of US dollars, and find that users would accept a performance losses of 10%, 20%, and 30% to their personal computer in exchange for \$2.27, \$4.07, and \$4.43 per day, respectively. However, while highly accurate the methodology is a painstaking process and does not scale with large numbers of participants. To allow for scalability, we introduce a second methodology -- a lab-based simulation experiment -- which finds that users would accept a permanent performance loss of 10%, 20%, and 30% to their personal computer in exchange for \$127, \$169, and \$823, respectively. Finally, to allow for even greater scalability, we introduce a third methodology -- a survey -- and observe that the lack of incentive compatibility and the lack of hands-on experience with throttled device performance skews the results significantly, thus demonstrating the need for lab-based or incentive compatible study designs. By quantifying the tradeoff between user satisfaction and performance, we enable architects and systems designers to make more nuanced tradeoffs between design requirements.