Fun Maximizing Search, (Non) Instance Optimality, and Video Games for Parrots

Computerized Adaptive Testing (CAT) measures an examinee's ability while adapting to their level. Both too many questions and too many hard questions can make a test frustrating. Are there some CAT algorithms which can be proven to be theoretically better than others, and in which framework? We show that slightly extending the traditional framework yields a partial order on CAT algorithms. For uni-dimensional knowledge domains, we analyze the theoretical performance of some old and new algorithms, and we prove that none of the algorithms presented are instance optimal, conjecturing that no instance optimal can exist for the CAT problem.