Adaptive Advice, MyMedia and TasteWeights
As recommender systems are increasingly deployed in the real world, they are not merely tested offline for accuracy, precision and coverage, but also “online” with real test users, to ensure a good user experience. I have developed a theoretical framework for the user-centric evaluation of recommender systems, and tested this framework in a wide variety of research projects. These projects are outlined below.
Within the MyMedia project, I was responsible for the user-centric evaluation of media recommender systems in several large-scale field trials. I provided the partners with experimental designs, user experience questionnaires, and requirements for click-stream logging.
With the results, I validated an evaluation framework for recommender systems (see figure below) that links objective aspects of a recommender system to objective user behavior through a series of subjective constructs. Specifically, changes in the recommender system (objective system aspects) are perceived by the users (subjective system aspects), and influence their experience and interaction with the system. The interaction with the recommender system is furthermore influenced by personal and situational characteristics.
Decision-making research has demonstrated in numerous experiments that humans differ in the way they make choices. Personal decision-making strategies tend to vary systematically: there are for example notable differences between experts and novices, and between people with different choice goals. Although recommender systems adapt their recommendations to user preferences, they typically do not adapt their interface to support the different decision-making strategies.
A notable example of a useful interface adaptation would be the Preference Elicitation (PE) method. Preference elicitation is the process of discovering what a user likes and dislikes. In our experiments with a recommender for energy-saving measures we compared several preference elicitation methods, and found notable differences between experts and novices.
When given a personalized set of excellent recommendations to choose from, users of recommender systems may experience choice overload: it may be too difficult for them to choose an item among the list of suggestions. Under the lead of Martijn Willemsen, we have investigated the phenomenon of choice overload in recommender systems, as well as potential solutions to the choice overload problem. Specifically, our research shows that a small set of diversified recommendations can decrease choice difficulty without sacrificing overall choice satisfaction. In this project I function as a main co-author and statistical advisor.
Collaborative recommender systems typically use a "nearest neighbors" approach to provide recommendations based on users similar to the current user. Social recommender limit the set of other users to your friends, thereby leveraging your personal connections. This gives the user an excellent opportunity to inspect and control how their friends' preferences are considered in the recommendation process. I tested the TasteWeights system (developed at UCSB), which provides a user interface geared towards such inspectability and control.
I collaborated with the researchers at UCSB to test the working premise of the TasteWeights system. We conducted an online user experiment to test the impact that the interface mechanisms for control (setting item- and friend-weights) and inspectability (inspecting the recommendation graph) have on the user experience. We found that these mechanisms indeed each have a positive effect on the understandability of the system, users' perceived control, their perceived quality of the recommendations, and their overall satisfaction with the system.
Additional publications can be found under menu item Adaptive Advice.
Knijnenburg, B.P., Bostandjiev, S., O'Donovan, J., Kobsa, A.: Inspectability and Control in Social Recommender Systems. Full paper at the ACM Conference on Recommender Systems (RecSys) 2012, DOI: 10.1145/2365952.2365966, download here.
Acceptance rate: 20%.
Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the User Experience of Recommender Systems. User Modeling and User-Adapted Interaction (UMUAI) 2012, DOI: 10.1007/s11257-011-9118-4, download here.
UMUAI is ranked #6 among 26 HCI journals (Microsoft Academic Search) and #20 among 445 Computer Science journals (ISI/Thompson). This paper was the 3rd-most downloaded UMUAI paper of 2012, with over 1200 downloads.
Willemsen, M.C., Graus, M.P., Knijnenburg, B.P., Bollen, D.: Not just more of the same: Preventing Choice Overload in Recommender Systems by Offering Small Diversified Sets. Working paper, available by request.
Knijnenburg, B.P., Reijmer, N.J.M., Willemsen, M.C.: Each to His Own: How Different Users Call for Different Interaction Methods in Recommender Systems. Full paper at the ACM Conference on Recommender Systems (RecSys) 2011. DOI: 10.1145/2043932.2043960, download here.
Acceptance rate: 20%
Knijnenburg, B.P., Willemsen, M.C., Kobsa, A.: A Pragmatic Procedure to Support the User-Centric Evaluation of Recommender Systems. Short paper at the ACM Conference on Recommender Systems (RecSys) 2011, DOI: 10.1145/2043932.2043993, download here.
Acceptance rate: 27%.
Willemsen, M.C., Knijnenburg, B.P., Graus, M.P., Velter-Bremmers, L.C.M., Fu, K.: Using latent features diversification to reduce choice difficulty in recommendation lists. Full paper at the RecSys 2011 workshop on Human Decision Making in Recommender Systems (Decisions@RecSys'11), download here.
Bollen, D.G.F.M., Knijnenburg, B.P., Willemsen, M.C., Graus, M.P.: Understanding Choice Overload in Recommender Systems. Full paper at the ACM Conference on Recommender Systems (RecSys) 2010, DOI: 10.1145/1864708.1864724, download here.
Acceptance rate: 19%.
Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Getting Recommendations and Providing Feedback: The User-Experience of a Recommender System. Full paper at the 11th International Conference on Electronic Commerce and Web Technologies (EC-Web) 2010, DOI: 10.1007/978-3-642-15208-5_19, download here.
Knijnenburg, B.P., Willemsen, M.C.: The Effect of Preference Elicitation Methods on the User Experience of a Recommender System. Short paper at the international conference extended on Human factors in computing systems (CHI) 2010. DOI: 10.1145/1753846.1754001, download here.
Knijnenburg, B.P., Willemsen, M.C.: Understanding the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. Short paper at the ACM Conference on Recommender Systems (RecSys) 2009. DOI: 10.1145/1753846.1754001, download here.
Winner of the short paper award (among 47 papers).
Knijnenburg, B.P., Meesters, L.M.J., Marrow, P., Bouwhuis, D.G.: User-Centric Evaluation Framework for Multimedia Recommender Systems. Short workshop paper at the first international conference on user centric media (UCMedia) 2009, DOI: 10.1007/978-3-642-12630-7_47, download here.
Knijnenburg, B.P.: Adaptive Advice: Adapting a recommender system for energy-saving behaviors to personal differences in decision-making. Graduation thesis. download here.
Graded 9.5/10, nominated for the Dutch SIGCHI chapter Thesis Award (Gerrit van der Veer prijs), nominated for the Eindhoven University of Technology Academic Awards, winner of the Department of Innovation Sciences Best Thesis Award.
Knijnenburg, B.P.: The user-centric evaluation of personalized systems. Poster presented at the Research Lab at Google I/O, June 24, 2014.
Willemsen, M.C., Knijnenburg, B.P., Bollen, D.G.F.M.: Recommending Less is More - Understanding Choice Overload Using a Movie Recommender System. Accepted poster at the 31st annual meeting of the Society for Judgment and Decision Making (JDM), Nov. 19-22, 2010.
Proceedings chair of the ACM conference on Intelligent User Interfaces (IUI) 2016.
Proceedings chair of the ACM conference on Recommender Systems (RecSys) 2014.
Program committee member of the International World Wide Web Conference (WWW) 2016, Behavioral Analysis and Personalization track.
Editorial Review Board member of the special issue on Human Interaction With Artificial Advice Givers for the ACM Transactions on Interactive Intelligent Systems (TiiS).
Program committee member of the FLAIRS RecSys special track in 2015 and 2016.
Program committee member of the ACM conference on Recommender Systems (RecSys) 2015.
Program committee member of the Conference on User Modeling, Adaptation and Personalization (UMAP) 2014 and 2015.
Program committee member of the Information Interaction in Context conference (IIiX) 2014.
Program committee member of the International AAAI Conference on Weblogs and Social Media (ICWSM) 2013.
Program committee member of the 14th International Conference on Electronic Commerce and Web Technologies (EC-Web) 2013.
Presenter of the RecSys2012 tutorial on Conducting User Experiments in Recommender Systems. summary, handout
Keynote speaker of the RecSys2011 workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces 2 (UCERSTI::2).
Organizer of the RecSys2010 workshop on "User-Centric Evaluation of Recommender Systems and Their Interfaces" (UCERSTI).