HCI Portfolio / Facade

Methodologies for evaluating task-based systems are well established, but more often today we work with complex experience driven systems, games, online communities and other applications have as their goal not the accomplishment of a task but the creation of a cohesive and pleasurable experience. Evaluating a successful experience, however, is complicated.

The Facade evaluation uses a complex AI based interactive drama (Facade) as the case study for exploring methods for evaluating experiences. The approach taken in this experiment uses a series of two retro-active protocols. Participants used the system while the screen was tape recorded. Afterwards, they watched the playback of themselves playing and described their interactions, motivations, and reactions. A coding scheme based on testing for aspects such as Agency, Interest in the Story, and a sense of Immersion, was created and each session was analyzed and coded using the scheme.

Several interesting patterns emerged during the analysis. Two main strategies emerged: those players spent the majority of their time exploring and tuning their gaming strategy and others were simply immersed in the experience. We were also able to determine that despite mistakes in recognition caused by the AI, the system successful maintained the experience. In the figure below, you can see that typically, when a particular strategy results in a mistake by the AI (CharRespNeg) it is followed by some interest in the background story being revealed and a subsequent change in strategy. This indicates that the while the system may make mistakes, it successfully entertains the user by revealing interesting backstory, that can often be used in a new strategy.

You can read the full report on this evaluation described in the paper published at CHI 2005:

A Preliminary Evaluation of the Interactive Drama Facade [ Word ].