A Review and Analysis of Evaluation Practices in VIS Domain Applications.

This paper presents a review and analysis of evaluation practices within the visualization and visual analytics (VIS) domain, with a focus on domain application work accepted at the IEEE VIS conference from 2018 to 2022. Through the analysis of 140 pertinent papers, we establish a detailed classification principle for evaluation practices, using the Who, When, What, and How indicators. This principle covers facets such as analysis methods, targets, scenarios, participant expertise, and stages of occurrence. By systematically categorizing the application domains presented in these works, we apply our established classification principle to discern and categorize the evaluation practices within them, identifying the prevailing characteristics and trends. The paper explores the variety of evaluation methods employed across different application domains and observes the distinctions in their usage. In conclusion, we provide insights and highlight concerns for conducting evaluations in upcoming domain application research. Our findings are intended to inform and guide subsequent studies in a similar context. All information about the reviewed papers, the coding results, and the detailed data analysis process can be accessed at https://github.com/narrating-complexity/vis_evaluation_analysis.