Abstract

As one of the important 21st-century skills, collaborative problem solving (CPS) has caught much attention in the assessment area. Two initiative approaches have been created: the human-to-human and human-to-agent modes. Between the two modes, the human-to-human interaction is much closer to the real-world situation and its process stream data can reveal more detailed information about the cognitive processes. In order to measure CPS ability effectively by this mode, how to extract indicators from the data and model it is crucial, while the existing methods all have some disadvantages. In the present study, we proposed a new paradigm for extracting indicators and modeling the dyad data in the human-to-human mode. Specifically, both individual and group indicators were extracted from the data stream, which served as evidence for demonstrating CPS skills. In addition, a within-item multidimensional Rasch model was used to fit the dyad data. To test the validity of the paradigm, we developed five tasks following the asymmetric mechanism and integrated them into an online testing platform. Four hundred thirty-four students from China participated in the assessment and the online platform recorded their crucial actions with time stamps. The new paradigm was used to deal with the generated process stream data. Results showed that the model fitted well. The item parameter estimates and fitting indexes were acceptable, and students were well differentiated. In general, the new paradigm of extracting indicators and modeling the dyad data is feasible for improving the human-to-human assessment of CPS. Finally, the limitations of the current study and further research directions are discussed.