Chatbot Language - crowdsource perceptions and reactions to dialogue systems to inform dialogue design decisions.

Conversational User Interfaces (CUI) are widely used, with about 1.8 billion users worldwide in 2020. For designing and building CUI, dialogue designers have to decide on how the CUI communicates with users and what dialogue strategies to pursue (e.g. reactive vs. proactive). Dialogue strategies can be evaluated in user tests by comparing user perceptions and reactions to different dialogue strategies. Simulating CUI and running them online, for example on crowdsourcing websites, is an attractive avenue to collecting user perceptions and reactions, as they can be gathered time- and cost-effectively. However, developing and deploying a CUI on a crowd sourcing platform can be laborious and requires technical proficiency from researchers. We present Chatbot Language (CBL) as a framework to quickly develop and deploy CUI on crowd sourcing platforms, without requiring a technical background. CBL is a library with specialized CUI functionality, which is based on the high-level language JavaScript. In addition, CBL provides scripts that use the API of the crowd sourcing platform Mechanical Turk (MT) in order to (a) create MT Human Intelligence Tasks (HITs) and (b) retrieve the results of those HITs. We used CBL to run experiments on MT and present a sample workflow as well as an example experiment. CBL is freely available and we discuss how CBL can be used now and may be further developed in the future.