Development of RobotHub: Integration of External System to Group Conversation System for Older Adults

Abstract

Towards an aging society, there becomes much more critical to conduct social activities such as conversations for older adults to keep their cognitive function in their lives (Livingstone et al. 2017). Along with this social background, our research group has conducted a group conversation experiment for older adults to verify cognitive training with a randomized controlled trial (Otake-Matsuura et al., 2021). The group conversation method is called Coimagination method, which is designed to have a talk in a well-balanced manner with a specialized system (named Fonopane) and moderator robot Bono-05. However, because of the COVID-19 situation, those face-to-face experiments have become challenging to conduct. Therefore, we need to extend system to overcome the above physical limitations by interfacing the system to a teleconferencing system such as Zoom. Active listening function is desirable in such a system, since active listening is an essential factor in addressing smooth conversation for older adults and providing rich human-computer interaction (Inoueet al., 2020). In general, integration of different systems has some technical challenges, such as different communication protocols. In addition, the existing active listening system has developed focusing a one-to-one style with humans and AI; hence, there are technical problems for integrating such a one-to-one style system into a multi-party group conversation system. Method We cope problems above with a step-by-step approach. As a first step, we designed to integrate the existing active listening systems which are used in an autonomous Android ERICA (Inoue et al., 2016). We have developed a system component (RobotHub) that allows each participant in a group conversation to activate multiple systems and link them to existing systems. The system runs on a cloud server and client devices with four components; FrontEndAudioExtraction, CloudAudioGateway, FonopaneRemoteBridge, and SystemDataMapper, as shown in Figure 1.  Specifically, audio data is not obtained from the videoconferencing system directly, but from a client application, FrontEndAudioExtraction. The videoconferencing system interoperates with systems developed in the role of using only conventional functions, such as sharing output with screen and voice. FrontEndAudioExtraction is a system that converts the elderly's voices into power (namely, audio data) and then sends these audio data to the cloud server from their home. The system is designed to deploy on the client devices such as laptops or tablets for individual participants. It is designed easy enough to avoid cumbersome operation for people who are not good at devices, especially older adults. CloudAudioGateway is deployed on the cloud server to properly resupply the audio data sent from the client devices to another component inside the cloud server. FonopaneRemoteBridge converts the communication protocol with an existing system and an external system. SystemDataMapper is designed to assign correct relationships between microphones and participants. We also carefully designed to integrate systems to prevent with the sub-modules so that actions/events would not conflict with existing systems and external systems. In addition, we have designed and developed a system to migrate from one-to-one mapping style to one-to-multiple users.  We also deployed our system on the cloud to provide experiments to release the physical restriction.  Results and Discussion We have integrated the existing group conversation systems to external systems. We have preliminarily tested on laboratory members on Zoom (See Figure 2). RobotHub and other systems are now running on the cloud, so we can experiment anywhere as long as Zoom and FrontEndAudioExtraction are set up. An internal system test in our research team was conducted to confirm the feasibility.