Thanks to the health professionals, patients and families who agreed to have their consultations recorded

1. Introduction

1Embodied Conversational Agents (ECAs) are virtual agents that exhibit humanlike verbal and non-verbal behaviours. They are increasingly being applied in contexts where the main mode of interaction is a dialogue between two or more humans (T. Bickmore, Gruber, and Picard 2005). In eHealth, ECAs can provide vital support to patients by being more reachable and available in their time of need (Richards and Caldwell 2017). The agents can not only act as a source of information on health issues (Lisetti et al. 2012) and their prevention or cure (Yin, Ring, and Bickmore 2012), but also motivate the patients to adhere to treatments (Bickmore et al. 2010); (Richards and Caldwell 2017). For teenagers and children, a virtual agent can act as an educator, buddy and motivator (Looije, Neerincx, and Lange 2008), such as those designed for childhood obesity intervention (Kowatsch et al. 2017) or for promoting well-being and positive thoughts in young people undergoing cancer treatment (Greer et al. 2019).

2In order to make ECAs more effective, humanlike empathy expressed during conversation is a vital component (T. Bickmore, Gruber, and Picard 2005). Empathy is an essential part of building rapport and creating a bond, or a working alliance, between the patient and therapist to improve patient adherence and satisfaction (Bennett et al. 2011). Empathy has been defined as ‘an observer reacting emotionally because he perceives that another is experiencing or about to experience an emotion’ (Paiva et al. 2005), (p.4), or ‘the process whereby one person feels her/himself into the consciousness of another person’ (Wispé 1987). Empathy is expressed through a wide range of non-verbal behaviours, such as mirroring head nods, and verbal behaviours, such as the choice of words in social dialogues, the use of greetings and farewell rituals and the art of bringing continuity in the conversation (Laver 2011). In this paper, we focus on empathy expressed through verbal dialogue.

3Empathy has been researched as a vital component of emphasis devices in human computer interaction to emphasize particular qualities and context (Wright and McCarthy 2008) which leads to building dialogue sets that contain the empathic cues, such as confirmation and adherence encouragement, that act as emphasis monitors. Rather than being specialized in a single quality or domain, a good open-domain conversational agent should be able to seamlessly deliver the necessary features into one cohesive conversational flow. Facebook’s blender bot is an example of such a conversational agent that has been specially trained on empathic dialogue sets (Roller et al. 2020). As in the Blenderbot approach, there is a tendency in artificial intelligence approaches to use mimicry and to replicate the behaviour of humans based on corpora without analysis of whether that behaviour is appropriate. However, mimicking/replicating human responses to a tragic event, for example, will not always be the best response (Lundqvist 1995) and will depend on the context such as the relationship between the parties. The importance of empathy in human dialogue has led to interest in ECAs expressing empathy particularly to bring about behaviour change (McRorie et al. 2009), (Ochs, Pelachaud, and Mckeown 2017), (Ravichander and Black 2018).

4In this paper we evaluate the use of relational cues in recorded patient-doctor dialogues and the ECA’s known as Dr Evie (eVirtual agent for Incontinence and Enuresis) and as SAM (Sleep Adherence Mentor) designed to increase adherent behaviour in the domains of paediatric incontinence and sleep disorders, respectively. Both conditions have long specialist waitlists. Paediatric Incontinence affects up to 20% of school aged children (Malhotra et al. 2020). It often leads to avoidance of social interactions, low self-esteem and poor quality of life. Sleep disorders also impacts quality of life for many children (Roth 2007).

5Due to COVID-19, there was an opportunity to record 30 online consultations (15 minutes to 2 hours) in the Incontinence Clinic and Sleep Clinic. We used qualitative research methods that help in building structured linguistic frameworks by analysing conversations (Alam, Danieli, and Riccardi 2018) and further utilizing them in building dialogue sets for ECAs. The objective is to semantically and pragmatically analyse the actual human dialogues involving a coding framework comprised of 16 relational cues identified in the literature using NVIVO and Discursis. Relevant to the context, our analysis identifies differences between new and follow-up patients and different clinician roles (paediatrician, psychologist, physiotherapist, nurse). The results from the coding framework are further analysed statistically to more deeply understand the relationships, context and use of relational cues between different health specialists and patients towards specification of an empathic dialogue framework that supports complex interactions. We also identify design features for future ECA dialogue development and improvement in their emphatic structures.

6In the next section we describe the background domain relevant to the online consultations and evaluated ECAs (Section 2), followed by review of the literature in empathy and relational cues in dialogue (Section 3). The methodology is presented in Section 4 followed by results in Section 5. Finally, discussion, conclusions and future work are presented.

2. Background

7According to the International Children Continence Society (ICCS), the medical condition of incontinence refers to intermittent or continuous bed wetting during the day or night or both (Maternik, Krzeminska, and Zurowska 2015). Paediatric incontinence is a common condition affecting up to 20 percent of school-aged children (Malhotra et al. 2020) in many of their social activities like sports and sleepovers, which often leads to avoidance of social interactions. The children feel embarrassed and anxious, which leads to frustration and low self-esteem (Theunis et al. 2002). Children report a negative self-image owing to the physical and psychological impact of having incontinence, which is often unrecognised and seen as just another milestone in their growth (Butler 1998);(Harter 1982).

8Despite the fact that incontinence impacts the patient’s quality of life and is a cause of stress for them and their families (Malhotra et al. 2020);(Thibodeau et al. 2013) long waiting times to receive treatment are common, up to two years. This is because of a shortage of specialists, as incontinence is categorised as non-life threatening. Hence, ECAs could provide more timely support (Richards and Caldwell 2017); (Laranjo et al. 2018).

9To address the problem of long public hospital waitlists, an incontinence specialist for children aged 3-18 at the Children’s Hospital at Westmead (CHW) in Sydney, Australia, created an interactive eHealth program known as eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation). eADVICE enables young patients accompanied by their parents to get an online consultation regarding incontinence treatment factoring in the patient’s medical history and encoded algorithmic response scenarios that capture the domain knowledge of the health experts. Developed in 2016, the website was evaluated in several pilots that found adherence to the six possible recommended treatments was around 50 percent. To allow patients and families to ‘discuss’ their treatments, eADVICE was enhanced through the addition of an ECA known as Dr Evie (eVirtual agent for Incontinence and Enuresis), which gave a human embodiment to the online consultation experience. Possessing the actual voice of the incontinence specialist, this ECA significantly improved the adherence and health outcomes of patients on the hospital waiting list (Richards and Caldwell 2017). The success of Dr Evie can be attributed to its availability and its empathic and empowering dialogue (Bickmore 2004).

10Owing to its success for incontinence patients, the eADVICE approach – involving a website to provide tailored recommended treatments and an ECA to discuss the treatments – has been deployed for sleep disorders (eADVICE-sleep), another condition that is not life–threatening but significantly reduces quality of life (Roth 2007). Roth associated sleep disorders with “the presence of long sleep latency, frequent nocturnal awakenings or prolonged periods of wakefulness during sleep periods” . This condition is considered chronic if the sleep environment is comfortable but the daytime routine is full of distress, light headedness and anxiety due to lack of sleep (Kredlow et al. 2015).

11As sleep disorder patients also suffer from long waiting periods to access specialists and poor treatment adherence, they can potentially benefit from an ECA (Horsch et al. 2012); (Yin, Ring, and Bickmore 2012). Known as SAM (Sleep Adherence Mentor), the ECA in eADVICE-sleep acts as a virtual sleep coach. SAM has eight dialogue sets to cover the range of treatments and to ensure the dialogues are appropriate for the child’s age.

3. Empathy, Relationship Building and Relational Cues

12This review first defines empathy and then briefly reviews its role in human relationships and past ECA work involving empathy and human-ECA relationship building. To provide the basis for the coding themes in the methodology, we provide a brief review from linguistics to identify and define verbal relational cues and types of dialogue expressions that have been found to assist relationship building.

13Empathy is a complex human behavioural phenomenon defined by Hoffman as ‘the cognitive awareness of another person’s internal states that is, his thoughts, feelings, perceptions and intentions’ (p.29). Hoffman refers to empathy as any emotional reaction compatible with (but not necessarily similar to) the other’s situation. (Rogers and others 1959) theory of positive psychology and his client-centred framework emphasise that ’for a person to ‘grow’, they need an environment that provides them with genuineness (openness and self-disclosure), acceptance (being seen with unconditional positive regard) and empathy (being listened to and understood)’(Mamarimbing 2021), (p.8).

3.1 Empathy and Relationship Building with ECAs

14There has been more than a decade of research on the importance of empathy in human dialogues, which has led to interest in how ECAs can express empathy to bring about behaviour change (McRorie et al. 2009);(Ochs, Pelachaud, and Mckeown 2017);(Ravichander and Black 2018). ECAs typically have a particular purpose referred to as task-oriented empathy (T. W. Bickmore, Caruso, and Clough-Gorr 2005); (Bickmore et al. 2010), which is more easily detectable in focused dialogues. However, social empathy, which is not task based, is also important, as it offers comfort and encourages long-term relationships (Bickmore 2004). According to Halpern , task-oriented empathy comes more naturally in doctor-patient dialogues while a doctor gathers the patient’s background information or recommends a certain treatment. Non-task-based, or social, empathy is more generic to the conversational themes in daily life. In designing ECAs, the component of social empathy is more complex and has been less commonly analysed (Halpern 2007).

15Owing to the importance of empathy in human relationships, many researchers have created and evaluated empathic ECAs such as GRETA (Hartmann, Mancini, and Pelachaud 2005) and REA, the Real Estate Agent (Cassell et al. 1999). Building and maintaining human-ECA relationships, however, is broader than congruent expression of empathy and includes other behaviours. Long-term relationships are highly influenced by the use of the right relationship-building dialogues (Stafford and Canary 1991). According to the psychology or medical literature, a working alliance is important for successful therapy (Halpern 2007). Many scales have been developed that emphasise the use of empathic and social dialogues during health-related consultations (Looije, Neerincx, and Lange 2008); (Yin, Ring, and Bickmore 2012).

16ECAs with empathic dialogues have been studied across a diverse range of health programs such as relational agents for anti-psychotic medication adherence (Bickmore et al. 2010), avatar-based health intervention to modify unhealthy lifestyles (Lisetti et al. 2012), exercise advisors that interact with older adults (T. W. Bickmore, Caruso, and Clough-Gorr 2005) and ECAs that can help cancer patients to adopt a positive lifestyle after chemotherapy (Greer et al. 2019). Research has suggested frameworks that determine the useful verbal and non-verbal behaviours for virtual agents, such as 10 cues including empathy, social dialogues and continuity (T. Bickmore, Gruber, and Picard 2005), the Big Five model of personality traits (Neff et al. 2010),five dialogue characteristics that exhibit relationship building (Richards and Caldwell 2017) and annotation schemes for negative emotions’ handling in customer care bots (Velsen et al. 2019).

3.2 Relational cues in Linguistics and Psychology

17Bardovi-Harlig defines pragmatics as ‘the study of language from the point of view of users, especially of the choices they make, the constraints they encounter in using language in social interaction and the effects their use of language has on other participants in the act of communication’ (p. 221). Research on social cognition through pragmatics (Bosco et al. 2015) has resulted in empathic pragmatic models (Zhanghong and Qian 2018) that refer to empathy in the context of verbal utterances that help build strong relationships.

18This paper focuses on identifying a set of verbal relational cues that have been reported in the literature, particularly in psychology, to build a working alliance or strong rapport between the patient and the health specialist. The literature has identified a number of relational cues as in the following examples.

• Empathy as a cue: Empathic phrases can be divided into three categories – Queries, Clarifications and Responses – which will be the foundation of empathic cue detection in the dialogue set (Coulehan et al. 2001).

• Social dialogue: Conversation can be broken into three phases: the opening, middle and closing phases with social dialogues playing their role in opening and closing phase (Laver 2011).

• Reciprocal self-disclosure:Conversations use a key social strategy defined as self-disclosure to build relations and increase conversational depth and as a process of disclosing details about yourself to the listener (Ravichander and Black 2018).

• Meta-relational Communication: The specific talk that results in enjoyable relationships, cooperation, building up self-esteem, giving compliments, being courteous and polite, mitigating criticism, fostering patience and forgiveness in the participants, encouraging openness and talking about the relationship’s quality and needs, and helping to acknowledge the relationship is categorized as the meta-relational communication (Stafford and Canary 1991).

• Continuity: During a conversation, the behavioral units that exhibit continuity are of three types: prospective, introspective and retrospective. For a doctor-patient interaction, prospective and retrospective are covered under verbal interaction (Gilbertson, Dindia, and Allen 1998).

• Reference to mutual knowledge: Some examples of mutual knowledge include knowledge of participant’s/patient’s biography, present life or habits, recent or future events (Planalp and Benson 1992).

• Affirmation: Something emotionally challenging or complex to comprehend is related to an affirmative response from the listener. The complexity can be due to multiple reasons ranging from whether or not take up a medicine or follow a recommended treatment that is not fully understood or experienced. The main purpose of affirmation is to enable to patient to express their frustration openly. Letting the patient voice out their concerns and responding in understandable utterances is the essence of affirmation. (Cameron et al. 2015).

• Confirmation: Means to reiterate the facts or validating the correctness of something previously believed or suspected to be the case is defined as confirmation. In eHealth, through confirmation, the physician recognises the situation conveyed by the patient and in return receives confirmation from the patient who sees the specialist as someone to confide in and understand their issues (Abramovitch and Schwartz 1996).

4. Methodology

4.1 Data Collection

19CHW recorded its live clinical calls for research purposes during the COVID-19 period from June to November 2020. In these calls, the patients are children aged 3–18 and they have specific issues such as incontinence of urine. The objective of this study is to analyse these actual human dialogues using qualitative research methods and draw findings focused on the element of empathy in doctor-patient conversations. This live-call dataset provides a unique view of how actual dialogues, recorded in a real-life environment, can suggest changes to the existing ECA dialogues and help to validate the existing component of empathy in these ECAs. The health specialists consists of two paedatricians (senior paediatrician is later referred to as senior doctor), one nurse, two physiotherapists and one registrar. All of them are experienced health specialists with experience ranging from 10-40 years. The data is composed of 23 unique patients and consultation sessions, with a total of 50,000 utterances, collected over six months from the incontinence and sleep clinics.

4.2 Data Pre-processing

4.3 Qualitative Analysis Overview

21Qualitative data analysis is an approach to finding patterns in conversations and interviews. The data itself can be unstructured, which means there are no predefined questions or predefined answers to choose from. The qualitative analysis method chosen depends upon not only the level of structure in the data but also how acquainted a researcher is with the subject being analysed. Figuring out the context of the data in the form of variables is known as coding. The description follows Bengtsson , who used content analysis in the domain of nursing, which is close to the domain of interest. Coding has two main approaches, deductive and inductive. In the deductive approach, the researcher is familiar with the content of the discussions and has developed an understanding of the context. Hence, the codes or themes are known beforehand. In the inductive approach, the researcher reads through the conversation and figures out the common words, semantics and context before grouping them into themes to define the coding framework. The next step is to decide whether to code the exact words or phrases from the conversation as codes or themes or to go deeper and understand the underlying meaning of the dialogue content and define that as a code. The former is known as manifest analysis, where the codes are the exact content; the latter is latent analysis, where the researcher goes under the surface and defines the codes according to the research aims (Bengtsson 2016).

4.4 Coding and Annotation Approach

22The present research approach is deductive latent analysis, which means that codes relevant to verbal behaviours leading to empathy will be extracted from a literature review and will have associated dialogues for further analysis. Each dialogue can be placed in multiple codes as well, depending upon the hidden context of utterances. Ensuring coding’s credibility increases the measure of trustworthiness of the coding (Graneheim and Lundman 2004). Credibility mechanism is how similar or dissimilar the coding results are when another person tries to reproduce the results. While qualitative methods do not claim to be reproducible, one way to improve credibility is to seek agreement between different researchers who do the coding independently and then establish a consensus. The approach is not to validate the coding quantitatively but to open the forum for discussion in which all experts come to consensus about each other’s way of coding (Woods and Catanzaro 1988).

4.4.1 Coding Process

23The coding phase in qualitative analysis begins by analysing the dialogues one by one to find themes according to the literature review on verbal and non-verbal behaviours found during conversations. In this study, the focus is on verbal behaviours because audio recordings cover only the verbal aspects of conversation. While finding themes, the focus is on the element of empathy; hence, all behaviours that exhibit empathy will be considered (e.g. politeness, inclusive pronouns). It is important to consider both task-oriented and social empathy. After defining and distinguishing themes from the literature to avoid overlaps, 16 themes were identified in relation to expressing empathy through dialogue: ten themes from Bickmore et al. and six themes from Richards and Caldwell as shown in Figure 1.

24For each relational cue, we have definitions, common key words and examples of sentences from the literature that can act as a set of guidelines for the application of the annotation scheme to other data sets. These details can be made available by contacting the first author.

4.4.2 Coding in NVIVO

25Coding was done in NVIVO which began with building a framework of nodes, cases and roles based on the research questions. The detailed review of sixteen themes from literature resulted in 13 parent codes, out of which 4 parent codes were further categorized into 14 child codes as shown in Figure 2. For simplicity and for further relative comparison of codes within roles each dialogue was coded in one category only. Also the child codes were given preference over the parent codes. For example, in case of motivational dialogue, the child codes were encouraging adherence, giving option and clarifying consequences. Only in rare cases where the dialogue was motivational but did not fit any child code was it assigned to the parent code. In the end, child code dialogues were aggregated to create the total count of the parent code.

4.4.3 Independent Coders’ Agreement through Cohen’s Kappa

26In order to seek agreement between how similar or dissimilar coding is, two annotators (SS and DR) took part in the validation process.SS was the main coder who had the context of approximately 50,000 dialogues. DR is an expert in qualitative analysis approaches but had not coded the whole dataset. After 200 dialogues were randomly selected from the first coder’s assessment, the second coder was briefed on the codes’ description, but the context remained missing until the first kappa was calculated so that consensus could be established in later discussions. The idea is to see how much the dialogues resonate with the theme even if the context is not given so that a more generalised coding can be obtained in the first iteration. The aim was not to validate the coding quantitatively but to open the forum for discussion in which all experts agree to each other’s way of coding (Woods and Catanzaro 1988). Various coder’s agreement techniques have been used in research including Krippendorff’s alpha as well as Scott’s pi and Cohen’s kappa (Artstein and Poesio 2008). Cohen’s kappa was selected as a measure of the agreement between the coder’s independent coding. An unweighted kappa score is calculated, which calculates the percentage agreement and the measure of coding by chance in each theme (Warrens 2015).

27Initially, a kappa score of 0.61 was calculated on 200 dialogues with 24 themes. The coders examined the percentage agreement within each theme and agreed that three themes – everyday conversational dialogues, inclusive pronouns and politeness strategies – should be re-coded since the dialogues that were coded in them were multi-thematic and were more appropriately placed in other themes like empowerment and motivation. Both coders re-coded the original 200 dialogues in these themes. Some confusion remained concerning dialogues that were specifically questions like ‘Do you have any more questions?’ or ‘Are you with me so far?’, as they were part of multiple themes. Based on a literature review, it was agreed that the appropriate classification was confirmation. These changes and clarifications resulted in a revised kappa score of 0.75. The remaining dialogues were reviewed one by one. After discussion about the categories, the second coder agreed to change allocations from affirmation to empathy, encouraging adherence to recommendation and from child theme to parent theme, if a dialogue had more than one child theme representation. Final coding resulted in agreement on 171 dialogues and a kappa value of 0.84.

28The remaining cases were resolved through discussion. In most cases, the context was mandatory for the assignment because the dialogues had more than one coding category in them such as affirmation or explanation and the context placed it correctly in empathy.

29Following consensus on the 200 dialogues, the main coder reclassified any dialogues in the 50,000 dialogues that had been reassigned to three themes – everyday conversational dialogues, inclusive pronouns and politeness strategies. Finally, the main coder confirmed that all of the dialogues used everyday conversational language and that use of personal pronouns would be automatically calculated by searching for the terms ‘us’ and ‘we’ as a rough but quick method of assessment that would enable comparison between roles and individuals.The coding process resulted in parent and child codes listed in Figure 2.A few annotated dialogues after consensus are shown in Figure 3 and Figure 4 to give a glimpse of dialogues and their chosen relational cues.

4.5 Empathic Cues: Analysing the Dr Evie and SAM Dialogue Sets

30Dr Evie and SAM both have structured dialogues with empathic cues already being a part of their semantics. The hypothesis of this study is that live unstructured sessions can bring out different sentence structures that are more beneficial in creating a level ground of adherence for the patients. This includes a validation process in which Dr Evie and SAM would be thoroughly screened for empathic cues finalised in the prior coding process. A similarity scale would determine the threshold, and based on the similarity scores further empathic cues could be recommended. The same coding process used in Sections 4.4 was followed to evaluate these. After normalising the results, a comparison has been done to show how empathic both dialogue sets are.

5. Results

31This section covers the analysis of 23 patient’s consultations. The age of the patients ranges from 9-12 years with 10 patients being females while the rest are male. The consultations consist of 12 follow-ups and 11 new patient sessions. In 22 session,patients are accompanied by one or both parents and the average session length is 40 minutes. Out of 23, 16 sessions have one health specialist while 7 sessions have 2 health specialists with one always being the senior doctor. The role of the medical specialist and whether the consultation was a new or follow-up meeting are used for the comparison of results. The findings from the recorded consultations are discussed in Sections 5.1 to 5.3. Section 5.4 analyses results from the recorded consultations with the Dr Evie and SAM dialogues. In Section 5.5 the dialogue structure and topic sequencing is formulated into the coding framework.

5.1 Percentage Distribution of Relational Dialogues

32The percentage of dialogue usage was compared for the following six categories:

1. Senior doctor’s consultations with first-time patients

2. Senior doctor’s consultations with follow-up patients

3. Average usage of relational cues by the senior doctor with follow-up versus new patients (Figure 5)

4. Physiotherapist’s consultations with patients

5. Nurse’s consultations with patients

6. Usage of relational cues in sessions where both the senior doctor and physiotherapist are present (Figure 6).

33It is evident from the coding percentages shown in Figure 5 that patients who visit the clinic for the first time need more information about their health issue and its remedies. The senior doctor uses more informational and motivational dialogues and decision-making is also encouraged. The health specialist also needs to ask about a new patient’s medical history and current health status more than for the follow-up patients.

34For follow-up patients, the percentage usage of cues is more variable as it depends on how many sessions the patient has had before and their progress to date. For some patients, it is more about encouraging adherence to a treatment discussed in previous sessions. For others, the effects of new recommendations need clarification and dialogues confirm their understanding. Figure 5 also shows that the senior doctor’s empathic cues’ usage is the same for both new and follow-up patients but the difference lies in encouraging adherence and informational and motivational dialogues. Informational dialogues are more for new patients, whereas motivational and decision-making cues are more for follow-up patients. The reciprocal self-disclosure and social dialogues feature less in percentage usage but they are relatively more used for first-time patients.

35The physiotherapist and nurse are the other most frequent roles found in the consultations. Most of the consultations contain one therapist role but in consultations where multiple specialists are present, as shown in Figure 6, where both senior doctor and physiotherapist appear, the physiotherapist uses more task-based dialogues and the senior doctor uses more adherence and decision-making dialogues. Social dialogues are uttered more by the physiotherapist, but empathic cues are uttered more by the senior doctor.

5.2 Role Differences and Similarities in the Use of Relational Dialogues

36The data helps in analysing the use of relational cues in multiple roles. In total, there were six roles to analyse, with more consultation sessions for the senior doctor, physicians and nurse. The remainder of the medical team – a paediatrician, registrar and male physiotherapist – had only one consultation each. More data is needed to analyse their use of relational cues, but for the purpose of this research they are grouped into one category under ‘others’. Owing to few data samples and lack of confirmation of normal distribution in the usage of a particular theme within a category under observation, non-parametric (i.e. Mann Whitney U) tests were chosen to understand the differences and similarities in the use of relational dialogues. The Mann Whitney U test compares outcomes between two independent groups based on the median of two distributions. The test was performed on all relational cues. The results that are significant at 90 percent confidence level or have high U-values include empowerment, explanation, social dialogues and reciprocal self-disclosure when comparing senior doctor’s usage of relational cues with junior doctors. These categories were further analysed to determine whether the senior or junior doctors used more of these cues.

37Dominance is defined as the higher usage percentage of the relational cue in most of the sample points for that role. It is evident that Senior doctor is dominant in using relational cues like explanation, reciprocal self-disclosure and clarifying consequences while junior doctors are dominant in using relational cues like empowerment, social dialogues and task-based dialogues. Examples of visualisations that support the statistics are presented in Figure 7.

5.3 Demographic Influence on Use of Relational Dialogues

38The influence of patient demographics including gender,age and first time visitation on the use of relational dialogues has been analysed using Mann-Whitney U scores in Figure 8. The higher the U-value, more significant is the difference of relational cues usage in the respective groups. The p-value and z-score further determine the significance of the differences in the given empathic cue’s group. Figure 8 also shows U-values of new and follow-up consultations. Although first time visitation is not the demographics of the patient but it has a major contextual influence on the usage of relational cues. The new patients needed to be walked through the whole treatment details and were encouraged to speak up about their current health issues. The follow-up patients were either asked for time and volume charts or about their adherence to a certain treatment that had been discussed in previous consultations.

5.4 Comparison of Dr Evie’s and SAM’s Dialogues with Live-call Dialogues

39Dr Evie’s dialogue set consists of multiple treatment-based dialogue streams, including alarm training, bowel program, caffeine intake, fluid increase, medication and time voiding. The same coding framework used for the recorded consultations was applied to the dialogues used in Dr Evie to facilitate a comparison between the usage of relational cues on live clinical calls and Dr Evie. Since the dialogue sets are those used for creating the Dr Evie application, they cannot be compared in terms of actual sessions that patients have with Dr Evie. The whole dataset provides a statistical presence of relational cues, which are shown in Figure 9.

40The dialogue set of Dr Evie was designed by experienced health professionals who are experts in incontinence and other related disorders; hence, percentage usage of dialogue cues in live clinical calls is similar to that of relational cues in Dr Evie, especially for education, empowerment, encouraging adherence and giving options. The live clinical calls had a lower percentage usage of cues such as reference to mutual knowledge, greetings and farewells and the same can be seen with Dr Evie. The other cues such as empathy, explanation, recommendation, confirmation and clarifying consequences are also present in Dr Evie in good proportion and comparable with their usage in live clinical calls. Relational cues such as talking about the past and future together and social dialogues are missing in Dr Evie’s context, Dr Evie’s use of relational cues could be enriched by recommendations from live calls.

41SAM’s dialogue sets are mostly around sleep routine management and diet options: caffeine intake, regular sleep, night terrors, sleep hygiene and snoring issues. The recommendations are also about the sleep routine and diet habits. The dialogue flows have informational dialogues, current health assessment and recommendation dialogues incorporated in empathic language (e.g. ‘I understand’ and ‘I know this is hard but’). SAM’s dialogues are rich in empathic cues and affirmation dialogues, but lack cues for encouraging adherence, clarifying consequences and explanation. Live calls dialogue cues for these themes can be used to enrich SAM’s dialogue set.

5.5 Structure, Topic and Inter-speaker Relationship Analysis

42Health consultations have a defined and specific structure that exhibits the ontology of the subject area (T. Bickmore, Gruber, and Picard 2005). In order to embed relational cues in a logical manner, it is important to understand the conversational structure, topic variance and inter-speaker contribution in the overall consultation. The recurrence of topics and the time taken by speakers determine the level of engagement and understanding among the participants.

43The dialogue structure in health consultations presented in Baker, Richards, and Caldwell places the dialogue cues from Bickmore et al. into a structure found in real consultations. Starting with greetings and farewells, social dialogues and previous treatment-related dialogues, it continues into more empathic dialogues and reciprocal self-disclosure cues. The last part of the conversation is more about future recommendations, adherence and continuity of the consultations. Our dataset is also mapped onto this logical structure to validate its existence in live calls, which suggests that even if the ECAs are built on natural language instead of structured questions, they would follow a similar structure as shown in Figure 10.

6. Discussion

44The role of relational cues was analysed by the percentage of their occurrence in conversations. The analysis showed that two relational cues were used in all conversations but their percentage of usage in terms of utterance count by each role is very low – greetings, farewells and continuity. This confirms the (Laver 2011) finding that social dialogues are uttered mainly during the greeting and farewell phases of a conversation. Four relational cues were used very rarely in all conversations – expressing happiness to see the users, reference to mutual knowledge, talking about the past and future together and reciprocal self-disclosure. These cues are highly connected to how long term the relationship is with the patient and the level of comfort.The highest usage of relational cues is for empathy and motivational, informational and decision-making codes. Relational cues differ according to the session type and whether it is with a new or follow-up patient (Norfolk, Birdi, and Walsh 2007). A new patient needs to be on-boarded with more communication related to treatment mechanics and education of the health issues. A follow-up patient needs more encouragement and dialogues related to clarifying consequences.

45This analysis provided detailed findings on roles and demographics in the live calls. These insights can be used to create customised responses for different gender and age groups. Two factors that make the ECAs humanlike are their appearance and intelligence (Raval 2020), which depend highly on dialogue management producing dialogues similar to actual human dialogues. The live calls were more contextualised and customised. They included cues such as self-disclosure, confirmation and explanation. If the ECA dialogues were modified to capture more of a patient’s context, these cues could be used to enrich Dr Evie and SAM dialogues.

46Multiple roles can be introduced in Dr Evie and SAM, as the senior doctor uses more explanation and empowering cues and the junior doctor takes care of certain routine tasks. This can make dialogue sets more empathic. To implement an approach where multiple ECAs can support a patient in different ways to provide holistic care, the Council of Coaches platform could be used (Akker et al. 2018) where multiple ECAs review the patient’s situation together and have separate conversations with the user according to their specific area of expertise or role (e.g. dietician, physiotherapist, friend).

47The above suggested extensions to Dr Evie and SAM, such as more data gathering leading to more personalised use of relational cues, can be applied more widely to the design of other ECAs. This would involve the inclusion of user/patient models that persist between consultations, which allow the knowledge of the ECA to grow and be updated each time it meets the patient. This would also allow the ECA to tailor its social dialogue to the interests of each human. The relational cues and approach used in this article can be used to evaluate other ECAs and more relational cues could be incorporated to improve the relationship built with the user, with the aim of improving health outcomes. Another contribution of this paper is the capture of valuable datasets. The dataset and findings can be used by others in different ways. It is evident that live calls contain relational cues (e.g. disclosure and social dialogue) and dialogue patterns that could be used to design relational dialogue for ECAs. This dataset can also be used in machine learning and AI-based agents (Van Welbergen, Yaghoubzadeh, and Kopp 2014), which can learn to respond and formulate conversation using natural language generation. The live-call transcripts could also be used for the medical training of patient-doctor communication. This would be an alternative or supplement to approaches that offer guidance for health practitioners such as those provided by Rogers’ (1959) client-centred therapy, which includes empathy, genuineness and unconditional acceptance. There is clear overlap in some of the relational cues and in the approach suggested by Rogers.

48Finally, live clinical calls provide utterances from various roles. These roles include not only health specialists but also patients. Hence, the data can be used to build multiple ECAs that have different roles to facilitate the training of practitioners. While the quality of the recordings of patients and family members was too poor to allow transcription and qualitative analysis and outside the scope of this study due to its focus on practitioner use of relational cues, some specific calls in the live-call dataset can be used to build virtual patient ECAs, so that doctors can practise and refine their patient-doctor conversational expertise.

7. Limitations and Future Work

49The dataset of live clinical calls was collected during the COVID-19 timeframe when normal clinical practice was disrupted. Live online clinical consultations became the new norm, which made their recording possible. Nevertheless, not all patients or practitioners were comfortable with this form of consultation, and it is possible that the dialogues were different to what might have been recorded in live face-to-face sessions in consultation rooms. The delay at the start to obtain consent prior to recording may have also inhibited the naturalness of the conversation and relationship.The ECAs for incontinence and sleep have been studied in conjunction with live calls, but more ECAs in domains specific to children should be explored to establish the use of relational cues especially in terms of the health specialist. As recommended by an anonymous reviewer, in the future it would be interesting to analyse empathy in a diachronic perspective, analysing the evolution of elements of empathy in doctor-patient conversations over time.

50Conversational unit interfaces (CUIs) in health care are able to analyse natural languages (Laranjo et al. 2018) and to build responses according to the patient’s situation and history. As we move into the digital era, reliance on virtual agents that talk and understand like humans is a big research area (Sas, Whittaker, and Zimmerman 2016). Dr Evie uses scripted dialogues, whereas SAM uses more sophisticated technology that takes into account the user’s goals and beliefs. The architecture SAM uses also allows preferences, medical history and other contextual features to be included in the ECA’s reasoning and to provide explanations (Abdulrahman and Richards 2019). As Sam was developed later, we tried to incorporate more empathy and affirmation cues into its dialogues. Now that we have the data from live calls, Sam can further be enriched with encouraging adherence and clarification cues.

51SAM and Dr Evie use fixed choice responses, primarily to ensure patient safety and accuracy, which is a current risk in health domains due to limitations in natural language processing (Xu et al. 2020). In the future when these limitations are addressed, safe and reliable solutions that use natural language input can be evaluated with a mix of controlled responses.

52In the future, the recommendations for ECA dialogue design can be utilised to produce more dialogue sets that can be generalised over certain situations and cultures. Since negative thoughts can aggravate health issues, an empathic ECA that is personalised to the individual could help both mental and physical well-being. Hence, future agents for all health issues can potentially benefit from the relational cues and their usage presented in this paper.