ARIA index to assess the quality of mobile health apps: A usability and reliability study

Abstract

Purpose: The ubiquity of smart devices has precipitated the growth of mobile health applications (mHealth apps), which aim to help users manage health conditions. Persons living with chronic health conditions, especially older adults, can benefit from the use of mHealth apps given their diverse uses. However, challenges persist with regulation of this industry. Thus, the quality of mHealth apps are often unknown. As no tools existed to assess the quality of mHealth apps for persons living with a health condition, care partners, or healthcare practitioners, we developed the Alberta Rating Index of Apps (ARIA), with a paper-based version validated in a previous study (Azad-Khaneghah, Roduta Roberts, & Liu, 2022). ARIA consists of 19 items rated on a Likert scale that ask users to reflect on the usability, trustworthiness, appropriateness, security, and satisfaction of the mHealth app. For example, one item reads “It would be easy for the user to understand the information provided by the app.” Subsequently, we developed a web-based version of the index. A web-based version may act to enhance accessibility, broaden the user audience, and improve functionality through computerized features, such as customized font size and automatic saving of progress. Older adults may benefit from these digitized features given the accessibility features available online to accommodate for a range of age-related impairments. Methods: In this study, the usability and reliability of a web-based version of ARIA was assessed. Eighty-four (84) participants trialed two mHealth apps (Calm and Breathe2Relax) and completed the ARIA Online questionnaire based on their experiences. A retest interval was observed before completing a second round of the questionnaire, in addition to the System Usability Scale (SUS). Frequency distribution described demographics, and statistical analyses included factor analysis, and reliability coefficients (internal consistency, inter-rater, and test-retest); descriptive analysis were applied for SUS scores (mean and standard deviation). Results and Discussion: 61 persons living with a health condition, 15 care partners, and 8 healthcare practitioners participated in the study. An exploratory factor analysis identified 11 questionnaire items to be appropriate according to the correlation matrix (coefficients >0.70, KMO 0.838, chi-square 1458.00, 171 df, p<0.001). Reliability analyses show high internal consistency among ARIA items (Cronbach’s ⍺ = 0.819), agreement among raters (ICC = 0.819), and reliability over time (r = 0.590, p<0.01). The SUS scores indicated that ARIA Online was well-accepted (mean = 83.85, SD = 9.85). Our results suggest that ARIA Online is both usable and reliable when assessing the quality of mHealth apps. An established and usable index for mHealth apps could equip app users to accurately assess the quality of mHealth apps. ARIA online can empower users who are considering adopting an application, ultimately enhancing their health management journey.