The trust in AI-generated health advice (TAIGHA) scale and short version (TAIGHA-S): Development and validation study

Abstract

Artificial Intelligence (AI) tools such as large language models (LLMs) are increasingly used by the public to obtain health information and support health-related decisions. Because such use involves advice-taking, following or rejecting AI-generated advice can have direct clinical and safety implications and consequences for the healthcare system. Although trust plays an important role in the adoption of health-related AI advice, existing instruments only assess trust in generic technology and perceived trustworthiness. There are currently no validated instruments that specifically measure users’ state trust in AI-generated health advice, and an alternative for self-developed one-item scales is missing.This study aimed to develop and validate the Trust in AI-Generated Health Advice (TAIGHA) scale and its four-item short form (TAIGHA-S) as theory-based questionnaires for measuring trust and distrust in AI-generated health advice. We used a generative AI approach to generate new use-case specific candidate items based on existing theory that each comprised cognitive and affective components. After automated validation, we conducted manual validation in three steps: (i) content validation with ten domain experts, (ii) face validation with 30 lay participants, and (iii) psychometric validation with 385 UK participants receiving AI-generated health advice for symptom assessment. After automated item reduction, 28 items were retained and further reduced to 10 based on expert ratings. The final TAIGHA scale showed excellent content validity (S-CVI/Ave = 0.99) and face validity (S-FVI/Ave = 0.99). CFA confirmed a two-factor model with excellent fit (CFI = 0.98, TLI = 0.98, SRMR = 0.03). Internal consistency was high (α = 0.94, ω = 0.94 for trust; α = 0.93, ω = 0.93 for distrust). The short form correlated highly with the full scale (r = 0.96) and showed high reliability (α = 0.88, ω = 0.89 for trust; α = 0.84, ω = 0.85 for distrust). TAIGHA and TAIGHA-S are validated instruments for assessing users’ state trust and distrust in AI-generated health advice, with excellent psychometric properties and stronger associations with reliance than existing general trust scales.