Creating a body of physical activity evidence to test the generalisation of annotation methods for automated evidence synthesis

Abstract

Background The Human Behaviour-Change Project (HBCP) aims to improve evidence synthesis in behavioural science by compiling intervention reports, annotating them according to an ontology, and using the resulting data to train information extraction and prediction algorithms. The HBCP used smoking cessation as the first ‘proof of concept’ domain but intends to extend its methodology to other behaviours. The aims of this paper are to (i) assess the extent to which methods developed for annotating smoking cessation intervention reports were generalisable to a corpus of evidence relating to a different behaviour, namely physical activity, and (ii) describe the steps involved in developing this second HBCP corpus. Methods The development of the physical activity corpus took place in four stages: (i) reviewing the suitability of smoking cessation codes already used in the HBCP, (ii) defining the selection criteria and scope of the corpus, (iii) identifying and screening records for inclusion, and (iv) annotating intervention reports using a code set of 200+ entities from the Behaviour Change Intervention Ontology. Results Stage 1 highlighted the need to modify the smoking cessation behavioural outcome codes for application to physical activity. One hundred physical activity intervention reports were reviewed, and 11 physical activity experts were consulted to inform the adapted code set. Stage 2 involved narrowing down the scope of the corpus to interventions targeting moderate-to-vigorous physical activity. In stage 3, 111 physical activity intervention reports were identified, which were then annotated in stage 4. Conclusions Smoking cessation annotation methods developed as part of the HBCP were mostly transferable to the physical activity domain. However, the codes applied to behavioural outcome variables required adaptations. This paper can help anyone interested in building a body of research to develop automated evidence synthesis methods in physical activity or for other behaviours.