1756-0500-7-725 1756-0500 Research article <p>Using postal questionnaires to evaluate physical activity and diet behaviour change: case study exploring implications of valid responder characteristics in interpreting intervention outcomes</p> ColeAJudithjcole04@qub.ac.uk GillespiePaddypaddy.gillespie@nuigalway.ie SmithMSusansusansmith@rcsi.ie ByrneMollymolly.byrne@nuigalway.ie MurphyWAndrewandrew.murphy@nuigalway.ie CupplesEMargaretm.cupples@qub.ac.uk

Department of General Practice and Primary Care, Queen’s University Belfast, Dunluce Health Centre, 1 Dunluce Avenue, Belfast BT9 7HR, UK

Economics, School of Business and Economics, NUI Galway, Galway, Ireland

Department of General Practice and HRB Centre for Primary Care Research, Royal College of Surgeons in Ireland, Lower Mercer Street, Dublin 2, Ireland

School of Psychology, Arts Millennium Building, NUI Galway, Galway, Ireland

Discipline of General Practice, Clinical Science Institute, NUI Galway, Galway, Ireland

BMC Research Notes
<p>Medical Research Methodology</p>
1756-0500 2014 7 1 725 http://www.biomedcentral.com/1756-0500/7/725 10.1186/1756-0500-7-72525315148
1642014610201415102014 2014Cole et al.; licensee BioMed Central Ltd.This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. PROMs Lifestyle Self report questionnaires Diet Physical activity Research methods

Abstract

Background

Patient reported outcome measures (PROMs) are used to evaluate lifestyle interventions but little is known about differences between patients returning valid and invalid responses, or of potential for bias in evaluations. We aimed to examine the characteristics of patients who returned valid responses to lifestyle questionnaires compared to those whose responses were invalid for evaluating lifestyle change.

Methods

We conducted a secondary data analysis from the SPHERE Study, a trial of an intervention to improve outcomes for patients with coronary heart disease in primary care. Postal questionnaires were used to assess physical activity (Godin) and diet (DINE) among study participants at baseline and 18 month follow-up. Three binary response variables were generated for analysis: (1) valid Godin score; (2) valid DINE Fibre score; and (3) valid DINE Total Fat score. Multivariate analysis comprised generalised estimating equation regression to examine the association of patients’ characteristics with their return of valid responses at both timepoints.

Results

Overall, 92.1% of participants (832/903) returned questionnaires at both baseline and 18 months. Relatively fewer valid Godin scores were returned by those who left school aged <15 years (36.5%) than aged 18 and over (50.5%), manual workers (39.5%) than non-manual (49.5%) and those with an elevated cholesterol (>5 mmol) (34.7%) than those with a lower cholesterol (44.4%) but multivariate analysis identified that only school leaving age (p = 0.047) was of statistical significance.

Relatively fewer valid DINE scores were returned by manual than non-manual workers (fibre: 80.8% v 86.8%; fat: 71.2% v 80.0%), smokers (fibre: 72.6% v 84.7%; fat: 67.5% v 76.9%), patients with diabetes (fibre: 75.9% v 82.9%; fat: 66.9% v 75.8%) and those with cholesterol >5 mmol (fat: 68.2% v 76.2%) but multivariate analysis showed statistical significance only for smoking (fibre: p = 0.013; fat: p = 0.045), diabetes (fibre: p = 0.039; fat: p = 0.047), and cholesterol (fat: p = 0.039).

Conclusions

Our findings illustrate the importance of detailed reporting of research methods, with clear information about response rates, respondents and valid outcome data. Outcome measures which are relevant to a study population should be chosen carefully. The impact of methods of outcome measurement and valid response rates in evaluating healthcare requires further study.

Background

A large body of evidence, from both clinical and observational studies, has been built up over recent decades to support the principle that a lifestyle which includes regular physical activity and a balanced, healthy diet helps to prevent disease 1 . The effectiveness of interventions incorporating different approaches to lifestyle change has been reported for both the primary and secondary prevention of coronary heart disease (CHD) 2 3 . However, gaps remain in the evidence regarding optimal intervention design. Crucial to evaluating the effectiveness of lifestyle interventions are the methods chosen to assess changes in physical activity and diet.

Self-report measures of physical activity, such as questionnaires or diaries, are used widely although they are subject to recall bias 4 5 and may have limited accuracy 6 . Objective measures such as accelerometers, pedometers or heart rate monitors 7 can also be used to assess physical activity but these are more expensive and study participants require instruction in their use. Similarly, a variety of measures can be used to assess diet, including specialist techniques involving doubly labelled water, direct observation, detailed food diaries and questionnaires: measurement errors can exist in every method 8 . Food frequency questionnaires are widely used, given that they are relatively easily administered, inexpensive and provide useful information 9 although multiple issues are relevant to their validity 10 . Higher numbers of items, giving greater detail, may increase validity 11 but higher response rates are more likely with shorter questionnaires 12 .

The SPHERE Study (Secondary Prevention of Heart Disease in General Practice) 13 14 aimed to evaluate individually tailored care plans to improve outcomes, including health-related behaviours, for patients with CHD in primary care. The study chose self-report measures to assess behaviour, using the Godin Leisure Time Questionnaire 15 to assess levels of physical activity and the Dietary Instrument for Nutrition Education (DINE) questionnaire 16 to assess diet.

The Godin questionnaire 15 provides a measure of type, duration and intensity of leisure-time exercise and is designed to be straightforward to use 17 . It has been validated 18 and tested for reliability 19 in studies in which subjects had opportunity to ask questions but other studies have reported posting it to participants 20 21 without offering this facility. It is a four-item questionnaire, which gives examples of mild, moderate and strenuous exercise and asks how many times per week the respondent undertakes 15 minutes of each exercise category.

The DINE questionnaire 16 is a short questionnaire which is considered an acceptable alternative to more detailed diet recall questionnaires and food diaries. It gives a measure of dietary intake of fibre and fat, focusing on these because of their important association with cardiovascular disease and cancer and excluding other diet components in order to keep the questionnaire as short as possible. It was validated amongst workplace employees 16 , to whom nurses administered the questionnaire: scores derived from it were compared with 4-day diet records. Face-to-face administration allowed ambiguous or unclear responses to be clarified immediately but in other studies DINE has been posted to participants 22 23 .

Within the SPHERE study, questionnaires were posted to participants and returned to practices by post, without opportunity for advice or review. Whilst the study achieved 92.8% follow-up 24 , assessment of change in physical activity and in diet was possible only for, respectively, 38% and 75% of the total sample (these were the proportions of participants who provided valid responses at both time points). Thus, potential respondent bias may have implications for the interpretation of the evaluation. Whilst people who live in affluent areas are more likely to respond to questionnaires than those who live in deprived areas 12 differences in characteristics of patients with CHD who do and do not return valid responses to lifestyle questionnaires, administered by post, are not known.

This study aimed to examine possible respondent bias within the SPHERE Study by conducting a secondary data analysis to determine if characteristics of those who returned responses to postal questionnaires which allowed valid assessment of their physical activity and diet differed from those who did not. We also incorporated qualitative observations from a later follow-up study during which the researcher directly observed participants completing questionnaires.

Methods

Background: the SPHERE study

This study is set in the context of the SPHERE Study, a large cluster randomised trial conducted in the two different primary healthcare systems on the island of Ireland 13 . Briefly, participants were recruited from three different centres (Belfast, Northern Ireland (NI), UK; Galway, Republic of Ireland (RoI) and Dublin (RoI). Forty eight practices (16 from each centre) were randomly selected by an individual independent of the research team, using computer generated random numbers.

Lists of patients with known CHD were compiled by practice staff and those with significant mental or physical illness likely to impair capacity to participate in the intervention were excluded. Patients were invited, by post, from random order lists until 20 from each practice agreed to take part. They were posted questionnaires, including Godin and DINE, with reply paid envelopes which were posted back to the practice, for collection by the researchers, without review by practice staff. Six weeks were allowed for non-response, during which reminders were posted.

Baseline data collection was completed before randomisation into intervention and control groups to minimise potential recruitment bias. During the following 18 months (2005–2007) those in the intervention group attended their practice nurse or general practitioner for 4-monthly consultations at which lifestyle, risk factors and medication were reviewed. Goals for lifestyle change and targets were set, for review at each subsequent consultation. After 18 months, in addition to objective measurement of bio-physical risk factors, participants were again posted questionnaires following the same process as at baseline.

The SPHERE Study was granted ethical approval by the Irish College of General Practitioners and the Queen’s University research ethics committee.

Qualitative study

A follow-up qualitative study of barriers and facilitators affecting lifestyle change was conducted after 4 years and has been reported previously 25 : 45 participants were interviewed and at the end of their interview they were invited to self-complete a Godin and a DINE questionnaire in the presence of the researcher who addressed any queries. Observational data relating to the completion of these questionnaires were extracted from that study in order to help contextualise the quantitative analyses described below.

Ethical approval for this follow-up study was granted by the Office for Research Ethics Committees (Northern Ireland), ref 10/NIR03/11 (April 2010), and the Irish College of General Practitioners Research Ethics Committee (May 2010).

Data management

The aim of this current paper was to use data from the SPHERE Study to compare characteristics of participants who did or did not return questionnaires from which valid outcome measurements of physical activity and diet could be calculated at both baseline and follow-up.

An invalid response to the Godin questionnaire was considered to include any response other than a clear number of 15-minute PA sessions. Responses were scored to indicate the level of health benefit resulting from the reported exercise level. A score of <14 units indicates an insufficient level of activity with low health benefits; 14–23 indicates moderate activity with some health benefits; ≥24 indicates sufficient activity with substantial health benefits 17 . An invalid response to the DINE questionnaire was attributed to anything other than a stipulated answer 16 . Questions relate to the frequency of consumption of 19 food groups which contribute about 70% of the fat and fibre in a common Western diet. Each food group is given a score to reflect the nutrient content of a standard portion size, and scores are weighted according to frequency of consumption. No attempt was made to interpret unclear or missing responses. Within the current study, analysis was confined to DINE scores relating to fibre and total fat.

Statistical analyses

Three binary response variables were generated for the statistical analysis: (1) valid Godin score; (2) valid DINE Fibre score; and (3) valid DINE Total Fat score. The statistical analyses examined rates of valid scores for each response variable for study participants with varying characteristics using a multivariate approach in which the choice of regression model was informed by the nature of the response variables under consideration and the hierarchical nature of the SPHERE dataset. In the case of the latter, there is a natural classification to the observations at the level of an individual general practice surgery, such that data are organised or clustered according to the practice with which a person is registered. With clustered data individual observations are not independent and multilevel analytical approaches are required for regression analysis 26 . In this study, generalised estimating equation (GEE) multivariate regression analysis was used to explore the effect of a range of independent variables on the three response variables of interest. When estimating a GEE model, in addition to identifying the appropriate linear predictor, it is also necessary to specify a suitable variance function, link function, and correlation structure 26 . The approach adopted in the current analysis was a binomial variance function and a log link function (both based on the binary nature of the response variables), and an exchangeable correlation structure (based on recommendations from the literature 26 ). The patient characteristics which were included as independent variables in the multivariate regression models were: age, gender, years since diagnosis, history of myocardial infarction, systolic blood pressure, diastolic blood pressure, total cholesterol, smoking status, body mass index, SPHERE treatment allocation, age left school, marital status and occupational status.

The results of the multivariate analyses allow us to identify statistically significant associations for three response variables of interest, with statistical significance set at p < 0.05 for all analyses. The statistical analyses were conducted using the statistics package Stata 13.

Results

In total, 903 patients took part in the SPHERE Study; all returned questionnaires at baseline and 832 (92.1%) completed response questionnaires at follow-up after 18 months.

Response rates

Table 1 shows that the rates of return of valid Godin scores at both baseline and 18 month follow-up were similar (approximately 60%); a higher percentage returned valid DINE scores at baseline (95% fibre; 90% fat) than at follow-up (86% fibre; 81% fat).

<p>Table 1</p>

Total questionnaires returned

Valid scores

Invalid scores

N (%)

N (%)

*Baseline total N = 903; **Follow-up N = 832; 71 lost to follow-up due to: death; withdrawn consent; left practice; too ill; too busy; admitted to nursing home (44 intervention; 27 control).

Godin

Baseline*

541 (59.1)

362 (40.9)

Follow-up**

523 (62.9)

309 (37.1)

DINE fibre

Baseline*

861 (95.4)

42 (4.6)

Follow-up**

716 (86.1)

116 (13.9)

DINE total fat

Baseline*

814 (90.1)

89 (9.90)

Follow-up**

673 (80.9)

159 (19.1)

Numbers who returned valid and invalid questionnaire scores at different time points during the SPHERE Study

Participants who did not return a follow-up questionnaire (n = 71) were excluded from our analyses. For the remaining 832 participants, characteristics were compared between those who returned questionnaires for which valid scores could be calculated at both time points and those who did not. Valid Godin scores were returned by 41.9% of the original sample (349/832), valid DINE fibre scores by 82.5% (686/832) and valid DINE total fat scores by 74.0% (616/832) at both time points.

Descriptive comparisons of patient characteristics

In relation to the Godin questionnaire, fewer of the participants who left school at age 14 or under returned valid scores (36.5%), compared to those who left school aged 18 and over (50.5%) (Table 2); 39.1% of manual workers returned valid scores, whilst 49.8% of participants in non-manual occupations did so. Fewer of those whose total cholesterol was >5 mmol/l returned valid Godin scores (34.7%) than did those with lower cholesterol levels (44.4%).

<p>Table 2</p>

GODIN

DINE fibre

DINE fat

Total study population (N = 832)*

Valid response

Invalid response

Valid response

Invalid response

Valid response

Invalid response

N (%)

349 (41.9)

483 (58.1)

679 (81.6)

153 (18.4)

616 (74.0)

216 (26.0)

Patient characteristics**

N (%)

N (%)

N (%)

N (%)

N (%)

N (%)

*Total N = 832; numbers for some characteristics are less due to incomplete data; percentages refer to row totals.

**Baseline data; ***Elevated SBP defined as systolic blood pressure >140 mmHg; elevated DBP defined as diastolic blood pressure >90 mmHg; elevated cholesterol defined as serum cholesterol >5.0 mmol/l.

Age (years) (Mean; SD)

Mean 67.2; SD 9.53 (Range 29–91)

66.6 (9.77)

67.6 (9.33)

67.3 (9.57)

66.7 (9.37)

67.2 (9.58)

67.3 (9.41)

70 years and older

340 (40.9%)

138 (40.6)

202 (59.4)

280 (82.4)

60 (17.6)

247 (72.6)

93 (27.4)

<70 years

492 (59.1%)

211 (42.9)

281 (57.1)

399 (81.1)

93 (18.9)

369 (75.0)

123 (25.0)

Gender

Male

582 (70.0)

250 (42.9)

332 (57.1)

479 (82.3)

103 (17.7)

435 (74.7)

147 (25.3)

Female

250 (30.0)

99 (39.6)

151 (60.4)

200 (80.0)

50 (20.0)

181 (72.4)

69 (27.6)

History of MI

Yes

415 (49.9)

182 (43.9)

233 (56.1)

338 (81.4)

77 (18.6)

310 (74.7)

105 (25.3)

No

417 (50.1)

167 (40.0)

250 (60.0)

341 (81.8)

76 (18.2)

306 (73.4)

111 (26.6)

History of diabetes

Yes

145 (17.4))

58 (40.0)

87 (60.0)

110 (75.9)

35 (24.1)

97 (66.9)

48 (33.1)

No

685 (82.3)

290 (42.3)

395 (57.7)

568 (82.9)

117 (17.1)

519 (75.8)

166 (24.2)

Elevated systolic BP***

Yes

281 (33.8)

115 (40.9)

166 (59.1)

229 (81.5)

52 (18.5)

202 (71.9)

79 (28.1)

No

548 (65.9)

234 (42.7)

314 (57.3)

447 (81.6)

101 (18.4)

412 (75.2)

136 (24.8)

Elevated diastolic BP***

Yes

98 (11.8)

37 (37.8)

61 (62.2)

77 (78.6)

21 (21.4)

69 (70.4)

29 (29.6)

No

733 (88.1)

312 (42.6)

421 (57.4)

601 (82.0)

132 (18.0)

546 (74.5)

187 (25.5)

Elevated cholesterol***

Yes

173 (20.8)

60 (34.7)

113 (65.3)

137 (79.2)

36 (20.8)

118 (68.2)

55 (31.8)

No

621 (74.6)

276 (44.4)

345 (55.6)

514 (82.8)

107 (17.2)

473 (76.2)

148 (23.8)

Smoker

Smoker

117 (14.1)

48 (41.0)

69 (59.0)

85 (72.6)

32 (27.4)

79 (67.5)

38 (32.5)

Non-smoker

685 (82.3)

296 (43.2)

389 (56.8)

580 (84.7)

105 (15.3)

527 (76.9)

158 (23.1)

BMI >25 kg/m2

Yes

673 (80.9)

278 (41.3)

395 (58.7)

551 (81.9)

122 (18.1)

493 (73.3)

180 (26.7)

No

156 (18.6)

70 (44.9)

86 (55.1)

126 (80.8)

30 (19.2)

121 (77.6)

35 (22.4)

Study arm

Intervention

400 (48.1)

159 (39.8)

241 (60.3)

330 (82.5)

70 (17.5)

295 (73.8)

105 (26.3)

Control

432 (51.9)

190 (44.0)

242 (56.0)

349 (80.8)

83 (19.2)

321 (74.3)

111 (25.7)

Marital status

Married/living with partner

565 (67.9)

250 (44.2)

315 (55.8)

471 (83.4)

94 (16.6)

427 (75.6)

138 (24.4)

Single/widowed/separated/divorced

241 (28.9)

92 (38.2)

149 (61.8)

200 (82.9)

41 (17.1)

180 (74.7)

61 (25.3)

Age left school (years)

14 and under

337 (40.5)

123 (36.5)

214 (63.5)

280 (83.1)

57 (16.9)

244 (72.4)

93 (27.6)

15 to 17

335 (40.3)

147 (43.9)

188 (56.1)

279 (83.3)

56 (16.7)

257 (76.7)

78 (23.3)

18 and over

95 (11.4)

48 (50.5)

47 (49.5)

82 (86.3)

13 (13.7)

78 (82.1)

17 (17.9)

Occupation

Manual

281 (33.8)

110 (39.1)

171 (60.9)

227 (80.8)

54 (19.2)

200 (71.2)

81 (28.8)

Non manual

325 (39.1)

162 (49.8)

163 (50.2)

282 (86.8)

43 (13.2)

260 (80.0)

65 (20.0)

Characteristics of patients at baseline and of those with valid and invalid responses to GODIN, DINE fibre and DINE fat questionnaires

Rates of return of valid scores observed for participants of different gender, age, marital status, study arm, BMI, blood pressure, history of MI or diabetes, or smoking status were similar.

In relation to the DINE fibre questionnaire, a smaller proportion of participants in manual occupations (80.8%) than in non-manual occupations (86.8%) returned valid scores. There was a trend towards those who left school at an older age being more likely to return valid scores than those who left school earlier. More of those who reported being non-smokers returned valid fibre scores than smokers (84.7% v 72.6%), Fewer of those without a history of diabetes than of those with diabetes returned valid responses (82.9% v 75.9%). Rates of valid and invalid DINE total fat scores were similar to those for DINE fibre in relation to history of diabetes, total cholesterol, smoking status and occupation (Table 2).

Multivariate analysis of patient characteristics

Table 3 presents the results from the multivariate regression analyses. In the Godin score analysis, after controlling across the full set of patient characteristics, only a school leaving age of 18 or over was found to be statistically significant. The regression coefficient indicates a positive and statistically significant response effect (0.508; p = 0.047). In the DINE fibre analysis, a history of diabetes (−0.506; p = 0.039) and smoking (−0.682; p = 0.013) were both associated with negative and statistically significant response effects. Similarly, in the DINE total fat score analysis, a history of diabetes (−0.457; p = 0.047), smoking (−0.529; p = 0.045), and high total cholesterol (−0.455; p = 0.039) were associated with negative and statistically significant response effects.

<p>Table 3</p>

Instrument

GODIN

DINE fibre

DINE total fat

Patient characteristics**

Coefficient

p-value

Coefficient

p-value

Coefficient

p-value

*Total N = 832; numbers for some characteristics are less due to incomplete data; Ref refers to reference category for valid responses for each characteristic.

**Baseline data; ***Elevated SBP defined as systolic blood pressure >140 mmHg; elevated DBP defined as diastolic blood pressure >90 mmHg; elevated cholesterol defined as serum cholesterol >5.0 mmol/l.

Figures in bold are statistically significant.

Age

−0.010

0.238

−0.01

0.695

−0.006

0.616

Gender

Male

−0.044

0.801

0.174

0.440

−0.070

0.744

Female

Ref*

Ref

Ref

History of MI

Yes

0.040

0.790

−0.258

0.198

−0.235

0.208

No

Ref

Ref

Ref

History of diabetes

Yes

−0.037

0.854

−0.506

0.039

−0.457

0.047

No

Ref

Ref

Ref

Elevated systolic BP***

Yes

0.074

0.680

0.082

0.735

0.009

0.966

No

Ref

Ref

Ref

Elevated diastolic BP***

Yes

−0.318

0.218

−0.379

0.238

−0.330

0.267

No

Ref

Ref

Ref

Elevated cholesterol***

Yes

−0.283

0.140

−0.324

0.178

−0.455

0.039

No

Ref

Ref

Ref

Smoker

Smoker

−0.109

0.636

−0.682

0.013

−0.529

0.045

Non-smoker

Ref

Ref

Ref

BMI >25 kg/m2

Yes

−0.239

0.222

−0.005

0.983

−0.225

0.368

No

Ref

Ref

Ref

Study arm

Intervention

−0.159

0.293

0.141

0.488

−0.023

0.909

Control

Ref

Ref

Ref

Marital status

Married/living with partner

0.233

0.204

−0.228

0.366

0.025

0.911

Single/widowed/separated/divorced

Ref

Ref

Ref

Age left school (years)

14 and under

Ref

Ref

Ref

15 to 17

0.240

0.147

−0.004

0.987

0.024

0.905

18 and over

0.508

0.047

0.134

0.712

0.475

0.179

Occupation

Manual

Ref

Ref

Ref

Non manual

0.226

0.243

0.306

0.267

0.275

0.250

Multivariate analysis of characteristics of patients with and without valid GODIN, DINE fibre and DINE fat scores at baseline

Qualitative study data Of the 45 interviewees 2 declined to complete either a Godin or a DINE questionnaire due to time constraints. Of the remainder, 72.1% (31/43) had valid scores for the Godin questionnaire; 93.0% (40/43) had valid scores for DINE fibre; scores for total fat were not computed. Questions asked by participants related to advice as to whether certain activities could be counted as ‘exercise’, help in translating activities into 15 minute periods of different intensities of exercise, whether certain foods ‘fitted’ the categories presented in DINE and translation of their diet recall into numbers of servings per week. It was observed that some participants did not take time to read questions carefully or take account of different question construction requiring a different format of answer.

Discussion

Summary of main findings

These findings show that study participants with a lower level of education returned relatively fewer valid Godin scores for assessment of change in their self-reported physical activity. Relatively fewer valid DINE scores were returned by smokers, people with diabetes and total cholesterol >5 mmol/l.

The Godin questionnaire seemed to be challenging for respondents, all of whom had CHD: more than 50% returned questionnaires which did not allow a calculation of change of their level of physical activity. The DINE seemed to be more straightforward to answer, with almost all at baseline and 80% at follow-up giving responses which allowed calculation of change in their fibre intake.

Comparison with existing literature

Details of the methods of administration, response rates and valid completion rates of lifestyle behaviour questionnaires have often not been reported clearly in previous studies of behaviour change interventions. For example, Hunt-Shanks et al. 20 investigated the effect of exercise among cardiac patients by posting Godin questionnaires, but response rates were unclear. Trinh et al. 21 examined correlates of physical activity among kidney cancer survivors by posting a survey ‘package’, including a Godin questionnaire. Completed questionnaires were returned by 42.5% (703/1,654) which is similar to our findings of the rate of valid returns: they found no differences in age, sex or surgery rate between responders and non-responders but, in contrast to our report, they did not report if any incomplete or invalid responses were received.

Rosenburg et al. 27 used postal Godin questionnaires among a disabled population and commented that the examples of exercise were inappropriate but did not report clearly details of responses received. A reason for poor return of valid scores in our study may be that participants considered that the questions were not relevant to them because of impaired mobility. Evaluation of intervention effectiveness is only meaningful if the outcome measures are appropriate to the study population which, in secondary intervention studies, often includes older and multimorbid people, with cognitive or physical impairments. Involvement of patients from the target study population in all stages of research design planning and in interpreting findings should help to ensure that relevant outcome measures are chosen and measured appropriately.

Our finding that return of valid scores is associated with higher educational attainment is in keeping with findings of Cash et al. 28 who surveyed diet and physical activity behaviours among employees who were mostly high school graduates or higher. Godin scores were reported for approximately 80% of the sample but reasons for the full sample not being included are not given or discussed.

A previous study 6 which used a self administered Godin questionnaire, sent and returned by post, in testing the effectiveness of an intervention for patients with multiple sclerosis offered a $10 incentive for a completed return. The report suggests that 90.5% (19/21) completed questionnaires successfully. Thus, the offer of an incentive may boost response rates but there needs to be recognition that an incentive which is more attractive to some participants than others may also introduce a further source of response bias in evaluating an intervention.

In keeping with our findings, good response rates have been reported for DINE in previous studies. John et al. 23 mailed a DINE questionnaire to participants in a study of a healthy eating intervention before they attended appointments with a nurse. Whilst a 95% (655/690) response rate was reported, it is unclear if the DINE was reviewed during the appointment and it was noted that participants were of a higher socioeconomic class than the UK average. Steptoe et al. 29 also used DINE to examine the effect of eating behaviour counselling but their report did not state if patients completed this in the presence of a researcher or health professional from whom they could seek advice. At 12-month review it was completed by 80% of the original participants (218/271) but no comments were made regarding invalid responses. Further detail of methods of administration and completion of outcomes in these studies would be helpful.

Strengths and limitations

The SPHERE Study involved a large sample (n = 903) with opportunity to compare participants’ responses to self report questionnaires at baseline and follow-up time points. The completeness of follow-up (98.2%) allowed comparisons between those who did and did not return valid scores in relation to many different characteristics.

Administering the questionnaires by post without giving participants an opportunity to ask advice if they had uncertainty regarding any questions may be considered to be a weakness of the original study, which limited the possibility of maximising the number of valid responses. Analysing returned questionnaires, however, without making assumptions regarding responses which were not in strict accordance with instructions reflects accurately the way in which the questions were answered without assistance and without the possible influence of observer bias.

A further strength of this study is its integration of quantitative and qualitative data. The researchers’ observations of the process of questionnaire completion confirmed how study participants struggled to understand the content and construct of some questions and to respond appropriately. Some had difficulty following the format of the forms, illustrating how completion of questionnaires requires literacy and an ability to follow instructions correctly 30 and the relevance of our findings which associate higher educational status with the return of valid responses.

Implications

Our results suggest that research study participants may have difficulty in providing appropriate responses to the Godin and DINE questionnaires when these are posted and completed without access to support, resulting in a lower valid response rate than when administered face-to-face. Having someone available to give advice is not always practical because of time, travel and financial constraints and whilst it may increase the number of meaningful responses, completeness of evaluation and confidence in conclusions regarding intervention effectiveness, it may also introduce bias due to the influence of the source of advice. However, self-completion of questionnaires, with review by a health professional or researcher and revision of responses if indicated, may reflect a pragmatic approach which is relevant to real-world clinical settings. For example, some respondents may leave boxes blank when an appropriate response would be to record a zero, indicating no relevant food consumption or activity. Careful consideration may be given to the possibility of making assumptions to interpret some responses in postal questionnaires, to allow these to be included in analyses.

The importance of reporting information about valid response rates is indicated by our findings that those with higher levels of cardiovascular risk, associated with manual occupational status and lower educational attainment, with higher cholesterol levels and with diabetes were more likely to return invalid responses. Those who are aware of their higher level of risk may have feelings of guilt about their eating and physical activity behaviours which may influence valid completion of questionnaires. However, exclusion of such individuals from analysis of the outcomes of lifestyle change interventions has implications for conclusions regarding their overall effectiveness.

Our findings also illustrate the importance of ensuring that outcome measures used in evaluating interventions are appropriate for the target population. There is an increasing focus on developing health services that are patient centred 31 and patient reported outcome measures (PROMs) can be used to achieve this aim 32 . However, whilst these can be useful in monitoring the quality of care delivery 33 , potentially harmful results may arise from inappropriate methodology, relating to the design of measures chosen or the method of their administration and data collection, leading to evaluations which are applicable only to a skewed sample of the eligible population. Attention should be paid to patients’ literacy 30 and health literacy 34 . The readability of information and patients’ views are of paramount importance in developing management plans 35 and should be key determinants of the choice and design of appropriate PROMs, such as lifestyle questionnaires.

Conclusion

More detailed reporting of research methods and findings, with clear information about response and valid completion rates is important to inform the evaluation of interventions to change diet and physical activity behaviour. Close attention should be paid to the setting and context of studies which have been used to validate outcome measures chosen to evaluate healthcare interventions. Further research is required to determine which lifestyle questionnaires are most suitable and how they should be administered in order to maximise valid outcome data, particularly for populations such as older adults with chronic illness and those with socio-economic disadvantage or lower levels of education. The impact of methods of outcome measure administration and valid response rates in evaluating healthcare requires further study.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

JC collected data, carried out the analysis and drafted the manuscript. SS and MB were involved in the conception and conduct of the SPHERE Study and in revising successive drafts of this manuscript. PG was involved in data analysis and revision of drafts. AM conceived the SPHERE Study and was involved in intellectual review of the draft manuscript. MC was involved in the conception of the SPHERE Study, conceived the current study, and shared in the analysis and writing of successive drafts of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

This study was supported by funding from the Centre of Excellence for Public Health (Northern Ireland), Queen’s University Belfast. The SPHERE Study was funded by the Health Research Board, Ireland and the Irish Heart Foundation. We wish to thank the SPHERE Study team for their conduct of the SPHERE Study, Edel Murphy and Claire Leathem for their support of this study and staff at all general practices involved, for their help in participant recruitment.

<p>European guidelines on cardiovascular disease prevention in clinical practice</p>De BackerGAmbrosioniEBorch-JohnsenKBrotonsCCifkovaRDallongevilleJEbrahimSFaergemanOGrahamIManciaGCatsVMOrth-GomerKPerkJPyoralaKRodicioJLSansSSansoyVSechtemUSiblerSThomsenTWoodDEur Heart J2003241601161010.1016/S0195-668X(03)00347-612964575<p>Multiple risk factor interventions for primary prevention of coronary heart disease</p>EbrahimSTaylorFWardKBeswickABurkeMDaveySGCochrane Database Syst Rev2011191CD001561. doi:10.1002/14651858.CD001561.pub3<p>Systematic review of the effect of diet and exercise lifestyle interventions in the secondary prevention of coronary heart disease</p>ColeJASmithSMHartNCupplesMECardiol Res Pract2011Article ID 232351, http://dx.doi.org/10.4061/2011/232351<p>Challenges and opportunities for measuring physical activity in sedentary adults</p>Tudor-LockeCEMyersAMSports Med20013129110010.2165/00007256-200131020-0000211227981<p>Assessment of physical activity by self-report: status, limitations, and future directions</p>SallisJFSaelensBERes Q Exerc Sport2000712S1S1410925819<p>Increasing physical activity in multiple sclerosis: replicating Internet intervention effects using objective and self-report outcomes</p>DlugonskiDMotlRWMcAuleyEJ Rehabil Res Dev20114891129113610.1682/JRRD.2010.09.019222234717<p>Assessment of physical activity – a review of methodologies with reference to epidemiological research: a report of the exercise physiology section of the European Association of cardiovascular prevention and rehabilitation</p>WarrenJMEkelundUBessondHMezzanifAGeladasgNVanheesLEur J Cardiovasc Prev Rehabil20101712713910.1097/HJR.0b013e32832ed87520215971<p>Validation of a self-administered food-frequency questionnaire administered in the European Prospective Investigation into Cancer and Nutrition (EPIC) Study: comparison of energy, protein and macronutrient intakes estimated with the doubly labelled water, urinary nitrogen and repeated 24-h dietary recall methods</p>KrokeAKilpstein-GrobuschKVossSMosenederJThieleckeFNoackRBoeingHAm J Clin Nutr19997043944710500011<p>Reproducibility and validity of a semiquantitative food frequency questionnaire</p>WillettWCSampsonLStampferMJRosnerBBainCWitschiJHennekensCHSpeizerFEAm J Epidemiol1985122151654014201<p>Food-frequency questionnaires: a review of their design, validation and utilisation</p>CadeJEBurleyDLWarmRLMargettsBMNutr Res Rev200417152210.1079/NRR20037019079912<p>Design characteristics of food frequency questionnaires in relation to their validity</p>MolagMLde VriesJHMOckeMCDagneliePCvan den BrandtPAJansenMCJFvan StaverenWAvan’t VeerPAm J Epidemiol2007166121468147810.1093/aje/kwm23617881382<p>Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial</p>SahlqvistSSongYBullFAdamsEPrestonJOgilvieDBMC Med Res Methodol2011116210.1186/1471-2288-11-62311012121548947<p>The SPHERE Study. Secondary prevention of heart disease in general practice: protocol of a randomised controlled trial of tailored practice and patient care plans with parallel qualitative, economic and policy analyses</p>MurphyAWCupplesMESmithSMByrneMLeathemCByrneMCCurr Control Trials Cardiovasc Med200561110.1186/1468-6708-6-11120892916053525<p>Development of a complex intervention for secondary prevention of coronary heart disease in primary care using the UK medical research council framework</p>ByrneMCupplesMESmithSMLeathemCCorriganMByrneMCAm J Manag Care200612526126616686583<p>A simple method to assess exercise behavior in the community</p>GodinGShephardRJCan J Appl Sport Sci19851031411464053261<p>Dietary intervention in primary care: validity of the DINE method for diet assessment</p>RoeLStrongCWhitesideCNeilAMantDFam Pract19941137538110.1093/fampra/11.4.3757895964<p>The godin-shephard leisure-time physical activity questionnaire (Commentary)</p>GodinGHealth Fit J Canada2011412011<p>Comparison of activity levels using the Caltrac accelerometer and five questionnaires</p>MillerDJFreedsonPSKlineGMMed Sci Sports Exerc19942633763828183104<p>A simultaneous evaluation of 10 commonly used physical activity questionnaires</p>JacobsDRJrAinsworthBEHartmanTJLeonASMed Sci Sports Exerc1993251819110.1249/00005768-199301000-000128423759<p>Gender differences in cardiac patients: a longitudinal investigation of exercise, autonomic anxiety, negative affect and depression</p>Hunt-ShanksTBlanchardCReidRDPsychol Health Med200914337538510.1080/1354850090286693919444715<p>Correlates of physical activity in a population-based sample of kidney cancer survivors: an application of the theory of planned behaviour</p>TrinhLPlotnikoffRCRhodesRENorthSCourneyaKSInt J Behav Nutr Phys Act201299610.1186/1479-5868-9-96348987022866956<p>Effect of dietary advice to increase fruit and vegetable consumption on plasma flavonol concentrations: results from a randomised controlled intervention trial</p>HuxleyRRLeanMCrozierAJohnJHNeilHAWJ Epidemiol Community Health20045828828910.1136/jech.2003.014274173271515026440<p>Effects of fruit and vegetable consumption on plasma antioxidant concentrations and blood pressure: a randomised controlled trial</p>JohnJHZieblandSYudkinPRoeLSNeilHAOxford Fruit and Vegetable Study GroupLancet200235993221969197410.1016/S0140-6736(02)98858-612076551<p>Effect of tailored practice and patient care plans on secondary prevention of heart disease in general practice: cluster randomised controlled trial</p>MurphyAWCupplesMESmithSMByrneMByrneMCNewellJBr Med J2009339b422010.1136/bmj.b4220<p>Do practitioners and friends support patients with coronary heart disease in lifestyle change? A qualitative study</p>ColeJASmithSMHartNCupplesMEBMC Fam Pract20131412610.1186/1471-2296-14-126376593123984815HardinJWHilbeJMGeneralized Estimating EquationsBoca Raton, FL: Chapman & Hall/CRC2003<p>Self-reported depression and physical activity in adults with mobility impairments</p>RosenburgDEBombardierCHArtherholtSJensenMPMotlRWArch Phys Med Rehabil20139473173610.1016/j.apmr.2012.11.01423164977<p>Dietary and physical activity behaviours related to obesity-specific quality of life and work productivity: baseline results from a worksite trial</p>CashSWBeresfordSAAHendersonJAMcTiernanAXiaoLWangCYPatrickDLBr J Nutr201210861134114210.1017/S0007114511006258335939122142517<p>Behavioural counselling to increase consumption of fruit and vegetables in low income adults: randomised trial</p>SteptoeAPerkins-PorrasLMcKayCRinkEHiltonSCappuccioFPBr Med J200319326(7394):855<p>Effect of low medical literacy on health survey measurements</p>Al-TayyibAARogersSMGribbleJNVillarroelMTurnerCFAm J Public Health20029291478148010.2105/AJPH.92.9.1478144726412197979<p>Using health status to measure NHS performance: another step into the dark for the health reform in England</p>ValderasJMFitzpatrickRRolandMBMJ Qual Saf201221435235310.1136/bmjqs-2011-00018421949439<p>The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature</p>ValderasJMKotzevaAEspallarguesMGuyattGFerransCEHalyardMYRevickiDASymondsTParadaAAlonsoJQual Life Res200817217919310.1007/s11136-007-9295-018175207<p>Routine use of patient reported outcome measures in healthcare settings</p>DawsonJDollHFitzpatrickRJenkinsonCCarrAJBr Med J20103407744464467<p>The grounded psychometric development and initial validation of the Health Literacy Questionnaire (HLQ)</p>OsborneRHBatterhamRWElsworthGRHawkinsMBuchbinderRBMC Public Health20131365810.1186/1471-2458-13-658371865923855504<p>Using the opinions of coronary heart disease patients in designing a health education booklet for use in general practice consultations</p>LeathemCSCupplesMEByrneMCByrneMCorriganMSmithSMMurphyAWPrim Health Care Res Dev200910318919910.1017/S1463423609001224