Article Text
PDF
Abstract
Objective A sexual health survey among rural Zimbabwean youth was used to compare the validity and reliability of sexual behaviour measures between four questionnaire delivery modes.
Methods Using a random permuted block design, 1495 youth were randomised to one of four questionnaire delivery modes: self-administered questionnaire (SAQ=373); SAQ accompanied by an audio soundtrack (Audio-SAQ=376); face-to-face interview with sensitive questions placed in a confidential voting box (Informal confidential voting interview; ICVI=365); and audio computer-assisted survey instrument (ACASI=381). Key questions were selected a priori to compare item non-response and rates of reporting of sensitive behaviours between questionnaire delivery modes. Qualitative data were collected on perceived method acceptability (n=115).
Results Item non-response was significantly higher with SAQ and Audio-SAQ than with ICVI and ACASI (p<0.001). After adjusting for covariates, the odds of reporting sexual activity among Audio-SAQ and ACASI users were twice as high as the odds for SAQ users (Audio-SAQ AOR=2.05 (95% CI 1.2 to 3.4); ACASI AOR=2.0 (95% CI 1.2 to 3.2)), with no evidence of reporting difference between ICVI and SAQ users (ICVI AOR=1.0 (95% CI 0.6 to 1.8)). ACASI users reported a lower age at first intercourse and were more likely to report a greater number of partners (mean difference=1.06; 95% CI 0.33 to 1.78; p=0.004). They reported an increased ability to answer questions honestly (p=0.004) and believed their answers would be kept secret. Participants claimed increased comprehension when hearing questions while reading them. ICVI users expressed difficulty answering sensitive questions, despite understanding that their answers were unknown to the interviewer.
Conclusion ACASI appears to reduce bias significantly, and is feasible and acceptable in resource-poor settings with low computer literacy. Its increased use would likely improve the quality of questionnaire data in general and sexual behaviour data specifically.
- Adolescents
- sexual behaviour
- survey methodology
- Zimbabwe
- randomised controlled trial
- Africa
- qualitative research
- sexual behaviour
- STD
Statistics from Altmetric.com
- Adolescents
- sexual behaviour
- survey methodology
- Zimbabwe
- randomised controlled trial
- Africa
- qualitative research
- sexual behaviour
- STD
Introduction
Questionnaire data in general and sexual behavioural data in particularly are subject to both social desirability and recall bias.1–4 Measurement of sexual behaviour in young people is particularly problematic, as penalties for these behaviours can be severe (eg, expulsion from school or home). This results in a noted under-reporting of sexual behaviour, causing difficulty with data interpretation (particularly in adolescent girls; there is also limited evidence that boys may exaggerate their experiences).5–10 This under-reporting is especially important when behavioural measures are used to determine intervention effectiveness in reducing sexual risk taking. Behavioural measures may be used in adolescent populations where prevalence of biological endpoints such as HIV is too low and require too large a study population. Moreover, behavioural measures are also essential in the interpretation of biological endpoints.
Considerable comparative methodological literature on questionnaire delivery modes exists, generally indicating that self-completion computer-assisted modes improve completeness and data consistency and may increase disclosure of socially censured behaviours.11–14 However, few studies have explored this issue in developing countries15–19 and among young people.6 8 20–25 Those studies that have been conducted show different rates of reporting sensitive behaviours based on questionnaire delivery mode but with the direction of effect varying by study. So, while evidence exists regarding the importance of questionnaire delivery mode, more research is required to ascertain the appropriate mode for different resource-limited settings. Moreover, while several studies have compared modes across a population, there are relatively few studies that have examined its reliability by comparing different modes using the same participants (test–retest), particularly in young people.8 22 24
The Regai Dzive Shiri trial was a community randomised trial set in rural Zimbabwe that measured the effectiveness of an adolescent reproductive health programme in reducing rates of HIV, HSV2 and unintended pregnancy. Between January and April 2006, we nested a rigorous experimental evaluation of the validity and reliability of four questionnaire delivery modes within the trial's interim survey, the results of which are reported here. The primary objectives of the trial were to compare differences in the item non-response rates and prevalence of reporting of prespecified sexual behaviours between questionnaire delivery modes. A secondary objective was the comparison of reported prevalence of sexual behaviours when the same individual uses two different questionnaire delivery modes. Qualitative data explored participants' perceptions on the acceptability and feasibility of the questionnaire delivery modes.
Methods
Study population
In 2003, the Regai Dzive Shiri baseline survey was conducted in 30 rural communities in three provinces in eastern rural Zimbabwe.10 Eighty-six per cent (n=6791) of pupils in their ninth year of schooling attending 82 secondary schools in the study communities took part. In 2006, this interim survey was conducted in 12 (of 30) randomly selected study communities. Survey participants were eligible if they were cohort members who had participated in the baseline survey and were currently residing in these communities.
Four questionnaire delivery modes were compared. Self-administered questionnaires (SAQ) on paper were self-completed; all completion instructions, including those on skip patterns, were written into the questionnaire booklet. Participants using SAQ accompanied by an audio soundtrack (Audio-SAQ) completed the SAQ paper questionnaire while listening through headphones to an audio soundtrack on a CD player. Complex skip patterns that asked the participant to answer additional questions within a skip or to skip backwards were eliminated from this version, as pilot testing revealed that it was difficult for respondents to keep the audio recording in sync with the paper version of the questionnaire. In a version adapted from Gregson et al (termed informal confidential voting interviews (ICVI)), a trained interviewer administered the questionnaire in a standard format, and responses to sensitive questions were privately recorded by the participant onto a bar-coded ballot sheet, which they posted into a locked ballot box.15 Participants using audio-computer-assisted survey instrument (ACASI) completed the questionnaire on a laptop computer while listening through headphones. The participant used a mouse to click on their chosen response. Skip patterns were preprogrammed. As electricity was not available in rural Zimbabwe, laptops were recharged using a truck battery connected to a solar panel. Modes that included an audio soundtrack (Audio-SAQ and ACASI) used gender-specific voices (eg, males heard a male voice) trained to read in a non-judgemental and empathetic manner.
Questionnaire development
The first questionnaire covered a wide range of domains (Round 1). A second, shorter version was created for a follow-up interview (Round 2); while some questions were omitted, question phrasing and order remained constant. A practice session was designed for each mode and run individually with each participant prior to their commencing the questionnaire.
Randomisation to one of four delivery modes and questionnaire completion
Using a random permuted block design, cohort members were randomised to one of four questionnaire delivery modes: SAQ, Audio-SAQ, ICVI and ACASI in Round 1, 43% of whom were randomised to complete the shortened questionnaire (Round 2), using one of the same four modes. These 20 different permutations which accounted for method completion and order (see figure 1) were assigned a code placed into sequentially numbered opaque envelopes in block and number order. Following written, informed consent, each participant selected the next envelope. Participants assigned to complete the questionnaire on two occasions returned to the survey site 1 week later to complete their Round 2 version.
Flow chart showing participation in the interim survey.
Additional data collection
Qualitative data from 115 randomly selected study participants and two gender-mixed focus-group discussions, purposively sampled to reflect all modes, described mode acceptability and feasibility. All qualitative data were transcribed electronically and coded for thematic issues using NVivo 7.0 (QSR, Melbourne, Australia).
Sample size considerations
Statistical power was estimated using the power simulation approach.26 The hypothesis is that at least one questionnaire delivery mode would be different from the other three modes in terms of reporting ever having had sex. For a baseline of 15% prevalence of reporting having had sex (estimated using data collected at baseline and from the Zimbabwe Young Adult Survey27), with 350 participants per group, the study has 85% power to detect a RR of 0.5 (or 2.0) in at least one group.
Data handling and statistical analysis
Data were entered into an MS Access database. Questionnaire data collected on paper (SAQ, Audio-SAQ and ICVI) were coded by trained data clerks prior to data entry (10% double-entered). ACASI data were downloaded from each laptop and imported directly into the database. Range and consistency checks were performed.
To compare the prevalence of self-reported sexual behaviours across the four delivery modes, we calculated item non-response rates and rates of reported sexual behaviours. Global item non-response was defined as the proportion of questions applicable to all participants but not answered. The mean number of non-responses was calculated. ‘I don't know’ was considered a valid response. In addition, rates of item non-response and reporting of sexual behaviours were assessed for specific preselected sexual behaviours: ever been kissed, ever had sex, age at first intercourse, condom use at last intercourse and lifetime number of sexual partners. Non-response rates were calculated as the proportion not answering each question (excluding where the question was not applicable).
Participants were considered to be clustered in communities; as such robust standard errors were used in all parallel group comparisons between the methods to allow for community effects. For binary variables, ever kissed, ever had sex and condom use at last intercourse, the reported prevalence (%) of each behaviour was calculated for each questionnaire delivery mode, and generalised linear models with binomial distribution were used to obtain 95% CIs around each parameter estimate. For continuous variables, age at first intercourse and number of lifetime partners, the mean of each was calculated for each delivery mode and 95% CIs estimated. Multivariate logistic regression was used to compare the proportions of each reported behaviour between the four delivery modes. Users of Audio-SAQ, ICVI and ACASI were compared with users of SAQ as the reference group. The robust Wald test was used to test the null hypothesis that the mode of questionnaire delivery had no effect on reporting each behaviour; p values for each test were reported. Analyses were adjusted for the following covariates, which were either determined a priori as factors that might influence the reporting of sexual behaviours or showed some imbalance between the four delivery modes: gender, education, orphaning, marital status, HIV and HSV2 status, and pregnancy (females only).
Given young people's difficulty in acknowledging their sexual behaviour, respondents who initially reported never having had sex were asked this question a second time. This was feasible in three of the four modes: SAQ, Audio-SAQ and ACASI. As this is the entry point for all other sexual behaviour questions, a participant's second response was considered the more accurate response. Changes between the first and second responses were recorded and compared by mode.
The answer to the question ‘Have you ever had sex?’ was compared between Round 1 and Round 2. The McNemar χ2 test was used to test whether individuals who changed their answer to this question between Round 1 and Round 2 did so differently dependent on the mode they used for completion of the Round 2 questionnaire—that is, discordant answers were compared between all modes in Round 1 and each mode in turn in Round 2. Those who had completed the same mode in both rounds were excluded from this analysis.
There were 20 possible skip patterns included in the questionnaire (mode-dependent). Data on successful skip completion from SAQ and Audio-SAQ (where participants were responsible for making the skip) were analysed. This was defined as a participant answering the next appropriate question after the skip was established.
All statistical analyses were performed using Stata 9.
Results
The 1495 participants who took part in the interim survey (males=827; females=668; mean age 18.2 years, range 15–23) represented 54% of those eligible overall but 96% of those eligible who were currently living in study communities (n=1557) (see figure 2). When baseline characteristics were compared between participants and non-participants, interim survey participants were more likely to be male than non-participants (55% vs 48%; p<0.001), were younger (26% <15 years vs 20%; p<0.001) and less likely to be orphaned (64% vs 66%; p=0.016) but were as likely to be sexually active (6% vs 8%; p=0.088).
Block randomisation of four questionnaire delivery methods, Round 1 and Round 2. ACASI, audio-computer-assisted survey instrument where respondent enters their response using a computer after hearing questions and response options through headphones; Audio-SAQ, self-administered questionnaire accompanied by an audio soundtrack; ICVI, informal confidential voting interview where a trained interviewer administers the majority of the questionnaire, but sensitive questions are recorded by a respondent onto a sheet placed in a locked ballot box; SAQ, self-administered questionnaire using paper and pen.
Participants' demographic characteristics are presented in table 1. Just over 4.3% (95% CI 3.3 to 5.3%) of participants were married (7.6% of females and 1.6% of males), and 86.6% (95% CI 84.8 to 88.3%) had completed at least 4 years of requisite secondary school. Overall, 40.0% (95% CI 37.5 to 42.6%) reported being orphaned. While there was a reasonable balance between the four arms of the study, there was some variation in orphanhood and educational status between arms.
Characteristics of interim survey participants by survey method (N=1495; males=827, females=668)
Item response rates by questionnaire delivery modes
The global item non-response varied significantly between the four modes (p<0.001), with the lowest number of non-responses achieved by those using ICVI (mean=2.2 questions missed; 95% CI 1.9 to 2.5) and the highest number achieved by SAQ users 8.0 (95% CI 7.2 to 8.9).
Participants only answered in-depth sexual behavioural questions if they had given a positive response to the initial question about their sexual activity. However, not all participants performed this task correctly, particularly those using SAQ and Audio-SAQ, as 14.0% and 12.0% respectively failed to answer the next appropriate question, compared with 2.0% of ICVI users and 0.0% of ACASI users (p=0.006). Moreover, for skip patterns involving less sensitive questions (eg, marriage and earning money), 100% of SAQ participants completed this skip instruction successfully (no skips in Audio-SAQ).
Differences in response rates cannot be attributed solely to failing to follow skip instructions. While not statistically significant (p=0.074), SAQ users were less likely than those using other modes to answer the potentially sensitive but non-skip-dependent question on kissing (see table 2A).
Non-response rates and reported prevalence (95% CI) of sexual behaviour by survey methods in parallel group comparison
Reporting sexual behaviours
Differences between modes of rates of reporting sexual behaviours are shown in table 2B. There is some evidence that Audio-SAQ and ACASI users were more likely to report sexual behaviours than those using other modes. For example, there was weak evidence that participants using Audio-SAQ and ACASI were more likely to report having kissed someone (p=0.051) and to report having had sex (p=0.068) than participants using the other two methods. After adjusting for covariates, the odds of reporting ‘ever had sex’ among Audio-SAQ and ACASI users were twice as high as the odds for SAQ users (Audio-SAQ: unadjusted OR AOR=2.0 (95% CI 1.2 to 3.4); ACASI AOR=2.0 (95% CI 1.2 to 3.2)), with no evidence of reporting difference between ICVI and SAQ users (ICVI AOR=1.0 (95% CI 0.6 to 1.8)) (table 3).
Odds of proportion reporting sexual behaviours by survey methods
Audio-SAQ and ACASI users also reported higher odds of having kissed someone than SAQ users (Audio-SAQ and ACASI AOR=1.7; 95% CI 1.1 to 2.6); ICVI users reported similar odds of this behaviour to those of SAQ users (ICVI AOR=1.0; 95% CI 0.6 to 1.8 (table 3). It did appear that ICVI and Audio-SAQ users were more likely to report a slightly older age of first intercourse than those using the other delivery modes, although the mean difference in age between the delivery methods was not statistically significant after adjusting for covariates (p=0.08) (table 4). However, there was evidence that ACASI users reported a higher number of partners than SAQ users (mean difference=1.1; 95% CI 0.3 to 1.8; p=0.016); there was no significant difference in the number of reported partners between ICVI, Audio-SAQ and SAQ users (table 4). When participants using SAQ, Audio-SAQ and ACASI who initially reported not being sexually active were asked this question a second time ACASI users were more likely to change their response to ‘yes’ (18/334; 5.4%) than Audio-SAQ (10/319; 3.1%) or SAQ users (4/275; 1.4%) (p=0.037).
Mean differences in reporting age at first intercourse and number of partners by survey methods
Difference between response to questions answered in Round 1 and Round 2
A total of 395 of the 641 participants randomly allocated to complete the questionnaire twice did so. There were no differences in demographic characteristics between those who completed and those who failed to complete Round 2. Participants were more likely to have attended Round 2 if they were from a control community (control=71.8% vs intervention=51.1%; p<0.001).
The number of changes (by direction) between rounds by questionnaire mode are presented in table 5A,B. Of the 331 participants who reported never having had sex in Round 1, 21 (6.3%) reported having had sex in Round 2. This happened more commonly among those using ACASI (n=14) in Round 2. Of the 52 participants who reported ‘yes’ to having sex in Round 1, 12 (23.1%) changed their answer to ‘no’ in Round 2. This happened more commonly in those who used SAQ (n=7) or Audio-SAQ (n=4) as their Round 2 mode. None of the respondents who used ACASI in Round 2 changed their response from ‘yes’ to ‘no.’
Response conversions between Rounds 1 and 2 (asked 1 week later)
The McNemar test examining these discordant pairs showed that despite no statistical evidence of mode effect with ICVI or Audio-SAQ in Round 2 (p=1.0 for both), there was weak evidence that responses converted negatively with SAQ (p=0.07). Although numbers were small, responses were more likely to be converted positively when ACASI was used in Round 2 (p=0.004) with ACASI users reporting ‘yes’ to ever having had sex in Round 2 increasing by 12.7% (95% CI 3.5 to 21.8).
Qualitative data results
Of 115 qualitative interviews, 23% were with participants who had used two different modes. Emerging core themes focused on the importance of privacy, ease of method use and interviewer presence, especially in relation to sensitive questions and questionnaire comprehension. Overall, participants stressed the importance of completing the questionnaire ‘on their own’ to ensure that others could not see their responses (see table 6, point 1). ICVI users attributed their sense of privacy to the interviewer encouraging them to cover their ballot sheet and the opportunity for them to place it into the locked ballot box (see table 6, point 2).
Qualitative data from interviews and FGDs
Many ACASI users perceived their mode as more private and confidential because the study ID was not linked to their answers on the screen (see table 6, point 3). ACASI users reported a greater likelihood of answering questions on sexual activity honestly. A minority expressed concern that their responses on the computer screen might be visible to others (see table 6, point 3).
Participants using Audio-SAQ, ACASI or ICVI attributed their ease of understanding to having the questions read aloud (see table 6, point 4). Participants who could compare an anonymous voice (Audio-SAQ, ACASI) against the interviewer's (ICVI) credited their truthful answers to the lack of an interviewer's presence. While a few highlighted the usefulness of seeking instant clarification from the interviewer (something they reported failing to do when using another mode), they predominantly reported feeling embarrassed hearing sexual terms read aloud or responding to sensitive questions with an interviewer (see table 6, points 5 and 6).
Participants also noted that questions on poverty and orphaning were as sensitive as those on sex (see table 6, point 7). ICVI users felt this most acutely (see table 6, point 7). During debriefing sessions, interviewers (trained to handle emotionally charged situations) reported participants breaking down and looking uncomfortable when asked about poverty and orphaning.
Discussion
This study is one of the few experimental studies from a developing country to assess validity and reliability of data relating to health outcomes collected using four different questionnaire delivery modes. Importantly this study shows an appreciable and statistically significant difference between modes in terms of global and item non-response rates and rates of reporting sexual behaviours. Owing to the small numbers of participants with biological evidence of having had sex (as determined by incident HIV, HSV-2 antibody or pregnancy) we were not able to examine an association between self-reports and biological outcomes (data not shown). We found that SAQ (the cheapest and logistically simplest mode) had the worst item non-response rates and lowest levels of global non-response, and this resulted in the lowest rates of reporting of sexual behaviours. While adding an audio soundtrack to SAQ (Audio-SAQ) improved reporting, presumably through a greater sense of privacy, item and global non-response rates remained unacceptably high. Unsurprisingly, ICVI and ACASI had low item non-response rates; in both modes, the participant was externally prompted (interviewer or computer). As others have found,15 having a ballot box increased participants' sense of privacy and control over their responses. Nonetheless, ICVI had lower levels of reporting socially sensitive behaviours which was not limited to sexual behavioural questions but included questions on poverty and mental health.28 ACASI users were more likely to report ever having been kissed and ever having had sex, and to report more sexual partners than participants completing the questionnaire using SAQ or ICVI. There was no difference between methods on reporting age at first intercourse. Therefore, while Audio-SAQ and ACASI have similar high rates of reporting sexual behaviours, ACASI is the more useful mode because it allows the researcher to use skip patterns (essential for sexual partner matrices). In addition, ACASI allowed those respondents who initially answered that they had never had sex to be asked this question easily a second time, with a consequent increase in reporting. When reporting was compared between two rounds of questionnaire delivery (table 5), a few ACASI users (but fewer than for other modes) changed their responses to sex from ‘no’ to ‘yes’ underscoring the high level of sensitivity around reporting sexual behaviours while re-emphasising that ACASI is not the panacea. While direction of effect by gender was not explored here, when we explored this in another study, no variation in effect by gender was found.29
Although the initial outlay for ACASI was greater and proved logistically difficult to administer, its advantages include not relying on highly skilled interview staff for successful implementation. Direct data entry into the laptop also reduces data-entry errors.
While there is substantial literature comparing reports of sexual behaviour between modes in developed countries11–14 and with immigrant populations,30 this research has only recently been extended to resource-limited settings,17–19 and to young people in particular.6 8 20–25 31 Previous studies which have largely compared ACASI against trained interviewers have expressed some scepticism with the ability of youth with limited literacy to use a laptop.20 22 This study provides encouraging evidence that rural youth with little or no experience of computers are able to answer questions using a laptop, albeit from a country with very high levels of secondary education.24 31 Ensuring adequate time for individual training was essential to each mode's success.
Due to out-migration, we experienced a lower response rate than expected. However, as 96% of all cohort members still living in study communities took part, we believe our survey remains representative of young people living in these rural communities. Of note, none of the participants in the interim survey declined to participate in the randomised comparison of questionnaire delivery modes.
There are two, not necessarily mutually exclusive, explanations for the lower item response rate among participants using SAQ and Audio-SAQ: reluctance to answer socially sensitive questions and/or inability to negotiate skip patterns. For SAQ, there is some evidence that these participants were less likely to answer socially sensitive questions—for example, fewer SAQ participants reported ever being kissed (a non-skip-dependent question) than other mode users. Moreover, despite all survey participants having encountered skip patterns in their primary school exams, there remains a substantial minority of SAQ and Audio-SAQ participants who failed to negotiate skips relating to sexual behaviour successfully. Additionally, SAQ users were less able than Audio SAQ users, suggesting social desirability bias (Audio-SAQ participants were 2.6 times less likely to miss (or ignore) the sexual activity questions than SAQ participants). In contrast, skips relating to non-sensitive questions such as marriage and earning money were almost always negotiated successfully.
The trial also compared responses using different modes within an individual (Round 1 and Round 2). We found increased reporting of sensitive behaviours in Round 2 with ACASI and a decrease with SAQ. As question order and wording were identical, we do not believe that question omission can solely account for this variation.
Qualitative data highlighted the importance of hearing the questionnaire read aloud, both in reinforcing comprehension of little used sexual terminology and by increasing a sense of privacy by using earphones. In Zimbabwe, similar to other developing-country settings where indigenous sexual terminology is limited and not usually seen in print or spoken aloud,32 33 including the audio component reinforced those concepts. Qualitative data also underscored how the use of earphones added a sense of privacy by allowing participants to be in ‘their own world.’ Participants reported that ACASI gave them an added sense of privacy over and above Audio-SAQ in that the ID number was hidden from view and that each question appeared singularly on the screen. Many of those interviewed expressed discomfort around disclosing sensitive behaviours to interviewers, even when their answers were recorded privately using a ballot box. While including the audio component has, in the past, meant relying on interviewer-administered questionnaires, technological advances (MP3 players, palm-top recorders, ACASI) allow us to benefit from an anonymous voice and improve a participant's ability to answer sensitive questions honestly.
In summary, this experimental evaluation supplemented by qualitative data suggests that ACASI offers significant advantages over the other three modes including a lower non-response rate, better completion of skip patterns and higher rates of reporting sensitive data. While Audio-SAQ also had higher rates of reporting, it had much lower rates of question completion and poor success with skip patterns. Furthermore, we have demonstrated that ACASI is feasible and acceptable, even in resource-poor settings with low levels of computer literacy and where power supplies are limited. An increased use of ACASI could improve the quality of sexual-behaviour data collection in these settings.
Key messages
The questionnaire delivery mode influences the extent of social desirability bias found in self-reported data.
Technological advances in computer-administered questionnaire delivery modes have not been widely researched in developing country settings which continue to rely heavily on interviewer-administered questionnaires.
This experimental evaluation of four questionnaire delivery modes suggests that computerised self-administration in the form of audio computer-assisted survey instrument (ACASI) is feasible and acceptable in resource-limited settings with low computer literacy.
ACASI improves the validity of sensitive data by minimising item non-response and making it easier for respondents to report sensitive data.
References
Footnotes
Trial registration: The Regai Dzive Shiri trial was registered retrospectively; the trial number is ISRCTN70775692—Randomised trial of HIV/sexually transmitted infection (STI) prevention in Zimbabwean youths.
Funding This study was supported by the US National Institute of Mental Health (an initial grant R01 MH66570-01, supplemented by R01 MH066570-04S2). The research reported here was conducted independently from the funders, who had no input in this manuscript.
Competing interests None.
Ethics approval Ethics approval was provided by the Medical Research Council of Zimbabwe and the ethics committees of University College London and the London School of Hygiene and Tropical Medicine.
Provenance and peer review Not commissioned; externally peer reviewed.