Effects of two telephone survey methods on the level of reported risk behaviours
- Community Research Program on AIDS, Centre for Epidemiology and Biostatistics, Faculty of Medicine, The Chinese University of Hong Kong
- Correspondence to:
Professor Joseph T F Lau, Centre for Epidemiology and Biostatistics, Faculty of Medicine, The Chinese University of Hong Kong, 5/F, School of Public Health, Prince of Wales Hospital, Shatin, NT, Hong Kong;
- Accepted 31 January 2003
Objective: Reporting bias due to social desirability is an important consideration in carrying out surveys on sensitive issues. The study compared the frequency of self reported sensitive behaviours and response rates between the conventional “telephone interviewer method” (TIM) and a combined interviewer and computerised data capturing method (telephone interviewer and computerised questionnaire method, or TICQM).
Methods: A total of 580 males and 582 females were recruited and randomly assigned to either of the two methods in a cross sectional study. The overall response rate was about 51.6%.
Results: While the two methods both had high completion and low item non-response rates, the TICQM respondents reported higher frequencies of sensitive risk behaviours. Sexually active female respondents interviewed by the TICQM were more likely to report that their sex partners were their steady boyfriend, instead of their husband; and were also more likely to admit that they had had “one night stand” experiences or had undergone an HIV antibody test, when compared with their counterparts in the TIM group. Similar contrasts were observed for sexually active male respondents, that the TICQM group were more likely to report that they had had sexual intercourse with female sex workers or non-regular sex partners. Sex differences in the strength of association were observed between some studied behaviours (for example, HIV testing and substance abuse) and modes of data collection.
Conclusion: The choice of data collection method has a significant impact on the results of sensitive studies; special attention should be given to designing the study and interpreting the results.
Methodological issues such as participation bias, reporting bias, and effects of modes of administration on respondents’ responses are of critical consideration in the study of sexual behaviours and other sensitive topics. Respondents tend to respond in socially desirable ways, especially in face to face interviews.1 Hence, data collection methods that ensure anonymity and confidentiality are of great importance.
There are some studies addressing the influence of different administration modes on sexual behaviour reporting.2–,8 Some found that the use of audio-questionnaires resulted in fewer missing responses and higher frequency of reported risk behaviours than self administered written questionnaires.2 Others found that audio computer interviewing methods resulted in higher frequencies of risk behaviours than face to face interview.3 The use of a computerised questionnaire data collection method in combination with the use of mobile phone was also found to be reporting higher frequencies of risk behaviours.5
As telephone interviews have frequently been used to study sexual behaviours,9–,12 an innovative data collection method which can reduce reporting bias of telephone surveys would have important implications. This study aimed to compare the frequencies of HIV related attitudes and risk behaviours as reported by two modes of telephone survey methods. The proportion completing the questionnaire and the number of item non-responses were also compared. The study tested the hypothesis that the results of HIV related studies, which examine sensitive risk behaviours, are in fact affected by the mode of data collection.
The target population consisted of Chinese speaking male and female adults (aged 18–60) living in Hong Kong. In total, 1162 respondents were recruited, among which 580 were male and 582 were female.
The two methods under comparison
The first method of the study was the conventional “telephone interviewer method” (TIM), which had the entire questionnaire (consisting of two parts, see below) administered by interviewers; the second method (the “telephone interviewer and computerised questionnaire method” or TICQM) used exactly the same questionnaire, but while part I of the questionnaire was being administered by the interviewers, part II used an interactive computerised method instead. The computerised questionnaire was presented in a female voice.
Four experienced interviewers (three females and one male) were recruited for the study. All of them had at least one year of experience in implementing telephone surveys of HIV related studies. Half of the respondents were randomly allocated to one of the two methods (TIM or TICQM). Each interviewer completed the same number of questionnaires using the two different methods. The potential confounding effect caused by differences of individual interviewers and sex was hence removed by the study design.
Telephone numbers were randomly selected from up to date residential telephone directories, stratified by the three major geographic regions of Hong Kong. Most of the Hong Kong residents have telephones at home (Hong Kong Office of the Telecommunications Authority, personal communication, 28 January 2000). The study was carried out from June to July 2001. All interviews were conducted between 6 pm and 10.30 pm in order to avoid oversampling of unemployed people and housewives. Unanswered telephone numbers were attempted at least three times before being classified as invalid.
Of each sampled household a member, aged 18–60 and whose past birthday was closest to the day of the interview, was invited to participate in the study. All respondents were invited by the interviewers to join the study with the same greetings and briefing. Prospective respondents were briefed that the study was going to collect opinions about HIV/AIDS prevention and it was sponsored by the Hong Kong Council for the AIDS Trust Fund, and that their contribution would help government’s policy formulation.
For those who agreed to join the study, irrespective of which method group they were being randomly assigned, they were asked the same non-sensitive questions (part I of the questionnaire). After completing part I, they were invited to join part II of the study which asked about sensitive personal questions. The phrasing of the invitation to join part II was the same for respondents of the two method groups. However, for the TIM group, respondents went on to be questioned by the interviewers; while for the TICQM group, respondents were told that the questions were prerecorded in a computerised phone system (the “dot line” service), and that they only needed to key in their responses. They were then connected to the telecommunication service in Hong Kong via a conference line service (allowing one of the two parties to call a third party without hanging up and allowing all three parties to have simultaneous phone conversation) after the briefing. The interviewer left the line after the connection was made. Respondents then completed the computerised questionnaire by keying in their responses after listening to the prerecorded questions. The “dot line” telephone number was not released to the respondents. The service has often been used by TV stations for public opinion polling and is widely known to the public. All respondents were guaranteed that their telephone number would not be recorded. Anonymity and confidentiality were therefore enhanced. The TICQM has been employed in a number of local studies.9,13–,15 A literature search showed that this method had been original.
Out of all telephone calls made, 23.8% were non-contacts (after three attempts) and about 20% were non-households. Among those contacted households (these households with someone answering the call), 40% were invalid households (without or could not contact an eligible respondent), 29% refused to join the study, and 31% completed the study. An overall response rate, defined as the number of respondents completing the study divided by the number of completed interviews plus refusals, was 51.6%. This is comparable with other telephone surveys conducted in Hong Kong.9,14,16–,18 On average, it took the TICQM respondents about 7–8 minutes and the TIM respondents about 5–6 minutes to complete part II of the questionnaire.
Two structured questionnaires, one for males and one for females and each consisting of two parts, were used. Part I but not part II of the questionnaire was common to both the male and female respondents. Part I asked about non-sensitive questions; part II asked sensitive questions on sexual risk behaviours and HIV/AIDS related topics.
Respondents were asked to mention three routes of HIV transmission and their perception on condom efficacy (“What is your perceived efficacy of condom use for HIV prevention?”). Data on age and education level were also collected.
In the female questionnaire, respondents were asked if they had had any sex partner in the past 6 months. Only those who answered that they had had at least one sex partner in the past 6 months went on to answer the subsequent questions, including whether they did have a husband in the past 6 months; and if so, whether they had had sexual intercourse with their husband in the past 6 months, whether they perceived that they were able to insist on using condoms if they suspected that their husband had patronised female sex workers (FSW) in the past 6 months, and their perceived chance of contracting HIV from their husband. A similar set of four questions with respect to steady boyfriend (including cohabiting boyfriend) was also asked. In addition, respondents were asked whether they had an experience of a “one night stand” in the past 6 months, whether they had contracted an STD, had undergone an HIV antibody test, and had indulged in substance abuse in the past 6 months. A total of 13 questions were asked.
In the male questionnaire, respondents were asked about whether they had patronised FSW in the past 6 months; if yes, whether they had always used (that is, used every time) condoms when having sexual intercourse with an FSW. Respondents were also asked if they had regular and non-regular female sex partners in the past 6 months. If so, whether they had always used condoms when having sexual intercourse with these partners, respectively. In addition, they were asked whether they had ever had sexual intercourse with another man, whether they had contracted an STD, had undergone an HIV antibody test, and had indulged in substance abuse in the past 6 months. They were also asked about their perceived susceptibility to HIV infection. A total of 11 questions were asked.
Separate statistical analyses were performed for the two sexes as two different part II questionnaires were used. The distributions of the studied variables were tabulated by the two methods. The χ2 test and logistic regression analysis adjusted for age and education level (adjusted odds ratios (adjusted OR) and their respective 95% confidence interval (CI)) were performed to examine the statistical significance of the differences between the two methods. SPSS for Windows (version 11.0) was used for statistical analyses and p <0.05 was considered statistically significant.
Percentages completing different parts of the questionnaires
All respondents (of both methods) who started part I completed that part of the study. There were a few (male, TIM: 2.1% and TICQM: 1.8%; female, TIM: 2.7% and TICQM: 1.7%) respondents who completed part I of the study but refused to join part II of the study. Among the TICQM respondents, about 1–2% (male, 1.0%; female, 2.1%) of them did not start part II because of technical problems. Less than 1% (TIM: 0.4%; TICQM: 0.7%) of the male respondents started but did not finish part II; similar figures were 0.4% (TIM) and 4.6% (TICQM) for female respondents.
In sum, the majority of the male respondents who started part I finished the entire questionnaire (97.6% and 96.6% respectively for the TIM and TICQM); for female respondents, similar percentages were 96.9% (TIM) and 91.8% (TICQM), respectively. In other words, once the respondents started the interview, most of them were likely to complete the entire questionnaire. Among females, however, the incompletion rate of the TICQM was about 4% higher than that of the TIM.
For both methods, there were only very few missing responses for individual part II questions. When TIM respondents did not answer a certain question, it was recorded by the interviewer; whereas for TICQM respondents, no data were entered into the system. Altogether, the TIM group only had eight cases of non-response and the TICQM group only had four of such cases.
Background characteristics of respondents
Responses to part I questions on age, education level, HIV related knowledge (HIV transmission routes), and perceived efficacy of condom use for HIV prevention are summarised in table 1⇓. No statistically significant differences were observed on these variables when the two methods were compared. Therefore, respondents to the two methods were comparable in general and these two methods were giving similar results for non-sensitive questions.
Frequencies of self reported risk behaviours, HIV testing, and attitudes among female respondents
About the same percentages of female respondents to the two methods reported that they had had at least one sex partner in the past 6 months (TIM: 64.8% and TICQM: 65.5%, χ2 test, p = 0.849; table 2⇓). However, over 90% (91.8%) of these sexually active women in the TIM group said that they had had sexual intercourse with their husband (while only 72.0% of the TICQM group said so) and only 5.5% of the TIM group said that they did not have a husband (compared to 23.4% of the TICQM group); 91.2% of these sexually active women of the TIM group said that they did not have a steady boyfriend and only 8.2% admitted that they had had sexual intercourse with their boyfriend. On the other hand, only 47.4% of the TICQM group said that they did not have a steady boyfriend and 40.6% of this group reported that they had had sexual intercourse with their boyfriend (table 2⇓).
In other words, although female TIM respondents were equally likely to state that they were sexually active, it was much more likely for such sexually active TIM respondents than sexually active female TICQM respondents to indicate that their sex partner was their husband rather than their boyfriend. Sexually active women in the TIM group were less likely than those in the TICQM group to report being unmarried but having had sexual intercourse with a steady boyfriend (4.9% in TIM versus 18.3% in TICQM, χ2 test, p <0.001; adjusted OR = 0.22, 95% CI: 0.09 to 0.53, p = 0.001; data not tabulated).
Further, of the sexually active women, 4% in the TICQM group admitted that they had had a “one night stand” experience in the past 6 months but none of the TIM group reported such behaviour (χ2 test, p = 0.006; table 2⇑). A similar observation was obtained for the response to the question asking whether they had indulged in substance abuse in the past 6 months (2.9% in TICQM versus 0.0% in TIM, χ2 test, p = 0.022; table 2⇑). As well, more female TICQM respondents reported having undergone an HIV antibody test in the past 6 months compared to female TIM respondents (10.9% in TICQM versus 1.7% in TIM; adjusted OR = 7.51, p = 0.002; table 2⇑). Female TICQM respondents were more likely to perceive some/high chance of contracting HIV from their husband than female TIM respondents (30.2% in TICQM versus 19.4% in TIM, adjusted OR = 1.81, p = 0.033; table 2⇑). For those female TICQM respondents who suspected their husband had patronised an FSW, they were less likely to perceive themselves as being able to insist on using condoms with their husband under such circumstances (TICQM: 60.3% and TIM: 73.5%, adjusted OR = 0.53, p = 0.015; table 2⇑).
Three other items did not reach statistical significance when the TICQM and the TIM groups were compared. They included the prevalence of contracting an STD in the past 6 months, perceived chance of contracting HIV from their steady boyfriend, and perceived capacity to insist on using condoms if suspecting their steady boyfriend had patronised an FSW (see table 2⇑).
Frequencies of self reported risk behaviours, HIV testing, and attitudes among male respondents
As seen from table 3⇓, the male TICQM group also reported higher frequencies of some sexually related behaviours than the male TIM group. These included higher frequencies for having had sexual intercourse with an FSW in the past 6 months (adjusted OR = 3.36, p = 0.001) and having had a non-regular sex partner in the past 6 months (adjusted OR = 3.18, p = 0.004). Similar to the findings obtained from the female respondents, sexually active male TIM respondents were more likely to state that they had had a regular sex partner in the past 6 months, compared to their TICQM counterparts (93.8% in TIM versus 85.4% in TICQM, adjusted OR = 0.4, p = 0.017). Two other comparisons were of marginal statistical significance (0.05 < p <0.1). The prevalence for ever having had sexual intercourse with another man was higher in the male TICQM group (2.2%), compared to the male TIM group (0.4%) (χ2 test, p = 0.055). A higher percentage of male TICQM respondents also self reported having contracted an STD in the past 6 months compared to male TIM respondents (χ2 test, p = 0.098; table 3⇓).
The other comparisons in table 3⇑ were not of statistical significance. These included whether always using condoms when having had sexual intercourse with an FSW in the past 6 months, whether always using condoms with non-regular sex partners among those who reported having had a non-regular sex partner in the past 6 months, whether having indulged in substance abuse in the past 6 months, self perceived susceptibility of HIV infection and HIV antibody testing behaviours, etc (table 3⇑).
The respondents were interviewed by the two methods during the same time period, using the same standardised questionnaire and a randomised design to remove interviewer effects. The two samples were comparable in terms of demographic background and HIV related knowledge and attitude (part I). However, substantial and significant differences were detected for both the male and the female respondents using the two modes of data collection (the only difference accounting for the observed differences between the two methods).
Much higher frequencies of various sensitive attitudes and behaviours were reported by female respondents of the TICQM group when compared to responses given by females of the TIM group. These behaviours/attitudes were likely to be socially undesirable. For instance, having premarital sex is not desirable in the Chinese culture. As a result, women who admitted that they were sexually active might be pressured to say that their sex partner was their husband rather than their boyfriend. Other differences between the female TIM and TICQM groups in their responses to sensitive questions such as having had an experience of “one night stand,” substance abuse, HIV antibody testing, etc, also supported the claim. A similar pattern was also found among the male respondents—sexually active male TIM respondents were more likely to indicate that they had had sexual intercourse with a regular sex partner and were less likely to report having had sex with a non-regular sex partner, compared to their counterparts in the TICQM group. Male TICQM respondents were also more likely to admit having engaged in commercial sex behaviours than male TIM respondents.
It is also noted that among sexually active female respondents in the TIM group, 91.8% reported having had sex with their husband and 8.2% with their steady boyfriend. In other words, very few of the women in the TIM group admitted having had extramarital sex. However, such percentages were 72.0% (with husband) and 40.6% (with steady boyfriend) respectively in the TICQM group. Therefore, about 12% in the TICQM group admitted having had sex with both their husband and boyfriend. There are no reported data on extramarital sex among women in Hong Kong. The above mentioned findings therefore suggest a direction for further research.
Similar trends were also found among males. Among sexually active male respondents in the TIM group, 93.8% reported having had sexual intercourse with a regular sex partner, 5.1% with a non-regular sex partner, and 3.9% with an FSW; whereas for the TICQM group, the three respective values were 85.4%, 14.6%, and 11.1%.
Further, a few comparisons of the male TIM and TICQM groups (such as the prevalence of STD and sex with another man) reached marginal statistical significance (0.05 <p <0.1), suggesting that were the sample size increased, statistical significance could have been reached for these variables.
In summary, though not all comparisons are significant, the results consistently show that TICQM respondents, especially the female ones, were more likely to report higher frequencies of risk behaviours. The social pressure faced by the TIM respondents may be larger than those of the TICQM respondents. The former was responding to a person (the interviewer) and the latter was responding to a machine (computer), in the absence of a person.
The findings reinforce the conclusion of a similar study carried out in Hong Kong, comparing three modes of data collection methods in behavioural surveillance studies.5 The results of that study concluded that the method giving a higher level of perceived confidentiality and anonymity would yield higher frequencies of self reported risk behaviours. In the present study, switching to a prerecorded and computerised questionnaire created a better perception of confidentiality, as the interviewers left the telephone line when the respondents were answering the questions. The simple “key in” responding mode also reduced the chance of being overheard by other family members when the respondents were answering the very sensitive questions. These may partially explain the differences in the pattern of responses given by the two methods.
The findings have very important implications for research design and data interpretation. It rejects the null hypothesis that results of risk behaviour studies are independent of the mode of data collection. The conclusion casts considerable doubts on interviewer administered risk behaviour surveys via telephone interviews. The results of risk behaviour surveys should therefore be reviewed and interpreted critically, with special attention given to the mode of data collection. The implications should not only be limited to HIV/STD studies but should be highly relevant to other studies asking sensitive questions.
Another very interesting observation is that the differences of response bias may be gender dependent. For instance, the question on whether one had undergone an HIV antibody test in the past 6 months was asked to both the male and the female respondents, using exactly the same wording. However, statistical significance was observed in the female sample but not in the male sample. The multiplicative interaction between the mode of data collection and gender to determine the frequency of HIV testing was of statistical significance (test of homogeneity, χ2 = 7.507, df = 1, p = 0.006). In other words, the strength of the association between reporting HIV testing and data collection methods was different between the male and the female populations. As the underlying cause of the differences in responses (or reporting bias) may be related to perceived social desirability of the given answers, and since different societies have different norms and expectations on its male and female members, it is hence possible that, under some circumstances, the perception of social desirability differs by gender. HIV antibody test may be seen to be associated with the practice of risk behaviour. Since it is much more socially undesirable for a female than a male to practise risk behaviours in the Chinese culture (such as casual sex or commercial sex), it is expected that the pressure to conform with social desirability would be much greater for the female than the male, thus explaining the above mentioned sex differences. Similar sex method interaction was also observed for substance abuse in the past 6 months. The results therefore illustrate the importance of understanding the social context and its cultural impacts on responses to questions on risk behaviours. Again, such is not only limited to HIV/STD studies.
Another sex difference was also observed as male TICQM respondents found it easier to complete the questions in part II (only 0.7% started but did not finish part II) than female TICQM respondents (4.6% started but did not complete part II). One possibility is that females may find the questions more sensitive than males and were less willing to answer such questions. When using the TIM method, respondents might give a false answer to avoid embarrassment. But when the TICQM method was used, there is less need for them to present a false answer as there was no human interaction. Instead, some of them might have chosen to quit the study by hanging up. This presents a limitation to the TICQM method in that it may have a slightly lower completion rate. However, the higher female completion rate in the TIM group should only be accepted with caution, as some of the answers might be false. The suggested sex method interaction, when asking sensitive questions, is an important topic requiring further research efforts.
Though the frequencies of risk behaviours and attitudinal responses differed strongly by methods, the percentages of incomplete questionnaires and item non-responses did not. This implies that questionnaire completion rates and number of missing responses were not adequate indicators for assessing performance of different survey methods.
Though consistent differences between the two methods have been observed, such differences should not be overstated. It is seen that some, but not all comparisons were significant. In fact, there are more non-significant than significant comparisons found among the male respondents. Further research efforts to understand factors affecting the differences in results between different methods are required.
The results of this study strongly remind HIV/STD researchers, and others studying sensitive behaviours and/or attitudes, that they should pay very careful attention to the problem of reporting bias and give all possible efforts to reduce such biases. Innovative data collection methods should also be devised and shared by the community of researchers. The design presented by this study should have wide applications as telephone surveys are quite commonly used in the study of sexual behaviour.
The results of research studies using interviewers and telephone survey methods to collect sensitive personal data, such as those related to HIV/STD, should be interpreted with extreme care, as they are method dependent. When results of different studies are compared, the data collection method should also be taken into account when the data are interpreted.
A new method combining interviewer administration and prerecorded computerised questioning reports higher frequencies of risk behaviours, which are likely to be subjected to social desirability effects and reporting bias. Such is especially true for female respondents. The difference is explained by an enhanced perception of confidentiality and anonymity when the new method of the study was used. This method should be applicable in different countries.
It should be noted that the extent of reporting bias, which is likely to be related to social norms, may be gender specific. The pressure on females may be larger than that on males in some traditional societies, such as the Chinese ones. Therefore, the between method differences are expected to be more drastic for females than males, as observed in this study. Additional care and attention should hence be given when studying subpopulations (such as females) that are under stronger social pressure to reveal their risk behaviours. Cultural context should always be taken into account.
Completion rate and item response rates may not be good indicators to evaluate different types of data collection methods.
The authors would like to thank Ms MW Chan for her assistance in data collection, etc. The study was funded by the Hong Kong Council for the AIDS Trust Fund.
JL is the principal investigator of the study. He participated in the conception, design, initiation and implementation of the study, interpretation of data, and drafting of the paper. He is the lead writer; HYT participated in the design of the study, assisted in the statistical analyses for the study and interpretation of data. She also contributed to drafting the paper; QSW was responsible for the statistical analysis of the study, interpretation of data and drafting of the paper.