Objectives: To compare the characteristics of a self-selected, convenience sample of men who have sex with men (MSM) recruited through the internet with MSM drawn from a national probability survey in Great Britain.
Methods: The internet sample (n = 2065) was recruited through two popular websites for homosexual men in Great Britain in May and June 2003. This sample was compared with MSM (n = 117) from the National Survey of Sexual Attitudes and Lifestyles (Natsal), a probability sample survey of adults resident in Great Britain conducted between May 1999 and February 2001.
Results: No significant differences were observed between the samples on a range of sociodemographic and behavioural variables (p>0.05). However, men from the internet sample were younger (p<0.001) and more likely to be students (p = 0.001), but less likely to live in London (p = 0.001) or report good health (p = 0.014). Although both samples were equally likely to report testing for HIV, men from the internet sample were more likely to report a sexually transmitted infection in the past year (16.9% v 4.8%, adjusted odds ratio 4.14, 95% CI 1.76 to 9.74; p = 0.001), anal intercourse (76.9% v 63.3%; p = 0.001) and unprotected anal intercourse in the past 3 months (45% v 36.6%; p = 0.064).
Conclusions: The internet provides a means of recruiting a self-selected, convenience sample of MSM whose social and demographic characteristics are broadly similar to those of MSM drawn from a national probability survey. However, estimates of high-risk sexual behaviour based on internet convenience samples are likely to overestimate levels of sexual risk behaviour in the wider MSM population.
- aOR, adjusted odds ratio
- MSM, men who have sex with men
- Natsal, National Survey of Sexual Attitudes and Lifestyles
- STI, sexually transmitted infection
- UAI, unprotected anal intercourse
Statistics from Altmetric.com
- aOR, adjusted odds ratio
- MSM, men who have sex with men
- Natsal, National Survey of Sexual Attitudes and Lifestyles
- STI, sexually transmitted infection
- UAI, unprotected anal intercourse
Online surveys are becoming increasingly popular due to the rapid growth in internet use. They are particularly effective for conducting research among certain hard-to-reach minority populations and have been used in a number of surveys of HIV risk behaviour among men who have sex with men (MSM) in The Netherlands,1,2 Sweden,3 the UK4–6 and the US.7,8,9,10
Behavioural research among MSM has traditionally relied on convenience samples as a cost-effective method of generating samples of sufficient size for detailed analysis of high-risk sexual behaviour.11 One approach is to recruit men who have contact with sexual health services.12,13 However, this restricts the sample to an at-risk population of men who use these services. A second approach is to recruit probability14 or, more commonly, non-probability samples15,16 of men from gay venues such as bars or clubs. This captures a more diverse population, but only reaches men who are affiliated with the gay community and attend such venues. The emergence of the internet provides yet another way of recruiting MSM for behavioural research.1,2,3,4,5,6,7,8,9,10,17
A key advantage of recruiting MSM through the internet is the relative ease and speed with which large samples of MSM may be drawn from a wide geographical area. For example, the UK Gay Men’s Sex Survey 2004, which was promoted on gaydar and gay.com (the two most popular websites for homosexual men in the UK) and on 35 gay community and health promotion websites, recruited a UK internet sample of nearly 12 000 men over a period of 4 months.6 The internet may also access men who are more geographically isolated, younger and more likely to be bisexually or heterosexually identified.4,18,19 These advantages suggest that the internet provides an attractive new venue for recruiting samples of MSM, especially as access to the internet is high in this group of men.5,20 On the other hand, as internet-based surveys are generally promoted through gay-interest websites, this clearly restricts the sample to users of such sites.
In view of the potential benefits associated with recruiting MSM through the internet, it is important to evaluate the composition of the self-selected samples that these surveys attract. All convenience samples are vulnerable to biases according to how and where they were recruited. This is reflected in studies that have compared self-selected internet samples of MSM with other convenience samples.4,9,18,19 To produce generalisable findings, however, an examination of the composition of internet samples requires comparison with a sample of MSM identified through a random probability survey rather than with another convenience sample.
The purpose of this study was to compare a self-selected convenience sample of MSM in Great Britain recruited through the internet with a sample of MSM identified from a national probability sample of adults resident in Great Britain.
The internet sample was recruited for an online survey of sexual behaviour among MSM that was conducted in 2003. The probability sample consisted of MSM who participated in the National Survey of Sexual Attitudes and Lifestyles (Natsal) 2000. The methods of both studies have been described in detail elsewhere.20–22
The research protocol was approved by the Royal Free Hospital and Medical School Local Research Ethics Committee, City University London Research Ethics Committee, the University College London, and North Thames Multi-centre Research Ethics Committees and all the local research ethics committees in the Great Britain.
The internet sample was recruited via gaydar (http://www.gaydar.co.uk/) and gay.com (http://uk.gay.com/). Over a 5-week period in May and June 2003, pop-up and banner advertisements appeared in chatrooms and profile pages asking men to participate in the survey. No incentives were offered for participation. Clicking on a pop-up or banner took respondents to the online survey, which they could complete and submit online. Only respondents who said they were at least 18 years old were allowed to answer the questionnaire. After questions on their sociodemographic profile, respondents were asked the sex of their sexual partners in the past year. At this point, only men who reported sex with another man over this period were asked to continue. Although the pop-ups and banners were restricted to UK chatrooms and profiles, men from anywhere in the world using these sites could participate.
Natsal 2000 adopted a multistage-stratified probability design to identify a sample of men and women aged 16–44 years living in private households in the UK. After an investigation into the use of incentives in the early stages of fieldwork, respondents were offered a gift voucher of £5 regardless of whether they participated. MSM were defined as men reporting at least one male partner with whom they had genital contact in 5 years before the interview.21,23 The Natsal sample used here refers to the subsample of men who reported sexual activity with a man in the past year. Fieldwork took place from May 1999 to February 2001, and interviews were conducted by trained interviewers using a combination of face-to-face interviews and a self-completion section administered by computer-assisted self-interviews, which covered the most sensitive questions.
To ensure the closest comparability between the two samples, only men aged 18–44 years living in Great Britain (England, Scotland, Wales) were included in the analysis. All men included in the analysis stated that they were sexually active with a man in the past year.
The samples were compared on all variables for which the questions in each survey were substantially equivalent (table 1). Even small differences in question wording and format may influence response patterns.24 There were notable differences between the two studies in questions about anal intercourse and unprotected anal intercourse (UAI). Internet respondents were asked if they had had anal intercourse and UAI with a male partner in the past 3 months, whereas Natsal respondents were asked if they had had anal intercourse with a male partner in the previous week, 4 weeks, 6 months and 1 year, but not 3 months. Natsal respondents reported UAI with a male partner in the 4 weeks and 1 year before the interview.
Survey samples were compared using the survey analysis functions of the statistical software STATA V.7. The Natsal sample was weighted to adjust for unequal probabilities of selection and differential non-response to make it representative of the population in terms of age, sex and region.22 Adjustment weights were not applied to the internet sample because probabilities of selection and level of non-response cannot be calculated for a convenience sample.
Means or percentages are presented for all the variables examined for each sample. Confidence intervals (CI) of 95% are presented with the Natsal percentages to provide a measure of precision for these estimates. CIs for the internet percentages were narrow, and are not presented here because they add little to the interpretation of the data from this non-probability sample.
The t test for independent groups and χ2 test were used to examine significant differences between the means and proportions of background characteristics of the samples. Binary logistic regression analysis was applied to examine the association between sample and HIV testing, sexually transmitted infection (STI), anal intercourse and UAI. Where there was a significant association between the sample and any of these outcomes (p<0.05), multivariable binary logistic regression analysis with forward stepwise selection was used to examine whether the sample remained a significant predictor of sexual risk when confounding variables were included in the model. Crude and adjusted odd ratios (aORs) are presented with 95% CI and p values.
Estimating anal intercourse and UAI in past 3 months
The Natsal sample reported anal intercourse in periods of up to 1 week, 4 weeks, 6 months, 1 year and ⩾5 years before the interview, whereas the internet sample reported anal intercourse only in the 3 months before completing the questionnaire. Figure 1 is a plot of when Natsal respondents who reported anal intercourse in the past year said that anal intercourse had occurred during that time. It shows the cumulative percentage of whether they reported anal intercourse in 1 week, 4 weeks, 6 months or 1 year before the interview. A logarithmic curve is fitted to the plot (y = 14.2ln(x)+42.2; y = percentage of men reporting anal intercourse; x = number of weeks). The curve indicates that, of those Natsal respondents reporting anal intercourse in the past year (n = 94), an estimated 78.8% had anal intercourse in 3 months before the interview (y = 14.2×ln(13)+42.2). In this way, we estimated that 74 Natsal respondents had anal intercourse in 3 months before the interview (94×0.788), and used this to estimate the percentage of all Natsal respondents (including those who did not report anal intercourse in the past year) who had had anal intercourse in the 3 months before the interview (74/117×100 = 63.3%).
The Natsal sample reported UAI for periods of 4 weeks and 1 year before the interview, whereas the internet sample reported UAI in the 3 months before completing the questionnaire. Fitting a logarithmic curve to the cumulative percentage of Natsal respondents reporting UAI in the 4 weeks and 1 year before the interview (y = 0.141ln(x)+0.396) indicates that an estimated 75.8% of Natsal respondents reporting UAI in the past year (n = 56) had UAI in the 3 months before the interview. This can be used to estimate the number and percentage of Natsal respondents who had UAI in the 3 months before the interview (n = 56×0.758 = 42.4 and 42.4/116×100 = 36.6%, respectively).
The χ2 test was used to examine differences in the percentage of respondents reporting anal intercourse and UAI over the past 3 months because crude and aORs cannot be calculated for data that are only available at the sample level.
The internet sample consisted of 2065 MSM living in Great Britain (18–44 years) who completed the online survey. These men are thought to represent <1% of men using the gaydar and gay.com chatrooms and profiles over the survey period, on the basis of estimates of usage provided by the website owners. Natsal interviewed a total of 11 161 adults resident in Great Britain, with a response rate of 65.4%. The subsample of 135 MSM (18–44 years) that was identified for this analysis was equivalent to an effective subsample of 117 men, after adjusting for differing probabilities of selection and non-response.
Table 2 shows the sociodemographic and behavioural characteristics of the two samples. There were no significant differences between the two samples in terms of reported ethnicity, education, social class, being currently employed, living in an urban area, country of birth, alcohol consumption, injecting drug use or age when they first had sex with a male partner (p>0.05). Regional distribution about the country was not significantly different, but there was strong evidence25 that, compared with Natsal respondents, men from the internet sample were younger (p<0.001), and more likely to be students (p = 0.001), but less likely to live in London (p = 0.001). There was weaker statistical evidence that men from the internet sample were less likely to report good health (p = 0.014) or be unemployed, retired or otherwise not working (p = 0.007).
We found no significant differences between the two samples in whether they reported testing for HIV (table 3). However, the data strongly indicated that, after adjusting for confounding factors, men from the internet sample were more likely to report having had an STI in the past year (16.9% v 4.8%, aOR 4.14, 95% CI 1.76 to 9.74; p = 0.001) and more likely to report anal intercourse: 76.9% of men from the internet sample reported anal intercourse in the past 3 months compared with an estimated 63.3% of Natsal respondents (p = 0.001). Men from the internet sample were also more likely to report anal intercourse in the past 3 months than Natsal respondents in the past 6 months (76.9% v 69.3%, aOR 1.59, 95% CI 1.03 to 2.45; p = 0.036). The statistical evidence for an association between sample and UAI was weaker. Of the internet respondents, 45% reported UAI in the past 3 months compared with an estimated 36.6% of Natsal respondents (p = 0.064).
This is the first study to compare a self-selected convenience sample of MSM recruited through the internet with a nationally representative sample of MSM. Previous investigations have compared self-selected internet samples of MSM with convenience samples that were recruited in gay venues.4,9,18,19
In our study, the self-selected internet sample of MSM living in Great Britain was broadly similar to the sample of MSM drawn from a probability sample of the general population resident in Great Britain on a range of sociodemographic and behavioural variables. The internet sample contained more students and fewer respondents who were unemployed, retired or otherwise not working. This difference was also found in a Swedish sex survey, which compared men and women recruited via the internet with a probability sample.26
However, we found strong statistical evidence of differences between samples in reporting STIs in the past year and anal intercourse in the past 3 months. The differences remained significant after adjusting for confounding factors, such as age at first sex or being a student. Men from the internet sample were also more likely to report UAI in the past 3 months, although the significant evidence for this was weak (p = 0.064), possibly owing to lack of power. These findings suggest that estimates of high-risk sexual behaviour based on internet samples of MSM are likely to overestimate levels of risk behaviour in the wider MSM population.
Men recruited while actively seeking male partners through websites such as gaydar and gay.com might be expected to be more sexually active than those recruited from the general population. A similar differential was found in the Swedish study where men and women recruited through the internet reported more sexual partners than the probability sample with which they were compared.26 However, the differential is not limited to convenience samples recruited via the internet. A comparison of a subsample of MSM from London recruited in Natsal 2000 with a convenience sample of MSM London recruited in gay bars, clubs and saunas found that men from the convenience sample were also more likely to report STIs, more male sex partners and more UAI partners.27
The difference seen here between internet and probability samples of MSM in reporting high-risk sexual behaviour suggests that adjustment weights may be usefully applied to internet samples of MSM to account for this potential selection bias. Although weights may be devised to successfully predict certain outcomes,28 we advise caution in using the data presented here for this purpose. Their construction would be better investigated using a broad range of variables collected from internet and probability samples surveyed simultaneously using research instruments containing equivalent questions. In addition, a larger sample of MSM identified through a random probability survey would provide greater confidence in the generalisability of the weighting scheme.
Our analysis has some limitations. For example, it was not possible to compare identical measures of reported anal intercourse and UAI between the samples. However, the data clearly indicate that men in the internet sample were more likely to report anal intercourse and there is also more evidence that they were more likely to report UAI. Comparing UAI among internet respondents (in the past 3 months) and Natsal respondents (in the past 4 weeks) yielded an aOR >1. There was also significant evidence for a differential when comparing UAI among internet respondents (in the past 3 months) with the 3-month Natsal estimate, although it was not strong (p = 0.064). However, circumstantial evidence suggests that the differences in reporting UAI were real. This is because a similar proportion of men from the internet sample reported UAI in the past 3 months, as was reported by the Natsal respondents in the past 12 months. This suggests that if the men from the internet sample had been asked, they would have reported more UAI in the past 12 months than Natsal respondents.
Data on the HIV status of the respondents and their UAI partners were available for men from the internet sample, but not from the Natsal sample. Consequently, we could not examine whether the difference in UAI between the two samples was with a partner of the same HIV status (serosorting) or with a partner of discordant or unknown HIV status.
The issue of question comparability arises in the case of other questions, which were not identical. Data were furthermore not gathered at the same points in time, which might account for the differences in estimates of risk behaviour. However, this is unlikely as no increase in high-risk sexual behaviour with a casual partner was found among homosexual men in London between 2001 and 2003.29
On the other hand, our study highlights an important advantage of recruiting national samples of MSM through the internet. The internet sample contained a total of 2065 men whereas the Natsal sample contained only 135 men (unweighted). The size of the internet sample facilitates detailed analysis of subgroups and multivariate analysis.
Overall, our findings suggest that the internet provides a valuable means of recruiting a large national sample of MSM whose social and demographic characteristics are broadly similar to those of a representative, probability sample. Internet samples are, however, likely to overestimate the prevalence of high-risk sexual behaviour, and the data they generate should be interpreted in this context.
We thank gaydar, gay.com and all the men who participated in the studies.
The paper by Evans et al,1 which compares a self-selected internet sample of gay men with a national probability sample of men who have sex with men (MSM), fills an important gap in the field of internet and MSM sampling. A decade ago, the internet was considered a vagary of electronics for the purpose of sampling, but studies such as that of Ross et al2 in Sweden demonstrated that rather than collecting “more of the same” urban and gay-acculturated gay men as traditional samples, an internet sample appeared to contain more men who were younger, bisexual or heterosexual, from small towns or rural areas and poorly educated, compared with traditional bar and venue samples. More recently, Ross et al3 compared an internet sample (heterosexual, bisexual, MSM) in Sweden with a gold-standard random sample from a national sexuality study and found that while there were expected differences in age (younger) and occupation (more students and fewer retired people) and a bias towards larger cities in the internet sample, the differences between the two samples were not major.
The comparison of a gay internet sample with a national random study population of MSMs, however, showed a significant gap until this study by Evans et al.1 Their data show trends similar to the Swedish national probability study3 in that internet respondents were more likely to be younger and students, but were also more likely to live outside major metropolitan areas, as Ross et al2 had found in their convenience sample of MSM on the internet in Sweden. Particularly important, however, is Evans et al’s findings that the internet sample of MSM was likely to be in worse health (despite being slightly younger on average), over four times more likely to have had a sexually transmitted infection (STI) in the past year and to have major differences in the reports of anal (and unprotected anal) intercourse. These latter findings seem to reinforce the earlier findings of this team,4,5 that the internet may act as a site for people with HIV infection to “serosort”—select other HIV-infected men as partners for unprotected sex.
The internet has become an electronic venue for partner selection for gay men and other MSM, which is unrivalled in its potential sample size. It has the classic advantages of anonymity, affordability, acceptability and what is called “approximation”—use of the internet for sexual experimentation or cybersex by men who might not otherwise be accessible for study in traditional venues.6 Evans et al confirm that the MSM who were previously most difficult to access (non-London-dwellers and younger men) can be reached through the internet. The internet is, according to their data, also significant as a source for accessing MSM whose sexual risk behaviour for HIV and other STI acquisition is high—contrary to the earlier finding in the Swedish MSM convenience samples.2 The internet can also access MSM, some of whom may not be identified as gay, who are spread across the country and away from prevention programmes which are concentrated in larger cities.
HIV/STI prevention programmes using the internet can deliver tailored interventions at any hour of the day or night anonymously and efficiently right into the bedroom (or wherever the computer is located!). And, as Evans et al now demonstrate, the biases within an internet sample, which were assumed to be massive given the usually <1% sample of banner-generated links to website to completed surveys,7 are quantifiable and are often smaller than many might imagine. Evans and colleagues have put an important capstone on internet sampling studies with regard to MSM. From these data, for this population, internet sample weighting or stratification will remove one of the last methodological barriers to sampling; will aid implementing HIV/STI prevention internet interventions; and will promote generalising from internet data.
Published Online First 29 November 2006
Funding: This research was funded by the Economic and Social Research Council. The “Internet and HIV” study was funded by the Medical Research Council (grant number GO 100 159). Natsal 2000 was supported by a grant from the Medical Research Council with funds from the Department of Health, the Scottish Executive and the National Assembly for Wales.
Competing interests: None declared.
Contributors: All authors contributed to the design of the study; ARE and CHM performed the statistical analysis with input from RDW; GJB was responsible for the implementation of the internet and HIV web survey with input from JE; ARE wrote the first draft and coordinated subsequent revisions; all authors read the manuscript, suggested revisions and approved the final version.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.