Online surveys are becoming increasingly popular due to the rapid growth in internet use. They are particularly effective for conducting research among certain hard-to-reach minority populations and have been used in a number of surveys of HIV risk behaviour among men who have sex with men (MSM) in The Netherlands,^1,2 Sweden,³ the UK^4–6 and the US.^7,8,9,10

Behavioural research among MSM has traditionally relied on convenience samples as a cost-effective method of generating samples of sufficient size for detailed analysis of high-risk sexual behaviour.¹¹ One approach is to recruit men who have contact with sexual health services.^12,13 However, this restricts the sample to an at-risk population of men who use these services. A second approach is to recruit probability¹⁴ or, more commonly, non-probability samples^15,16 of men from gay venues such as bars or clubs. This captures a more diverse population, but only reaches men who are affiliated with the gay community and attend such venues. The emergence of the internet provides yet another way of recruiting MSM for behavioural research.^{1,2,3,4,5,6,7,8,9,10,17}

A key advantage of recruiting MSM through the internet is the relative ease and speed with which large samples of MSM may be drawn from a wide geographical area. For example, the UK Gay Men’s Sex Survey 2004, which was promoted on gaydar and gay.com (the two most popular websites for homosexual men in the UK) and on 35 gay community and health promotion websites, recruited a UK internet sample of nearly 12 000 men over a period of 4 months.⁶ The internet may also access men who are more geographically isolated, younger and more likely to be bisexually or heterosexually identified.^4,18,19 These advantages suggest that the internet provides an attractive new venue for recruiting samples of MSM, especially as access to the internet is high in this group of men.^5,20 On the other hand, as internet-based surveys are generally promoted through gay-interest websites, this clearly restricts the sample to users of such sites.

In view of the potential benefits associated with recruiting MSM through the internet, it is important to evaluate the composition of the self-selected samples that these surveys attract. All convenience samples are vulnerable to biases according to how and where they were recruited. This is reflected in studies that have compared self-selected internet samples of MSM with other convenience samples.^4,9,18,19 To produce generalisable findings, however, an examination of the composition of internet samples requires comparison with a sample of MSM identified through a random probability survey rather than with another convenience sample.

The purpose of this study was to compare a self-selected convenience sample of MSM in Great Britain recruited through the internet with a sample of MSM identified from a national probability sample of adults resident in Great Britain.

METHODS

The internet sample was recruited for an online survey of sexual behaviour among MSM that was conducted in 2003. The probability sample consisted of MSM who participated in the National Survey of Sexual Attitudes and Lifestyles (Natsal) 2000. The methods of both studies have been described in detail elsewhere.^20–22

The research protocol was approved by the Royal Free Hospital and Medical School Local Research Ethics Committee, City University London Research Ethics Committee, the University College London, and North Thames Multi-centre Research Ethics Committees and all the local research ethics committees in the Great Britain.

Internet sample

The internet sample was recruited via gaydar (http://www.gaydar.co.uk/) and gay.com (http://uk.gay.com/). Over a 5-week period in May and June 2003, pop-up and banner advertisements appeared in chatrooms and profile pages asking men to participate in the survey. No incentives were offered for participation. Clicking on a pop-up or banner took respondents to the online survey, which they could complete and submit online. Only respondents who said they were at least 18 years old were allowed to answer the questionnaire. After questions on their sociodemographic profile, respondents were asked the sex of their sexual partners in the past year. At this point, only men who reported sex with another man over this period were asked to continue. Although the pop-ups and banners were restricted to UK chatrooms and profiles, men from anywhere in the world using these sites could participate.

Natsal sample

Natsal 2000 adopted a multistage-stratified probability design to identify a sample of men and women aged 16–44 years living in private households in the UK. After an investigation into the use of incentives in the early stages of fieldwork, respondents were offered a gift voucher of £5 regardless of whether they participated. MSM were defined as men reporting at least one male partner with whom they had genital contact in 5 years before the interview.^21,23 The Natsal sample used here refers to the subsample of men who reported sexual activity with a man in the past year. Fieldwork took place from May 1999 to February 2001, and interviews were conducted by trained interviewers using a combination of face-to-face interviews and a self-completion section administered by computer-assisted self-interviews, which covered the most sensitive questions.

Sample comparability

To ensure the closest comparability between the two samples, only men aged 18–44 years living in Great Britain (England, Scotland, Wales) were included in the analysis. All men included in the analysis stated that they were sexually active with a man in the past year.

Variable comparability

The samples were compared on all variables for which the questions in each survey were substantially equivalent (table 1). Even small differences in question wording and format may influence response patterns.²⁴ There were notable differences between the two studies in questions about anal intercourse and unprotected anal intercourse (UAI). Internet respondents were asked if they had had anal intercourse and UAI with a male partner in the past 3 months, whereas Natsal respondents were asked if they had had anal intercourse with a male partner in the previous week, 4 weeks, 6 months and 1 year, but not 3 months. Natsal respondents reported UAI with a male partner in the 4 weeks and 1 year before the interview.

Statistical analysis

Survey samples were compared using the survey analysis functions of the statistical software STATA V.7. The Natsal sample was weighted to adjust for unequal probabilities of selection and differential non-response to make it representative of the population in terms of age, sex and region.²² Adjustment weights were not applied to the internet sample because probabilities of selection and level of non-response cannot be calculated for a convenience sample.

Means or percentages are presented for all the variables examined for each sample. Confidence intervals (CI) of 95% are presented with the Natsal percentages to provide a measure of precision for these estimates. CIs for the internet percentages were narrow, and are not presented here because they add little to the interpretation of the data from this non-probability sample.

The t test for independent groups and χ² test were used to examine significant differences between the means and proportions of background characteristics of the samples. Binary logistic regression analysis was applied to examine the association between sample and HIV testing, sexually transmitted infection (STI), anal intercourse and UAI. Where there was a significant association between the sample and any of these outcomes (p<0.05), multivariable binary logistic regression analysis with forward stepwise selection was used to examine whether the sample remained a significant predictor of sexual risk when confounding variables were included in the model. Crude and adjusted odd ratios (aORs) are presented with 95% CI and p values.

Estimating anal intercourse and UAI in past 3 months

The Natsal sample reported anal intercourse in periods of up to 1 week, 4 weeks, 6 months, 1 year and ⩾5 years before the interview, whereas the internet sample reported anal intercourse only in the 3 months before completing the questionnaire. Figure 1 is a plot of when Natsal respondents who reported anal intercourse in the past year said that anal intercourse had occurred during that time. It shows the cumulative percentage of whether they reported anal intercourse in 1 week, 4 weeks, 6 months or 1 year before the interview. A logarithmic curve is fitted to the plot (y = 14.2ln(x)+42.2; y = percentage of men reporting anal intercourse; x = number of weeks). The curve indicates that, of those Natsal respondents reporting anal intercourse in the past year (n = 94), an estimated 78.8% had anal intercourse in 3 months before the interview (y = 14.2×ln(13)+42.2). In this way, we estimated that 74 Natsal respondents had anal intercourse in 3 months before the interview (94×0.788), and used this to estimate the percentage of all Natsal respondents (including those who did not report anal intercourse in the past year) who had had anal intercourse in the 3 months before the interview (74/117×100 = 63.3%).

The Natsal sample reported UAI for periods of 4 weeks and 1 year before the interview, whereas the internet sample reported UAI in the 3 months before completing the questionnaire. Fitting a logarithmic curve to the cumulative percentage of Natsal respondents reporting UAI in the 4 weeks and 1 year before the interview (y = 0.141ln(x)+0.396) indicates that an estimated 75.8% of Natsal respondents reporting UAI in the past year (n = 56) had UAI in the 3 months before the interview. This can be used to estimate the number and percentage of Natsal respondents who had UAI in the 3 months before the interview (n = 56×0.758 = 42.4 and 42.4/116×100 = 36.6%, respectively).

The χ² test was used to examine differences in the percentage of respondents reporting anal intercourse and UAI over the past 3 months because crude and aORs cannot be calculated for data that are only available at the sample level.

RESULTS

The internet sample consisted of 2065 MSM living in Great Britain (18–44 years) who completed the online survey. These men are thought to represent <1% of men using the gaydar and gay.com chatrooms and profiles over the survey period, on the basis of estimates of usage provided by the website owners. Natsal interviewed a total of 11 161 adults resident in Great Britain, with a response rate of 65.4%. The subsample of 135 MSM (18–44 years) that was identified for this analysis was equivalent to an effective subsample of 117 men, after adjusting for differing probabilities of selection and non-response.

Table 2 shows the sociodemographic and behavioural characteristics of the two samples. There were no significant differences between the two samples in terms of reported ethnicity, education, social class, being currently employed, living in an urban area, country of birth, alcohol consumption, injecting drug use or age when they first had sex with a male partner (p>0.05). Regional distribution about the country was not significantly different, but there was strong evidence²⁵ that, compared with Natsal respondents, men from the internet sample were younger (p<0.001), and more likely to be students (p = 0.001), but less likely to live in London (p = 0.001). There was weaker statistical evidence that men from the internet sample were less likely to report good health (p = 0.014) or be unemployed, retired or otherwise not working (p = 0.007).

We found no significant differences between the two samples in whether they reported testing for HIV (table 3). However, the data strongly indicated that, after adjusting for confounding factors, men from the internet sample were more likely to report having had an STI in the past year (16.9% v 4.8%, aOR 4.14, 95% CI 1.76 to 9.74; p = 0.001) and more likely to report anal intercourse: 76.9% of men from the internet sample reported anal intercourse in the past 3 months compared with an estimated 63.3% of Natsal respondents (p = 0.001). Men from the internet sample were also more likely to report anal intercourse in the past 3 months than Natsal respondents in the past 6 months (76.9% v 69.3%, aOR 1.59, 95% CI 1.03 to 2.45; p = 0.036). The statistical evidence for an association between sample and UAI was weaker. Of the internet respondents, 45% reported UAI in the past 3 months compared with an estimated 36.6% of Natsal respondents (p = 0.064).

DISCUSSION

This is the first study to compare a self-selected convenience sample of MSM recruited through the internet with a nationally representative sample of MSM. Previous investigations have compared self-selected internet samples of MSM with convenience samples that were recruited in gay venues.^4,9,18,19

In our study, the self-selected internet sample of MSM living in Great Britain was broadly similar to the sample of MSM drawn from a probability sample of the general population resident in Great Britain on a range of sociodemographic and behavioural variables. The internet sample contained more students and fewer respondents who were unemployed, retired or otherwise not working. This difference was also found in a Swedish sex survey, which compared men and women recruited via the internet with a probability sample.²⁶

However, we found strong statistical evidence of differences between samples in reporting STIs in the past year and anal intercourse in the past 3 months. The differences remained significant after adjusting for confounding factors, such as age at first sex or being a student. Men from the internet sample were also more likely to report UAI in the past 3 months, although the significant evidence for this was weak (p = 0.064), possibly owing to lack of power. These findings suggest that estimates of high-risk sexual behaviour based on internet samples of MSM are likely to overestimate levels of risk behaviour in the wider MSM population.

Men recruited while actively seeking male partners through websites such as gaydar and gay.com might be expected to be more sexually active than those recruited from the general population. A similar differential was found in the Swedish study where men and women recruited through the internet reported more sexual partners than the probability sample with which they were compared.²⁶ However, the differential is not limited to convenience samples recruited via the internet. A comparison of a subsample of MSM from London recruited in Natsal 2000 with a convenience sample of MSM London recruited in gay bars, clubs and saunas found that men from the convenience sample were also more likely to report STIs, more male sex partners and more UAI partners.²⁷

The difference seen here between internet and probability samples of MSM in reporting high-risk sexual behaviour suggests that adjustment weights may be usefully applied to internet samples of MSM to account for this potential selection bias. Although weights may be devised to successfully predict certain outcomes,²⁸ we advise caution in using the data presented here for this purpose. Their construction would be better investigated using a broad range of variables collected from internet and probability samples surveyed simultaneously using research instruments containing equivalent questions. In addition, a larger sample of MSM identified through a random probability survey would provide greater confidence in the generalisability of the weighting scheme.

Our analysis has some limitations. For example, it was not possible to compare identical measures of reported anal intercourse and UAI between the samples. However, the data clearly indicate that men in the internet sample were more likely to report anal intercourse and there is also more evidence that they were more likely to report UAI. Comparing UAI among internet respondents (in the past 3 months) and Natsal respondents (in the past 4 weeks) yielded an aOR >1. There was also significant evidence for a differential when comparing UAI among internet respondents (in the past 3 months) with the 3-month Natsal estimate, although it was not strong (p = 0.064). However, circumstantial evidence suggests that the differences in reporting UAI were real. This is because a similar proportion of men from the internet sample reported UAI in the past 3 months, as was reported by the Natsal respondents in the past 12 months. This suggests that if the men from the internet sample had been asked, they would have reported more UAI in the past 12 months than Natsal respondents.

Data on the HIV status of the respondents and their UAI partners were available for men from the internet sample, but not from the Natsal sample. Consequently, we could not examine whether the difference in UAI between the two samples was with a partner of the same HIV status (serosorting) or with a partner of discordant or unknown HIV status.

The issue of question comparability arises in the case of other questions, which were not identical. Data were furthermore not gathered at the same points in time, which might account for the differences in estimates of risk behaviour. However, this is unlikely as no increase in high-risk sexual behaviour with a casual partner was found among homosexual men in London between 2001 and 2003.²⁹

On the other hand, our study highlights an important advantage of recruiting national samples of MSM through the internet. The internet sample contained a total of 2065 men whereas the Natsal sample contained only 135 men (unweighted). The size of the internet sample facilitates detailed analysis of subgroups and multivariate analysis.

Overall, our findings suggest that the internet provides a valuable means of recruiting a large national sample of MSM whose social and demographic characteristics are broadly similar to those of a representative, probability sample. Internet samples are, however, likely to overestimate the prevalence of high-risk sexual behaviour, and the data they generate should be interpreted in this context.