Background

Effective planning for the prevention of HIV and sexually transmitted infections (STIs) requires adequate data about the distribution of risk and precaution behaviours, as well as the extent of unmet prevention needs (that is, motivations, opportunities and capabilities to engage in precautions).

Data on HIV/STI among men-that-have-sex-with-men (MSM) remain very variable between European countries, including those on infections and diagnoses, risk and precaution behaviours, prevention needs, and the performance of related services. Infection surveillance varies by coverage, representativeness and comprehensiveness making it difficult to compare annual HIV/STI diagnosis rates across countries and sometimes within countries across time.

Reliable surveillance data are essential for monitoring the HIV/STI disease burden. Knowing our epidemics is the key to tailoring effective responses. The European Centre for Disease Prevention and Control (ECDC) is the European Union (EU) agency that strengthens the capacity of countries to prevent and control infectious diseases, including HIV and STIs. ECDC has a mandate to strengthen and coordinate behavioural surveillance of infectious diseases at the EU level and support countries to conduct behavioural surveillance related to STIs and HIV (ECDC & WHO Regional Office for Europe, 2018).

Current Overview of HIV/STI Diagnoses Among MSM in Europe

HIV infection remains a major public health concern in Europe with over 55,000 new diagnoses in 2017, of which 46% were in the European Economic Area (EEA) (ECDC & WHO Regional Office for Europe, 2018). In most EEA countries, HIV is concentrated in specific sub-populations, such as gay men, bisexual men and other MSM, sex workers, particular migrant populations and people who inject drugs. MSM accounted for precisely 50% of all HIV infections diagnosed in 2018 in the EEA (and whose route of transmission was determined) (ECDC & WHO Regional Office for Europe, 2018).

In the EEA, five countries account for 70% of new HIV diagnoses in MSM: France, Germany, Italy, Spain and the UK. Since 2014, all five of these countries have observed declining numbers of new HIV diagnoses among MSM (ECDC & WHO Regional Office for Europe, 2018), so has Switzerland (Chapin-Bardales et al., 2018). Consequently, the overall number of new diagnoses of HIV infections acquired during sex between men in the EEA overall is currently in decline. On the other hand, in other European countries, new HIV diagnoses have been increasing over the last years.

Other sexually transmitted infections (STIs) also remain common among MSM in Europe, and a number of countries are reporting increasing numbers of diagnoses of bacterial STIs (such as syphilis and gonorrhoea), as well as outbreaks of rarer STIs such as lymphogranuloma venereum (ECDC, 2019b) or sexually transmitted hepatitis A (WHO Regional Office for Europe, 2017).

Methods of Generating Other MSM Data

It remains challenging to gather robust information on the behaviours and needs of stigmatised minority groups. Difficulties in defining groups like MSM make estimating the size of the true population complex. The sensitivity attached to sexual behaviour in general, the stigma attached to same-sex sexual activity in particular, and the absence of sampling frames for MSM mean random sampling is challenging, if not impossible in most countries. Small representative samples of MSM (and other sexual minorities) can be generated during national general population surveys of sexual lifestyles, given sensitive design (Mercer et al., 2016). However, all larger samples of MSM rely on convenience samples and provide less representative data (Semlyen & Hagger-Johnson, 2016).

Most online surveys have concentrated on regional or national populations of MSM. Between-country comparability is undermined by varying recruitment methods, varying questionnaire items, and varying question-response sets. Moreover, different ways of defining and accessing MSM lead to different sample compositions. Collaboration on data collection from MSM across Europe has become more common in the last 10 years, with projects funded by the EU Health Programme including EMIS-2010 (The EMIS Network, 2013; Weatherburn et al., 2013); SIALON I (Mirandola et al., 2009); SIALON II (Gios et al., 2016; Mirandola et al., 2016) and this survey.

ESTICOM and EMIS-2017

ESTICOM (European Surveys and Training to Improve MSM Community Health) was a 3-year project (August 2016 to August 2019) funded by the European Commission Health Programme 2014-2020 through a tender commissioned by the Consumers, Health, Agriculture and Food Executive Agency (Chafea). ESTICOM aimed to strengthen community responses to the legal, political and social obstacles to the prevention of STIs among MSM, including HIV and viral hepatitis.

The second European MSM Internet Survey (EMIS-2017) was designed to consolidate progress made in multi-country research projects, and to contribute to ESTICOM. It was a multi-language, pan-European, collaborative HIV prevention needs assessment, including measures of morbidity, behaviours, unmet prevention needs and intervention performance. Following on from EMIS-2010 (Weatherburn et al., 2013) it served as the second wave of a pan-European, behavioural surveillance system focussed on national samples, encompassing prevention needs as well as morbidities and behaviours. EMIS was first undertaken just prior to the emergence of online venues, such as smartphone-based geo-spatial apps, as a common means for men to find new male sex partners. This increase in online partner seeking could be assumed to be especially salient for areas without established gay communities and/or men without strong connections to those communities (such as men without a gay or bisexual identity and some migrants). These changing patterns of sexual partner seeking make online behavioural surveillance of MSM even more relevant, and online surveys ever more efficient.

Methods and Design

A Collaborative Survey

We conceived the project as a large-scale partnership, the EMIS Network. Our development process ensured we had a national lead for the promotion and/or a national lead for data analysis in almost all the countries included in the survey. Each step of the design process engaged with the EMIS Network and drew on its collective knowledge and expressed needs.

As well as generating data from MSM, EMIS-2017 also intended to transfer knowledge about online surveys to our research collaborators; to build research capacity; to generate datasets in countries with fewer research resources; to facilitate dialogue between community, academic and public health sectors; and to maximise the educational impact of survey completion for respondents.

Population of Concern

Our population of concern was set by the tender documents from Chafea, which specified the survey should be for ‘men-who-have-sex-with-men’. We operationalised this, first, as adults who identify as men, including both cismen and transmen. Adult we took to be the age at which males can legally consent to sex with other males in their country of residence, which varied by country.

Transwomen as a group are disproportionately affected by HIV and are an important target group for prevention. Transwomen require dedicated, funded and targeted HIV prevention services as well as research to develop those services. Women (both cis and trans) were not part of the population of concern for EMIS-2017 and were not intended to be so. If they had been, we would have developed the survey differently, actively engaged transwomen’s organisations and developed an independent recruitment strategy.

Respondents were told “In this survey, we use ‘sex’ to mean physical contact to orgasm (or close to orgasm) for one or both partners”. We used a positive response to this definition to identify men who ever had sex with another man. As the survey data is to be used for prospective planning (that is, it is concerned with the needs of men who will have sex in the future), men who provided a negative response but who are sexually attracted to men were included in the sample.

Since the survey was online only, there was no geographic limit to the countries we could be concerned with. As the survey was designed to have an educational as well as data generating function – wider participation was a good thing as long as data and outputs could be managed appropriately. On the other hand, every additional country (and especially language) increased the volume of engagement, management and data handling needed.

Drivers for inclusion of countries and consequently the languages the survey was offered in were diverse. Ultimately, there were 46 target countries (33 core and 13 non-core). The project was funded by the EU Health Programme which includes: the 28 EU member states; two European Free Trade Area (EFTA) countries that are not EU member states (Iceland and Norway); two EU enlargement Area countries (Serbia and Bosnia & Herzegovina); and one European Neighbourhood Policy country (Moldova). These 33 ‘core’ countries were mandated by the tender and our core funding included questionnaire translation and promotion in these countries.

In addition, we had existing relationships with, or were contacted by stakeholders in a number of countries who sought inclusion, several of whom had been involved in EMIS-2010. Non-core partners provided translations and invested resources in recruitment for their countries or were supported by other countries to do so. On this basis we included another 13 ‘non-core’ countries: Switzerland (the only large EFTA country not in the Health Programme); Albania, Kosovo,Footnote 1 Montenegro, North Macedonia, and Turkey (all EU Enlargement Area countries); Belarus, Lebanon, Israel and Ukraine (all European Neighbourhood Policy countries); and Russia, Canada and the Philippines. In addition, we invited participation by men living in the four (non-EU) microstates adjoining or encompassed by target countries. While we did not expect to recruit over 100 men in these microstates, we did not want to deny them the opportunity to participate. We include microstate cases in the neighbouring national datasets: Andorra with Spain; Liechtenstein with Switzerland; Monaco with France; and San Marino with Italy.

Recruitment Targets

Our goal was to recruit at least 100 eligible residents in each of the 46 target countries, which we considered a viable sample for national planning purposes. We committed to create and pass back national datasets to lead agencies in countries with at least 100 eligible respondents although we recognised this might not be possible in countries with a small overall population.

(Re-)Building the Network

In August 2015, in preparation of EMIS-2017, the 2010 EMIS Network, consisting of more than 80 academic, governmental, and civil society organisations, was contacted by email with a request to identify sexual health centres for gay men throughout Europe, the results of which were presented in Malta in January 2017 (Schmidt, Sander, & Noori, 2017). This exercise was important for re-building the EMIS network starting in November 2016. We contacted all people and institutions active in EMIS-2010, identified new key players in the field of HIV/STI prevention and epidemiology, approaching LGBT organisations as well as people in academia and national public health institutions. All researchers who had worked with EMIS-2010 data after the end of the funding period, including people external to the 2010 EMIS Network, were contacted and asked for input for the new questionnaire. Ultimately, the re-vitalised EMIS Network consisted of seven funders, six pan-European organisations, 157 national organisations from 44 countries (no partners in the 4 European non-EU microstates, or in Luxembourg or Kosovo), and 41 individuals.

Questionnaire Design

The tender asked for a survey for HIV, STI and viral hepatitis, generating data for planning interventions and monitoring change. The requirement was not for a generalised health survey for MSM. The goal was therefore to generate data useful for the planning of HIV and STI prevention and treatment activities and the monitoring of change in these areas. It does this by describing the level and distribution of HIV/STI related morbidities, the risk and precaution behaviours implicated in the morbidities, prevention needs, and interventions (including services). Since HIV/STIs are closely related to both substance use and mental health, and since many of our collaborators work across these three health outcomes, we took a holistic approach to sexual health.

The primary prevention-planning aims of EMIS-2017 were to identify prevention needs commonly unmet across diverse groups of MSM (priority aims) and to identify subgroups of men who had multiple prevention needs poorly met (priority target groups).

The starting point for the EMIS-2017 questionnaire was the EMIS-2010 questionnaire. There were four conflicting imperatives in its development: keeping topics we asked about in EMIS-2010 to examine changes, versus replacing them with new topics; keeping actual questions identical to EMIS-2010 for comparability, versus changing questions to improve measurements; seeking larger numbers of possible responses to better reflect people’s experience and increase data specificity, versus smaller numbers of responses to increase ease of completion and data management; aiming for a longer survey capturing more variables but with higher attrition and the sample becoming less representative of disadvantaged groups, versus a shorter survey capturing fewer variables while retaining more men, especially those from disadvantaged groups.

The survey development was informed by a review of current evidence (Cawley & Marcus, 2017) about HIV/STI and related behaviours among MSM, including a mapping of the likely legal, structural, provider and individual barriers to prevention and treatment. It was also informed by a scoping exercise of available national MSM questionnaires published since EMIS-2010.

Ultimately, the questionnaire arose from a three-stage iterative consultation across the EMIS Network. All consultations occurred in English.

Prior to First Consultation

We developed a conceptual map for the EMIS-2017 questionnaire, based on EMIS-2010. We then compiled and reviewed the EMIS-2010 question stems, omitting some that had proved problematic at analysis and/or had generated little interest among collaborators. We added new question areas which would be required in 2017 (for example, pre-exposure prophylaxis (PrEP) and chemsex).

Network Consultation Round One: Prioritising Questions

In November 2016 we sent the Priorities Consultation document to 318 unique email addresses. Consultees were asked to: nominate questions for cutting; suggest new topics to be included; and highlight problems with existing question stems. Fifty responses were received from 21 countries. The nominations provided clear direction on question priorities and ensuring the data was relevant to potential future users. We made numerous changes to existing questions and accounted for major decisions to include/exclude or change items.

The first complete draft of the survey was the outcome of balancing tensions between inclusivity and brevity. While prioritising respondent experience, we aimed to make the survey as useful as possible to the largest number of stakeholders. This first draft, along with an account of round one and instructions for round two, formed the subsequent Acceptability Consultation document.

Network Consultation Round Two: Collaborator Acceptability

The initial first full draft of the survey was subject to several small-scale pre-testing activities on paper during March 2017 with gay men known to the research team. The purpose of these pre-pilots was to test sections of the questionnaire for acceptability, completeness, phrasing, and comprehension in English.

The Acceptability Consultation document was sent out for EMIS Network consultation in March 2017. Consultees were asked to Track-Changes or use Comments to add, delete or change the survey. They were specifically asked to confirm that the regional response set for their country was appropriate. Forty responses were received from 37 organisations in 19 countries. Nine responses affirmed the questionnaire was acceptable (and added little or no comment) and 31 gave specific feedback. The consultation provided a large amount of detailed feedback and clear direction, leading to multiple changes to the questionnaire. While the core study group ultimately decided which feedback would lead to changes all suggestions for amendment, comment, and criticism was considered and all those that commented received feedback about what had occurred as a consequence.

A second draft of the survey was constructed indicating changes that had occurred as a result of the second round of consultation. This draft was agreed with the funder and transferred to an online survey application (Demographix.com) in English. The transfer of a paper survey to an online survey requires numerous small modifications. This first online draft was then pre-tested.

Talk-aloud Pre-testing Interviews

We recruited ten MSM through community sources in London to take part in ‘talk-aloud’ interviews. Respondents were asked to complete the online survey and to voice their thoughts about what they were reading and how they were responding. Interviewers probed all new questions; all hesitations and reversals (going back to a previous page). In response to user comments we changed many small details including: correcting typographic errors; correcting routing errors; standardising the way individual questions and their answers were presented; re-ordering pages; re-ordering questions on pages; re-wording of question stems; re-ordering of response sets; and expansion of response sets. No questions were cut, although notes were made on what men found repetitive. The talk-loud pre-testing and corrections resulted in the second online questionnaire draft.

Network Consultation Round Three: Online Performance and Trimming

We invited the EMIS Network to test and provide feedback on the first online draft of the English questionnaire in May 2017. Overall, 17 agencies responded from 12 countries. Responders raised a very wide range of issues. Every issue was considered and a response given to the consultee. The process resulted in several larger and smaller changes to the survey, as well as identifying potential cuts. Identifying cuts was important as at this stage as we were concerned about survey length.

We carried out an item analysis of the content, looking at the balance between the five main categories of question (demographics, morbidities, behaviours, needs and interventions). Debate within the study-staff identified the least essential questions within each category and several questions were cut. This resulted in the third online draft composed of 325 questions over 50 online pages. This online draft was subject to a time-trial.

Time-trial

We engaged with five health promotion agencies in the UK and asked them to recruit up to ten men each to complete the timed online survey. After adding in three trials carried out by English speakers in Germany, we estimated the median completion time of the second online draft to be 21 min with a very wide range (12 to 64 min). This time was marginally longer than that of EMIS-2010, so we again identified a small number of ‘least vital’ questions to drop. Several minor issues mainly related to layout and design were also corrected as they arose.

We estimated the final online questionnaire to have an average completion time of 20 min, the same as EMIS-2010. While we had aimed to make the questionnaire shorter in order to reduce attrition, the range of stakeholders and their priorities meant further cutting became untenable. The average completion time of 20 min was noted for potential participants on the entry page to the survey. A small number of adjustments resulted in the final questionnaire.

Final Questionnaire

The final questionnaire sought 409 different data items from respondents, although no men were asked for all 409 items. To minimise completion time the survey was tailored using question filters (routing) wherever possible (for example, only men with a steady partner were asked how long they had been in a relationship).

It was presented on 34 core pages and, depending on answers to some questions (for example, HIV testing history, sexual partners and substance use), another 25 non-core pages. Of the 25 non-core pages, one was served only to those who did not know the age of male homosexual consent in their country of residence, and four were exit pages for non-qualifiers. The remaining 20 were served depending on answers to earlier questions.

To reduce the burden on respondents, they were randomly allocated to being served one of two psychometric scales, each consisting of 7 or 8 items. These were two sub-scales of the Social Provisions Scale (SPS) and the Short Internalised Homonegativity Scale (SIHS). This was the only point in the survey where respondents were randomly split and served different questions.

The final English-language questionnaire formed the basis of the EMIS-2017 ethics application. The study was given a positive opinion by the Observational Research Ethics Committee at the London School of Hygiene and Tropical Medicine (reference 14421/RR/8805) on 31 July 2017.

The content of the final questionnaire was organised into five conceptual layers. This is not the order in which the questions were asked:

  • Demographics: items that describe people and which we are not trying to change: their age, gender and sex at birth, education and financial coping; where they live; migration history; perceived ethnic minority status; employment; sexuality; relationships; engagement in sex work.

  • Morbidities: items about the health outcomes we are trying to change: mental health (anxiety/depression, suicidal thoughts, sexual unhappiness and alcohol dependency); diagnosis of infections (HIV, syphilis, gonorrhoea, chlamydia, ano-genital warts, hepatitis A, B, C).

  • Behaviours: items on the things that people do that directly contribute or detract from the morbidities above: sex (first and most recent sex with men; sexual behaviour in the last year with steady and non-steady male partners, intercourse risks with non-steady partners, sex with women); their last sexual encounter with a non-steady male partner; psychotropic substances used and injecting of drugs; combining sex and drugs; PEP (post-exposure prophylaxis) use; PrEP use; testing for HIV and taking treatment; disclosing homosexual behaviour to STI services; disclosing STI diagnosis to recent sex partners.

  • Needs: items about opportunities, capabilities and motivations to engage in the behaviours above: Social Provisions Scale (SPS) and Short Internalised Homonegativity Scale (SIHS); safer sex capability; condom access; drug use motivation; PEP knowledge and access; PrEP knowledge and access; HIV testing and treatment needs; viral hepatitis knowledge and vaccine access; bacterial STI prevention knowledge.

  • Interventions: items about the actions of others that meet or undermine the needs above including homophobic abuse; sources of condoms; drug services; PrEP-related services; HIV/STI education services; HIV testing services; offers of viral hepatitis vaccination; hepatitis vaccination status; STI testing services.

The layers form a putative causal pathway (interventions influence needs, needs determine behaviours, behaviours cause health outcomes). We used this framework to design the survey and to balance the number of items across different areas. However, it was not the purpose of the survey to test the relationships between the conceptual elements but to identify significant behavioural targets and to reveal the unmet needs that might fruitfully be addressed.

Question Formats

A further tension in survey design was between many concepts measured simply (a single question to measure it), or fewer measured more precisely (with each concept requiring multiple questions). Our collaborative approach to survey design meant there were very many concepts (issues, concerns, questions) for inclusion. On the other hand, increasing the validity and precision of measurements was also a pressure. We took a pragmatic approach, mixing validated scales with single items that had been asked in EMIS-2010.

As in EMIS-2010, many event-based questions used a recency format where men are asked ‘When did you last X?’ and offered a range of increasing periods in the past. Data can be used to give the proportion of the group who had done X within a given period (for example, a month, a year), as well as indicating those who had ever done X. The format does however not distinguish frequency at the individual level (that is, the group indicating ‘within the past 24 h? includes those who do it every day and those who did it last night for the first time in a long period or ever). The recency format gives a sense of the density of an activity in the population.

The final questionnaire allows for the generation of seventeen ECDC Dublin Declaration Monitoring (DDM) indicators (ECDC, 2019a) and two Global AIDS Monitoring (GAM) indicators (UNAIDS, 2017) related to HIV/STI among MSM. We collaborated with ECDC to decide which DDM indicators could be constructed using EMIS-2017 variables.

Translations

Translations were outsourced using EMIS Network connections. Translators in the 31 core countries (i.e. excluding UK and Ireland) were offered payments (only some of which were taken up). Translations were carried out directly online, using the survey hosting software to display the English version on the left half of the screen and a duplicate on the right half, which was over-written with the translation. This process minimised routing errors and copy-and-paste errors.

Translators were provided with the EMIS-2010 questionnaire in their language and directed to where questions were being used again or modified. They were asked to use identical translations as in 2010 for the 26% (107/409) of data items that were identical (unless there was good reason to change it), and to make comparable changes to the EMIS-2010 translation for the 19% (78/409) that were similar, but not identical to 2010.

National partners checked the translated versions for routing between questions, and the study team checked that the data were saved in an identical format across all languages. National partners confirmed the terminology fitted with their perceptions of the norm for the target group in their country, reviewed the final survey, and signed-off the main language version for their country.

We involved several multi-language proof-readers to compare the translations with the English original and with each other (for example comparing French, Spanish and English; or Czech, Slovak, Polish and English; or Russian, Ukrainian and English). The proof-readers ensured a harmonised multi-language questionnaire while deliberately maintaining certain differences, identified as culturally appropriate, such as explicitness of language, or addressing respondents formally or informally. In all languages, simple rather than specialised terms were preferred.

Ultimately, the survey was available in 33 languages, including 23 of the 24 official languages of the EU: Bulgarian, Bosnian/Croatian/Serbian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish and Swedish (it was not available in Gaelic Irish). It was also available in 10 other languages: Norwegian; Russian (minority language in Poland, Lithuania, Latvia and Estonia and most common immigrant language in the EU); Turkish (minority language in Bulgaria and second most common immigrant language in the EU); Arabic (the most common language among refugees coming to Europe since 2014); Albanian; Hebrew; Macedonian; Ukrainian; and two Filipino languages (Cebuano and Tagalog). Providing our questionnaire in right-to-left languages such as Arabic and Hebrew was challenging, and required late stage software adaptation.

Once the translated surveys were signed off we shared the Spanish, Portuguese, and Dutch version, visual identity and promotional materials with another group of researchers, public health professionals and gay/HIV community organisations working across Mexico and the Central and South Americas. We facilitated their opening an account with our survey application supplier (Demographix©) and copied over the three language versions of the survey. Thereafter the second collaboration conducted an entirely separate project, rebranding EMIS as LAMIS (Latin American MSM Internet Survey) and successfully recruiting 64,655 qualifiers across 18 Latin American countries.

Recruitment

Identifying Recruitment Settings

In terms of primary recruitment EMIS-2017 could not replicate what had happened in EMIS-2010. In 2010 geo-spatial ‘dating’ smartphone applications (henceforth apps) were only just being launched and the majority of recruitment occurred via websites with completion occurring on computers. By 2017, smartphone apps were widely felt to be the way most MSM made contact with each other.

We planned to promote EMIS-2017 by advertising on: (1) websites of supportive organisations (national/trans-national, public/commercial/NGO HIV/LGBT etc.); (2) general population social networking sites (such as Facebook, Twitter and Instagram); (3) MSM targeted ‘dating’ smartphone apps and web applications (henceforth, apps).

From March 2017 we used multiple online search strategies to identify how many apps in the European marketplace were specific to MSM (including sub-groups or sexual sub-cultures). We identified 38 apps, available in English and at least one other language. These included apps that ran on mobile devices with GPS (often known as mobile apps). Though not technically mobile apps, we included cross-platform web applications, accessible via any web browser, that allowed users to set a specific location and order search returns by distance.

We undertook online consultation about promotional strategies in specific countries. In May 2017 we emailed all 412 unique addresses in the EMIS Network to report on the survey progress and invite input to the promotion strategies via a short questionnaire. Partners were asked which apps/sites they recommended for promotion in their country using free text responses followed by selection of five from a prompted list of 38 multi-lingual apps. We also asked about preferred general social media and networking platforms and any other websites which they thought important for reaching MSM in their country.

We received 54 complete survey responses, representing 38 target countries. Responses established the likely top ten apps for EMIS recruitment and their country variation. We also received intelligence on other significant websites and social media and multiple offers of support in national recruitment.

Advertising Management

EMIS-2017 study staff commissioned advertising from ten multi-country online platforms: three which we agreed would use all 33 languages and recruit in all 50 target countries (PlanetRomeo, Grindr, Hornet) and seven focussed on specific sub-sets of countries and languages (Qruiser, RECON, Scruff, Gaydar, Manhunt/Jack’d, GROWLr and Bluesystem), based on their popularity in those countries.

Prior to and during fieldwork, study-staff supported leads to contact national webmasters and reach agreements about feasible promotional activities and any fees payable. Our objective was maximum visibility for minimum investment. For advertising fees, study-staff liaised directly with webmasters and drew-up precise contracts. Local partners covered national advertising fees where possible.

All online promotors of EMIS, paid or unpaid, were allocated a specific URL to use in online advertisements. This directed potential respondents to the EMIS landing page, so they could make their language selection and proceed to the survey. If the respondent continued with the survey, their first data item was the URL they had come to the survey through (the source code). We tracked in real time which websites or apps respondents were recruited from, allowing us to monitor advertising success (and failure) on a daily basis and make adjustments to our strategy as required.

Online promotional materials used the basic identity plus additional words in a language appropriate for the country. These were adapted to the size and resolution requirements of each platform. The additional words broadly followed those used successfully in EMIS-2010. The core slogan in English was “Be part of something huge!” (a phrase hopefully intriguing and mildly sexually suggestive). National partners could use these texts for recruitment or modify them.

EMIS Network country leads co-ordinated national advertising, requesting materials which were supplied by the study-staff. The study-team produced 204 different buttons and banners in 31 languages and provided several national leads with source imagery to produce their own buttons and banners. We did not develop any printed (offline) promotional materials but a small number of national leads created their own.

(Soft) Launch

Following testing of translations by Network members, we ‘soft-launched’ the survey on 13 October 2017. With minimal promotion, these first few days of recruitment were the final test that the online system served the questionnaire in all languages and consolidated the incoming data from all language versions into a single database. The research team submitted a number of fake cases, which were flagged as tests and removed before analysis. The soft launch recruited 204 qualifying respondents and revealed only one small routing error that was remedied without interrupting the survey.

We did not try to ‘make a splash’ with the launch, rather to slip-in to potential participants’ cyber-environment and expand our presence over time. Repeat exposure in different settings and opportunity to participate were our objectives. Active promotion began on the 18 October 2018 in Swedish and on 19 October for all other languages. The survey was open for submissions for 15 weeks from 13 October 2017 to 31 January 2018. No incentives were offered to participants beyond the opportunity to be heard. From the start, submissions were monitored on a daily basis and recruitment tactics were revised weekly to maximise return on investment and geographic spread of recruits.

Results: Response Rates and Geographic Distribution

Returns and Exclusions

At the close of fieldwork there were 144,305 cases in the consolidated data file. To be included in the dataset all cases had to have completed the whole questionnaire and pressed submit at the end, or have been exited from the survey early because their answers revealed they did not qualify. Partial cases were not recorded if they exited the survey early by closing their web browser, as they chose not to submit their data. Individuals could feasibly have participated more than once, submitting repeat or duplicate responses, but there was no obvious incentive to do so, and any measures we put in place to counter this, such as recording TCP/IP addresses, might have compromised participant anonymity.

Of all cases submitted, 95.2% (137,358 out of 144,305) were qualifiers. Figure 1 shows how we excluded cases by applying inclusion criteria.

Fig. 1
figure 1

EMIS-2017 returns and exclusions

Full size image

Recruitment Rates by Country

Table 1 shows the number of qualifying cases by country, with summary figures (in red) for: all countries; countries in the EU Health Programme 2014–2020; EU member states; EFTA member states not in the EU; EU Enlargement Area countries; and European Neighbourhood Policy (ENP) countries.

Table 1 EMIS-2017 Qualifiers and recruitment rates
Full size table

For each country and the aforementioned summary measures for country groups we estimated crude recruitment rates for each by dividing the number of respondents (qualifying cases) by the adult male population (15–65 years). For all recruits the overall recruitment rate was 4.3 respondents per 10,000 adult males in the population (mean 6.4, median 5.6). For countries, this ranged from 0.7 (for Turkey) to 20.1 (for Malta).

Table 1 also shows the percentage of the adult male population in the country that used the internet (International Telecommunications Union, 2019), which is almost ubiquitous across Northern and Western Europe but somewhat less common in parts of the East and South. Adjusting for the proportion of people in each country using the internet did narrow the distribution of recruitment rates (in fact the standard deviation increased from 4.3 to 4.8) but increased the overall recruitment rate to 5.3 (mean 7.7, median 7.7) with a range from 1.1 (Turkey) to 25.1 (Malta).

The countries with the highest recruitment rates tended to have relatively small overall populations (Malta, followed by Norway and Sweden) and to have active gay communities and strong involvement in survey promotion. Lowest recruitment rates were in countries where promoting a survey for gay men was illegal or at least highly stigmatised (Belarus, Turkey, Russia, Ukraine). In some countries a somewhat lower recruitment rate than might have been expected was probably attributable to competing MSM surveys directly before or overlapping with EMIS, including Denmark, Finland, Portugal, Slovenia and Ukraine. The relative low recruitment rate in the Philippines may have been due to poorer internet connectivity, a more fragmented gay community and less experience with MSM surveys.

Demographix® capture metadata on the type of device used for survey completion. This information did not come from respondents answering a survey question. Overall, two thirds (67%) completed the survey on a mobile phone. There was a highly plausible association between completion on the telephone and age, with twice as many men under 20 years old using a phone compared to men over 60 years old. The proportion of qualifiers using a smartphone for survey completion in each country ranged between 39.2% (Moldova) and 94.6% (Lebanon, see Table 1).

EMIS Returns from International Recruitment

All online promotions of EMIS (paid and unpaid) included a unique ‘source code’ at the end of the URL, that indicated the source of recruitment for each individual response. Ninety per cent of qualifying cases had a source code. The source code allowed us to monitor advertising impact and to adjust spending to maximise recruitment spread. It also revealed the number of unique recruitment sources there are in addition to the ten paid-for recruitment sites described below.

The proportion of each country sub-sample coming to the survey through each of the main recruitment routes varied substantially by country (see Table 2).

Table 2 Recruitment sources
Full size table

PlanetRomeo recruited 39.1% of all qualifiers (n = 49,924). They charged a nominal fee for their advertising. The primary promotion was a substantial direct message to all members in all target countries and in every language. Secondary promotion was banner advertising.

After 5 weeks of recruitment through PlanetRomeo we started Grindr advertising, which then ran to the end of fieldwork. This was the second most common recruitment source, the route-in for 20.3% of qualifiers (n = 25,979). Grindr discounted their advertising. Grindr recruited over 50% of qualifiers in 4 countries and 25–50% in 8 others. While all countries received targeted Grindr advertising, we focussed repeated (weekly) adverts on countries with fewer recruits (relative to their population). This allowed for highly efficient use of our promotion budget.

The third single recruitment setting bringing about 5% of qualifiers was Hornet (n = 7013). Hornet discounted their advertising. The primary promotions were in-app broadcasts. Advertising began after PlanetRomeo and ran alongside Grindr in weeks 6 to 10. Geographic targeting was hard to achieve and responses varied substantially between countries.

Other multi-country dating platforms we advertised with were Qruiser (3.3%, n = 4160); RECON (1.9%, n = 2458); Scruff (1.6%, n = 2080); Gaydar (0.9%, n = 1123); Manhunt/Jack’d (0.5%, n = 675); GROWLr (0.5%, n = 625) and Bluesystem (0.2%, n = 214). These seven sites were considered important by partners in at least two countries but fewer than half of all countries. The percentage of qualifiers from these seven settings is combined into a single column in Table 2.

EMIS Returns from National Recruitment

Many EMIS partners actively promoted EMIS-2017 via their own websites and social media channels and encouraged other organisations in their country to participate in recruitment.

There were at least 282 different recruitment sources into EMIS-2017 (including the 10 transnational sites discussed above). Among the 272 known national promotions: 156 (57%) were websites; 84 (31%) were Facebook pages or groups; 18 (7%) were Twitter accounts; 9 (3%) were email lists; and 5 (2%) were Instagram accounts. The vast majority of these online promotions, including the Facebook and other social media promotions were free. In relatively few cases (< 10%) EMIS partners boosted or targeted posts via payments to the service provider.

In Table 2 we differentiate national recruitment using generic social media (Facebook, Twitter, Instagram) from more traditional website and email list recruitment. In total 16% of the sample were recruited from these national sources, including 7% recruited via social media (Facebook, Twitter, Instagram) and 9% recruited via websites and email lists of partner agencies.

As Table 2 illustrates the contribution of national recruitment to total recruitment by country varied enormously, from < 1% in Italy to 83% in Norway and 88% in Moldova. Apart from Moldova and Norway, national partners recruited more than half their national sample in two other countries (Denmark and Slovenia); more than a third of their entire sample in five more countries (Malta, Croatia, Canada, Republic of Ireland and Lithuania); and more than a quarter of their sample in nine others (Finland, Israel, Iceland, Philippines, Cyprus, Hungary, Slovakia, Estonia, UK).

Datasets

The data were divided into national datasets based on current country of residence, regardless of the language used to complete the survey or country of birth, and also combined into a pan-European dataset. National datasets are available for all 43 countries with 100 or more qualifying cases. To date we have signed Data Transfer Agreements with organisations in 38 of these countries and supplied them with a full data-set for all recruits resident in that country.

Seven countries did not reach 100 qualifying cases. For the four smallest of these countries, because a viable sample size was never expected, cases are included in the datasets of adjoining or surrounding countries: Andorra (18 cases) was included in both the France and Spain datasets; Liechtenstein (5 cases) was included in the Switzerland dataset; Monaco (8 cases) was included in the France dataset; and San Marino (4 cases) was included in the Italy dataset.

Three other countries are not included in any national dataset: Albania (55 cases), Montenegro (77 cases) and Kosovo (39 cases).

Discussion

The EMIS-2017 online survey collected comparable data from MSM in 33 languages, promoted on at least 282 trans-national apps and on the social media platforms and websites of a range of mainly non-governmental organisations. We have previously demonstrated the feasibility of such a large multi-country, multi-language online-only survey (Weatherburn et al., 2013) and the validity of the resultant self-reported HIV prevalence data (Marcus, Hickson, Weatherburn, & Schmidt, 2012). In this survey we demonstrated that such an approach remains viable in the age of the smartphone, when internet use does not routinely involve computer access and is cheaper, simpler and a far more mundane aspect of everyday life. We also demonstrated for the first time that it is feasible to include right-to-left languages (like Arabic) in online surveys, and to extend the geographic scope beyond one continent.

Ultimately, we recruited 112,732 qualifying respondents in the core 33 countries and 137,358 in all the countries covered. In terms of eligible returns it was the second largest MSM survey ever, since EMIS-2010 recruited 181,434 men (166,579 in the same 33 core countries). The smaller overall return arises from slightly different priorities. Here we did not chase higher absolute numbers of recruits after week six of recruitment, but focussed advertising spend instead on reaching viable sample sizes in the countries with smallest overall populations (such as Malta, Cyprus, Iceland) and the countries with less gay-infrastructure and a less permissive social and legal environment (such as Romania, Poland, Ukraine, Russia, Turkey). Compared to 2010, EMIS-2017 has more countries with over 100 qualifiers (43 countries); more with over 300 qualifiers (35 countries); more with over 1000 qualifiers (30 countries); and more with over 3000 qualifiers (13 countries).

Of course, the volume of recruits is also a function of advertising expenditure, which was 55,000 euros, somewhat more than in EMIS-2010. It is worth noting that we do not feel we reached recruitment saturation in any country, especially those with a substantial gay infrastructure, a large overall population and a relatively permissive environment (such as Germany, UK, France, Italy, Spain). For those researchers seeking higher numbers of overall recruits a shorter, more focussed survey would be better, as the majority (67%) completed EMIS on a phone, which was not straightforward when average completion time was 20 min and men were served adverts away from their home, when they had priorities other than completing a lengthy survey.

EMIS national datasets are already being analysed by partners in 38 countries, with a view to a common understanding of sexual health needs. A viable national sample of MSM is particularly valuable for countries in which gay communities and HIV prevention responses are less well established and where policy supporting MSM HIV prevention might be less developed. The cost savings on generating data in this way were considerable.

The coverage and breadth of recruitment achieved were the result of meaningful involvement of a large number of collaborators in the survey design, and in its promotion. Funders of surveys can foster such positive outcomes through encouraging projects to include a range of stakeholders, to make explicit their research values and objectives and to define transparent development and decision making processes. In our international collaboration, flexibility in accommodating a range of differing national needs was essential to maintaining the network and to the appropriateness of the survey across a range of settings.

The three principal sources of bias in the data are uneven access to the internet across countries, self-selection bias in the recruitment process (Evans, Wiggins, Mercer, Bolding, & Elford, 2007; Mercer et al., 2016; Prah et al., 2016) and the validity of precise comparisons between different linguistic version of the questionnaire.

Access to the internet is increasingly common worldwide, but remains more widespread in the western and northern parts of Europe than in eastern and some southern parts (International Telecommunications Union, 2019). Table 2 shows in only four of our target countries did less than two thirds of adult men use the internet at the time of the survey (Ukraine, Philippines, Italy and Romania). While the samples from these countries may be smaller than hoped, and perhaps less representative, access to the internet is a far less of a problem than when EMIS occurred in 2010.

While there are no reasonably sized representative samples of MSM to which EMIS respondents could be compared, it is unlikely that they are truly representative. In the last decade the internet has become an important setting for recruiting large samples of homosexually active men. These online samples have been demonstrated to be more diverse in terms of age, education, sexual identity and geographic distribution than those recruited through gay community settings and, for the men who have sex only with men, are demographically broadly representative (Prah et al., 2016). However, compared to all MSM, those recruited opportunistically through gay dating sites are younger, more sexually active, engage in more sexual risk and sexual precaution behaviours, and encounter more sexual harms (Evans et al., 2007). Since this is the sub-set of the population that prevention programmes aim to serve, these are acceptable recruitment biases. In addition, in a survey of this scale national samples of MSM may be more representative of migrant MSM than national convenience surveys because all men could complete the survey in any of the 33 languages.

Finally, usability testing was limited to the English language version, which went through two rounds of pre-test and a time-trial. Translations concentrated on comparable meanings and were rigorously cross-checked by translators and national partners but were not pretested. We acknowledge that the exporting of survey questions from one cultural and linguistic group to another makes them vulnerable to cultural incompatibility and translation problems and that to ensure that translated items are equivalent to original versions, both statistical and qualitative analyses are necessary (Ware, Keller, Gandek, Brazier, & Sullivan, 1995).

Careful design, piloting and presentation ensured the survey was acceptable and had both authority and perceived community benefit. To reach substantial coverage across Europe, a patchwork of recruitment was required across a large number of commercial and community partners. A high degree of partner involvement in recruitment and substantial commitment of the commercial partners were crucial. Recognising and accommodating the range of uses a diverse group of stakeholders find for community-based surveying was central to maintaining this wide-ranging collaboration.