Article Text

Field evaluation of the performance and testing costs of a rapid point-of-care test for syphilis in a red-light district of Manaus, Brazil
1. A S Benzaken1,
2. M Sabidó2,
3. E G Galban1,
4. V Pedroza1,
5. F Vasquez1,
6. A Araújo3,
7. R W Peeling4,
8. P Mayaud2
1. 1
Fundação Alfredo da Matta, Manaus, Brazil
2. 2
London School of Hygiene & Tropical Medicine, London, UK
3. 3
Escola Nacional de Saúde Pública/ENSP/FIOCRUZ, Manaus, Brazil
4. 4
World Health Organization, Geneva, Switzerland
1. Meritxell Sabidó, París 179-181 1-1B, Barcelona 08036, Spain; xellsabido{at}gmail.com

Abstract

Objectives: To assess the performance, usefulness and cost of a rapid treponemal antibody assay (VisiTect Syphilis) to detect syphilis in high risk populations.

Methods: People who attended STI clinics in Manaus, Brazil, were screened for syphilis using the fluorescent treponemal antibody absorption (FTA-Abs) test and a non-treponemal test (Venereal Diseases Research Laboratory (VDRL)), and for HIV. Finger prick blood samples were tested with VisiTect Syphilis. The rapid test was evaluated against the reference FTA-Abs and for its usefulness in detecting active syphilis (FTA-Abs and VDRL positive). Operational performance was assessed through providers’ and patients’ interviews. An economic evaluation was conducted from the provider’s perspective.

Results: 510 patients (60% men) were enrolled, of whom 13 (2.5%) were HIV-1 seropositive. Syphilis prevalence (FTA-Abs) was 18% and active syphilis prevalence was 7.5%. 11% (57/506) of samples were positive by VisiTect. The sensitivity, specificity, positive and negative predictive values of VisiTect Syphilis were 57% (95% CI 45.8 to 66.7), 99% (95% CI 97.0 to 99.6), 91% (95% CI 80.0 to 96.7) and 91% (95% CI 88.0 to 93.5), respectively. VisiTect Syphilis identified 79% (30/38) of active syphilis cases. The cost per case of syphilis was $16.8 for VDRL,$33.2 for low cost and $56.3 for high cost VisiTect Syphilis; the cost per case of active syphilis was$21.3, $57.5 and$97.6, respectively. Patients identified finger prick pain and preference for venous blood collection as minor barriers to test use.

Conclusion: VisiTect Syphilis had low sensitivity in field use and was less cost effective than conventional VDRL. However, rapid and correct identification of a high proportion of active syphilis cases combined with operational characteristics suggest a role in high risk populations.

Statistics from Altmetric.com

Sexually transmitted infections (STIs) remain a major public health problem in many countries.1 Epidemiological studies have shown the important role of sex workers and the populations who interact with them as “core” groups and “bridging” populations, respectively, in the epidemiology of STIs and HIV.2 3 STIs have proven to be among the most important modifiable risk factors of HIV transmission in these populations.2 48 Thus, efforts should focus on the rapid identification and management of treatable STIs.9 Screening programmes have been effective in reducing syphilis rates10 11 and could prove effective in reducing HIV transmission in high risk groups.8 However, routine implementation may be hampered by operational and technical difficulties.12 13 These include the lack of a rapidly available results obtained from cheap, reliable Treponema pallidum specific tests, which could be easily used under field conditions. The current specific diagnostic assays for syphilis (for example, the fluorescent treponemal antibody absorption (FTA-Abs) test and the T pallidum haemagglutination or particle agglutination assays (TPHA, TPPA)) must be performed in the laboratory.14 They need at least 24 hour turnaround time from specimen collection to result, resulting in delayed treatment or low return rates.15 Furthermore, the interpretation of syphilis serological tests in populations where syphilis prevalence is high can be difficult.16 The alternative non-treponemal assays (for example, the Venereal Diseases Research Laboratory (VDRL) or rapid plasma reagin (RPR) tests) identify cases of recent syphilis but are prone to false-positive reactions and still require basic equipment and refrigeration facilities.13

The answer may lay in rapid point-of-care (POC) treponemal tests in a dipstick or cassette format, which can be stored at room temperature, require no equipment and minimum training.17 These tests allow onsite screening and treatment, which may improve the effectiveness and coverage of control programmes. In previous laboratory-based evaluation studies, some of these tests have shown comparable performance to gold-standard treponemal assays.14 18 POC tests have been evaluated under field conditions1924 with some using whole blood from finger prick specimens.19 2124 Such minimally invasive tests can potentially be incorporated into screening programmes at community-based venues, at clinics lacking laboratory facilities and in outreach programmes for high risk populations. Their evaluation is considered an STI research priority.14

The aims of this study were to evaluate, in an outreach clinic for high risk populations in Brazil, the field performance of a rapid POC treponemal test using whole blood finger prick specimens against the FTA-Abs assay as a gold standard; its usefulness and cost compared to the non-treponemal VDRL test; and its operational suitability from both the providers’ and clients’ perspectives.

METHODS

Study setting and populations

The Sexually Transmitted Diseases Diagnostic Initiative (SDI) of the World Health Organization (WHO) has evaluated six rapid POC syphilis tests using stored sera at eight laboratories around the world.14 Our study was part of the SDI’s second phase of assessments, during which the best performing tests were evaluated in various populations and settings outside reference centres.

The study was conducted by the Fundação Alfredo da Matta (FUAM), which runs a reference outpatient hospital specialised in STI care in Manaus, Brazil—the largest city in the Amazon Region—and an outreach clinic offering sexual health services in the city harbour area next to a large red-light district. Clinic staff were trained to perform the rapid POC test.

Study procedures

Between May and October 2006, consecutive unselected patients with no age restriction presenting at the FUAM harbour clinic who showed interest in being screened with the rapid POC VisiTect Syphilis test (Omega Diagnostics, Alloa, Scotland) were invited to participate in the study. We included male and female sex workers, sex worker clients, and other patients living and working in the area who accessed the clinic. A sex worker client was defined as those men or women who reported having had sex with a male or female sex worker in the last 12 months.

Target populations were informed about the study aims during an advertising campaign and received further information about study procedures at the clinic. Participation included the provision of informed signed consent to complete a short questionnaire and submission of a finger prick sample. Participants completed a 10 minute interviewer-administered questionnaire, which included sections on sociodemographic indicators, substance abuse, sexual behaviour, sexual and reproductive health history, including history of syphilis testing and treatment, and presence of current STI symptoms.

A laboratory technician collected 20 μL finger prick capillary blood sample for onsite testing with VisiTect Syphilis test. Results were ready within 20 minutes. Tests were considered negative if only one of the control lines on the cassette was visible, positive if both lines were visible and void’ if no lines were visible, in which case another sample was collected and procedures repeated. In addition, an 8 mL venous blood sample was drawn for syphilis and HIV serologies, which were performed at the FUAM reference laboratory. Participants with a positive POC test were treated immediately with one intramuscular injection of 2.4 MU benzathine penicillin.

Laboratory procedures

At the end of each clinic, venous blood samples were transferred to the FUAM laboratory. A laboratory technician, blind to the POC test results, tested sera using the FTA-Abs (WAMA Diagnostica, São Paulo, Brazil) and VDRL (Winer Laboratorios, Rosario, Argentina), and VDRL titres were determined. The reference standard test for syphilis was FTA-Abs. This widely used test was considered the most appropriate comparator since the rapid POC test was also Treponema specific.25 Active syphilis was defined as sera reactive with both FTA-Abs and VDRL, with high and low titre active syphilis defined based on VDRL titres ⩾ or <1:8, respectively.

HIV-1 and HIV-2 serum antibodies were detected by GENSCREEN HIV 1/2 v.2 ELISA (BIORAD, Rio-de-Janeiro, Brazil). Positive samples were confirmed by a second ELISA (GENSCREEN PLUS Ag/Ab, BIORAD, Rio de Janeiro, Brazil) and an indirect anti-HIV-1 immunofluorescent assay (Biomanguinhos, Rio de Janeiro, Brazil). Discrepant results were resolved by Western Blot (GLD HTLV BLOT 2.4, Genelabs Diagnostics, Singapore).

Evaluation of operational characteristics

To identify factors that might influence test acceptability and operational performance, both staff (10 clinical staff and 2 laboratory technicians) and participants attending the health facility for two consecutive weeks (n = 60, 60% female) were interviewed. Questionnaires were administered to patients while they were waiting for their results. Information was gathered on financial, time and other potential barriers to testing, and about the participants’ acceptability of the test, their experience of discomfort and their confidence in the results the test would provide.

Economic analysis

We compared the costs of screening and managing serological syphilis using the different tests. Incremental recurrent costs only were estimated, assuming all other costs would be equal. Costs of infrastructure, training and supervision were not included. Direct costs were labour, diagnostic supplies and drugs, which were estimated using the ingredients approach in which the total quantity of goods and services used were estimated and multiplied by their respective unit prices.26 Cost of labour was determined through an observational time allocation study and time units were multiplied by the relevant salary units of staff performing various tasks (clinician or laboratory technician). The costing exercise was conducted from the provider’s perspective, considering FUAM as the sole provider. Financial and economic costs were the same because there were no donated goods or services. Costs of tests included the costs when procured by WHO at a discounted price, and when procured by the Brazilian Ministry of Health (MOH) for the year 2006, and converted into US dollars ($) using the fixed exchange rate of 2.14 Reais per$. All research-related costs were excluded. Costs of diagnostic inputs, as well as formulas for economic evaluation, can be found in table 1. Total testing costs were divided by the number of people tested (or retested in the case of VisiTect indeterminate results) to obtain the cost per case identified by test, and were divided by the number of relevant laboratory confirmed cases to obtain the cost per case of syphilis (people testing positive and being FTA-Abs positive) or per active case of syphilis (people testing positive and being FTA-Abs and VDRL positive).

Table 1 Economic evaluation of syphilis serological testing using VisiTect Syphilis or VDRL in high risk populations in Manaus, Brazil

Statistical analysis

Data were analysed using Epi-Info 8.0 (CDC, Atlanta, GA, USA). The performance characteristics (sensitivity, specificity, and positive and negative predictive values) of the POC test against the gold standard (FTA-Abs) were calculated according to standard methods. Results were presented with their 95% confidence intervals (CI).

RESULTS

A total of 510 consenting patients attending the Manaus harbour STI clinic were included. The characteristics of study participants by patient group are shown in table 2. The majority (304, 60%) were men. Male sex workers were younger than male sex worker clients and other men, with a median difference of 18 and 16 years, respectively. Women across groups had similar ages. Education level was low especially among male and female sex workers. Although a large majority (>70%) of non-sex worker male and female patients reported having permanent partners and large proportions (37–75%) also reported having casual partners in the previous year.

Table 2 Population characteristics, harbour sexually transmitted infection (STI) clinic, Manaus, Brazil

HIV-1 seroprevalence was 2.5% overall (2% in men, 3% in women). The results of serological markers of syphilis by test used and patient category is shown in table 2. Treponemal antibody prevalence (FTA-Abs positive) was 18% overall (23% in women, 15% in men) highest in the sex worker populations and lowest in the other patients’ category for both sexes. Prevalence of other serological markers followed similar trends.

Performance and costs of POC syphilis test

Four patients were excluded from subsequent analyses because of missing FTA-Abs result. We categorised syphilis serology according to the combined results of FTA-Abs and VDRL testing (table 3). Overall, 23% (47/205) of women and 15% (45/304) of men had evidence of syphilis (FTA-Abs positive), with 10% (21/205) women and 6% (17/304) men having evidence of active syphilis (dually positive for FTA-Abs test and VDRL). Only 38% (8/21) of the women and 24% (4/17) of the men with active syphilis had a previous treatment for syphilis, which might have included penicillin injections.

Table 3 Syphilis serological status by sex, VisiTect Syphilis result and reported syphilis treatment in Manaus, Brazil

After retesting of initially indeterminate results (n = 17, 3.3%), 11% (57/506) of samples were positive by VisiTect Syphilis. The POC test detected 57% (52/92) of syphilis cases and 79% (30/38) of active syphilis cases. The sensitivity of VisiTect Syphilis compared to FTA-Abs was 57% (95% CI 45.8 to 66.7), specificity 99% (95% CI 97.0 to 99.6), positive predictive values 91% (95% CI 80.0 to 96.7) and negative predictive values 91% (95% CI 88.0 to 93.5).

Table 1 shows the costs of syphilis detection using the two available screening tests at various purchase costs for VisiTect Syphilis and for different definitions of syphilis serostatus. The cost per case of syphilis was $16.8 for VDRL,$33.2 for low cost and $56.3 for high cost VisiTect Syphilis, whereas the cost per case of active syphilis was$21.3, $57.5 and$97.6, respectively.

Operational characteristics

Clinic staff (n = 12) found VisiTect Syphilis easy to use (75%) and to interpret (67%), although only half trusted its results. The most cited reason was the perceived inability of the test to differentiate between old and recent syphilis leading to over-treatment, particularly of patients who might repeatedly attend the clinic. Staff felt they needed to back up their clinical decisions by incorporating the patient’s syphilis testing or treatment history or add the results of other tests, such as the VDRL. Another reason was staff experience of frequent discordant results between VisiTect Syphilis and either the VDRL or the FTA-Abs assays.

Few of the 60 participants identified waiting time (7%), cost of transport (10%), opening hours (25%) or lack of trust in POC test results (3%) as obstacles for being tested for syphilis. Pain caused by finger prick (57%) and preference for venous blood collection (38%) were identified as minor barriers to rapid POC test use.

DISCUSSION

Our field evaluation of the rapid POC VisiTect Syphilis showed a low sensitivity compared to FTA-Abs in populations at high risk of syphilis in Manaus. This contrasts with excellent results obtained in previous laboratory based evaluations15 and validation in reference STI clinic settings of this and similar assays.16 19 21 24 In the latter studies, rapid POC tests compared favourably with standard treponemal tests, with high specificities (>95%) and adequate sensitivities (range 64–100%), which were, however, lower when whole blood finger prick samples were used rather than serum.24 In our previous evaluation of VisiTect Syphilis against FTA-Abs at the FUAM reference STI clinic using whole blood, and at the FUAM laboratory using serum, we found that the POC test had high sensitivity (96.1%) and specificity (98.5%).24 Such promising results have prompted large scale evaluation of POC tests in alternative settings and, if confirmed, may lead to the recommendation of their use in syphilis control programmes.17 It is possible that more adverse field conditions (greater variability in temperature and humidity) might affect the test performance, although it is claimed not to require refrigeration. As in other studies,19 21 the POC test was found operationally suitable but it was less cost effective than conventional VDRL to identify patients who needed to receive syphilis treatment. Whatever the definition of syphilis case that was adopted, VisiTect Syphilis appeared at least twice as expensive as VDRL.

One of the major drawbacks of all Treponema specific tests, including current POC tests, is their inherent inability to distinguish between old, already treated and active syphilis cases. The persistence of test reactivity is a problem for high risk populations such as sex workers who would attend STI clinics regularly and would either need to be treated every time or only receive treatment the first time with no possibility to detect new infections. In such instances, the use of a non-treponemal test such as the VDRL is recommended, but this would make use of POC tests for syphilis less relevant and not cost effective. POC tests for syphilis are better suited for populations who are only seen infrequently or who are difficult to access. In the case of high risk populations, the POC tests may be used in outreach programmes—for example, with street-based sex workers and possibly their clients. In our study, the high proportion of participants who had never tested for syphilis suggests that they have difficulty in accessing services, and use of the rapid treponemal test until they become positive would increase the effectiveness of syphilis control in these populations. In contrast, POC tests for syphilis appear more useful in prenatal populations in resource-constrained settings, where access to screening is a major barrier and the benefits of preventing serious sequelae clearly outweighs the risks and costs of over treatment.

Interestingly, VisiTect Syphilis identified a high proportion of active syphilis cases in this high risk population. By avoiding delays in treatment and reducing losses to follow up,27 this may provide considerable epidemiological advantages in hard to reach populations in whom high syphilis rates and lack of access to services may increase the risk of acquisition of HIV or onward transmission of both syphilis and HIV.8

Questions remain on the usefulness and frequency of screening using rapid treponemal tests in high risk populations, either alone or in combination with non-treponemal tests. There is also a lack of evidence about the performance of the tests depending on HIV serostatus. Because of the small numbers of HIV infected individuals, we were not able to verify this in our study. Other studies have found that the performance of herpes serological assays may vary in high or low HIV prevalence settings.28 Further studies are required to elucidate this hypothesis for syphilis POC tests, as this would impact on their practical use in high HIV prevalence settings.

In conclusion, the performance and cost effectiveness of rapid POC treponemal tests may differ according to the epidemiological setting. Further operational research is needed to assess the usefulness and impact of these tests in high risk groups. The development of tests that would detect cases of active syphilis with similar operational characteristics would help increase the effectiveness of syphilis screening programmes in all populations.

Key messages

• VisiTect Syphilis had a low sensitivity in field use.

• Under field conditions, its cost effectiveness might not surpass that of cheaper conventional tests in high risk populations who can be screened regularly.

• Rapid and correct identification of a high proportion of active syphilis cases combined with favourable operational characteristics support a role in high risk populations.

Acknowledgments

The authors thank the STI clinic and laboratory staff from the Fundação Alfredo da Matta (FUAM) for their technical assistance, and patients who volunteered to be study participants.

Footnotes

• Funding: The study was supported by a grant from the UNICEF/UNDP/World Bank World Health Organization (WHO) Special Programme for Research and Training in Tropical Diseases.

• Competing interests: None.

• Ethics approval: The study was approved by the Ethics Review Committee of the World Health Organization (WHO), the Research Ethics Committees of FUAM and the London School of Hygiene & Tropical Medicine.

• Contributors: ABS designed and supervised the overall study; MS designed and supervised the operational component of the study, conducted analyses and wrote the initial draft of the manuscript; EGG, AA, VP, FV conducted data analysis; RWP and PM were involved in study design, data interpretation and manuscript preparation. All authors revised and approved of the final contents of the manuscript.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.