Article Text

Download PDFPDF

Evaluation of seven rapid tests for syphilis available in Brazil using defibrinated plasma panels
  1. Maria Luiza Bazzo1,
  2. Leonardo Rapone da Motta2,
  3. Renata Cristina Messores Rudolf-Oliveira1,
  4. Alisson Bigolin1,
  5. Lisléia Golfetto1,
  6. Fábio Mesquita3,
  7. Adele Schwartz Benzaken4,
  8. Pamela Cristina Gaspar4,
  9. Ana Flavia Nacif P Coelho Pires4,5,
  10. Orlando da Costa Ferreira Júnior6,
  11. Miriam Franchini4
  1. 1 Department of Clinical Analysis, Health Sciences Center, Laboratory of Molecular Biology, Microbiology and Serology, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
  2. 2 Research Laboratory on HIV/AIDS, Center for Life Sciences and Health, University of Caxias do Sul, Rua Francisco Getúlio Vargas, Caxias do Sul, Brazil
  3. 3 Faculty of Medicine, University of São Paulo, São Paulo, Brazil
  4. 4 Department of Surveillance, Prevention and Control of Sexually Transmitted Infections, HIV/AIDS and Viral Hepatitis, Secretariat of Health Surveillance, Ministry of Health, Setor Administrativo Federal Sul, Brasília, Distrito Federal, Brazil
  5. 5 Faculdade de Medicina, Faculdade de Ciências de Saúde, Programa de Pós Graduação em Saúde Coletiva, Universidade de Brasília, Campus Universitário Darcy Ribeiro, Brasília, Distrito Federal, Brazil
  6. 6 Laboratory of Molecular Virology, Institute of Biology, Federal University of Rio de Janeiro, Avenida Carlos Chagas Filho, Rio de Janeiro, Brazil
  1. Correspondence to Dr Maria Luiza Bazzo, Laboratório de Biologia Molecular, Microbiologia e Sorologia Departamento de Análises Clínicas. Centro de Ciências da Saúde Universidade Federal de Santa Catarina Campos Universitário- Trindade Florianopolis, SC, Brazil; m.l.bazzo{at}ufsc.br, marialuizabazzo{at}gmail.com

Abstract

Objectives In 2012, the WHO estimated that 6 million new cases of syphilis per year would occur worldwide, including 937 000 in Brazil. Early diagnosis and treatment of syphilis are essential to reduce morbidity and prevent transmission. The availability of rapid tests (RTs) for this diagnosis means that testing can be performed more quickly, as a point-of-care test, even in non-laboratory environments and requires only simple technical training to antibodies detection. The objective of this study was to evaluate the performance and operational aspects of seven commercially available RTs for syphilis in Brazil.

Methods Seven rapid treponemal tests were evaluated for sensitivity, specificity, accuracy and Kappa value, according to a panel composed of 493 members. The operational performance of the assay was also determined for these tests.

Results The seven RTs showed sensitivity ranging from 94.5% to 100% when compared with the reference tests and specificity of between 91.5% and 100%. All the RTs evaluated presented good operational performance, and only one failed to present the minimum specificity as defined by Brazil’s Ministry of Health.

Conclusion All the tests presented good operational performance, and the professionals who performed them considered them to be easy to use and interpret. This evaluation is important for making informed choices of tests to be used in the Brazilian Unified Health System.

  • Syphilis
  • Antibodies
  • Diagnosis
  • Screening

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

Statistics from Altmetric.com

Introduction

Syphilis constitutes a major public health problem. In 2012, the WHO estimated that six million new cases of syphilis per year would occur worldwide.1 Between 2014 and 2015, Brazil experienced an increase of 32.7% in acquired syphilis and 21.0% in congenital syphilis.2

Early diagnosis and treatment of syphilis are essential to reduce morbidity and prevent vertical transmission (congenital syphilis), sexual transmission and further spread of the disease.3 However, the complexity of the natural history of syphilis and the absence in many cases of specific clinical manifestations are barriers to the implementation of surveillance programme in developing countries.4

Conventional syphilis testing is usually based on serological tests, including antibodies detection by treponemal and non-treponemal tests, performed in laboratories. These tests require the training of laboratory teams.5 In contrast, rapid tests (RTs), often referred as point-of-care tests, can be conducted by staff with minimal training at a community setting, rather than sending the sample to a laboratory. The WHO’s worldwide policy to eradicate the vertical transmission of syphilis has emphasised the importance of RTs in enabling pregnant women to access diagnosis.3 Therefore, the use of treponemal RTs is advocated for screening populations in less developed settings.

The RTs can be used in a variety of situations, such as in difficult-to-reach places, for diagnosing key HIV and indigenous populations and especially for priority testing and assisting pregnant women. The availability of RTs has made it possible for syphilis testing to be done in non-laboratory environments by professionals with no laboratory experience. The objective of this study was to evaluate the performance and operational aspects of seven RTs for syphilis that are commercially available in Brazil.

Methods

Sample panel

The members that comprised the panel used were obtained from bags of plasma that presented reagents and non-reagents for syphilis. These were donated in 2011–2013 by different blood banks in Brazil to the Laboratory of Molecular Biology, Microbiology and Serology (LBMMS) of the Federal University of Santa Catarina.

The selected samples were converted to serum (defibrinated plasma) by adding 1 mL of thrombin (100 U/mL) in 1M calcium chloride for each 100 mL of plasma, followed by treatment with 10% (wt/vol) Kaolin, according to Castro et al.6

The truly positive (127 samples that were reagent in all reference tests) and truly negative (366 samples that were non-reagent in all reference tests) syphilis plasma samples were fractionated into five aliquots of 350 µL each and stored at −70°C in the LBMMS, until the time of the evaluation. Each aliquot was thawed only once. Online supplementary table S1  presents the results of the panel characterisation with treponemal tests and the non-treponemal  Venereal Disease Research Laboratory (VDRL) test and the results of the RTs included in the study.

Supplementary file 1

Reference tests

The reference assays used by the LBMMS were: ARCHITECT System Syphilis TP (Abbott Laboratories, Wiesbaden, Germany), TPHA100 (bioMérieux, Marcy l’Etoile, France) and Immuno-COM FTA-ABS Syphilis (WAMA Diagnóstica, São Carlos, SP, Brazil). While the Fluorescent Treponemal Antibody Absorption test (FTA-ABS) results were adopted as the gold standard, the results obtained in the Treponema pallidum Hemagglutination assay (TPHA) and the chemiluminescence immunoassay tests were also all coincident with those obtained with the gold standard. The LBMMS participates in the WHO/Centers for Disease Control and Prevention (CDC) Syphilis Serology Proficiency Programme.

Treponemal RTs

In this study, we evaluated seven commercially available RTs (table 1), which are officially registered by Brazil’s National Health Surveillance Agency (ANVISA). Valid ANVISA registration, a mandatory requirement for marketing RTs in Brazil, was a major criterion for inclusion in this evaluation. Tests were performed according to the manufacturers’ instructions. The results were read by two independent professionals who were unaware of the results obtained in the reference tests. The kit manufacturers had no participation in the design of the study or the performance of the tests, and only learnt the results from a report disclosed at the end of the evaluation.

Table 1

Characteristics of the seven rapid tests evaluated according to the information provided by the manufacturers

Criteria for inclusion in algorithms recommended for use in Brazil

The Department of STI, HIV/AIDS and Viral Hepatitis (DIAHV) has established the following criteria for the selection and use of RTs on flowcharts recommended for syphilis testing. The test is required to: (A) have a sensitivity of 94.5% or more; (B) specificity equal to or greater than 93.0%7 8; and (C) an operational performance equal to or greater than 4 points out of a total of 5.9 10

Operational performance

The analysis of the operational performance of the trial was performed according to criteria previously described with an additional Ministry of Health (MoH)/Brazil criterion for RTs that should provide a result in less than 30 min.10 A score 0 (negative) or 1 (positive) was assigned to each operational characteristic. To achieve good operational performance RTs had to meet at least four of the following five criteria: storage temperature: (1) ambient temperature, (0) need for refrigeration; number of reagents required to perform the assay: (1) only one reagent, (0) need for more than one reagent; total number of steps to perform the test: (1) four steps or less, (0) more than four steps; technical skill needed by the operator: (1) laboratory experience not required, (0) need for laboratory experience; and total test execution time: (1) 30 min or less, (0) more than 30 min.11

Ethical considerations

All plasma bag identifications were unlinked from the panel members prior to analysis, and no clinical and/or demographic data were obtained from the donors.

Statistical analysis

The sensitivity, specificity and accuracy of the RTs were calculated and expressed with a binomial CI of 95%, using the MedCalc software (V.12.0.3.0, -Ostend/Belgium). Accuracy values ​​were transformed into percentages and interpreted as excellent (90%–100%), good (80%–90%), fair (70%–80%), poor (60%–70%)%) and weak 50%–60%).12

Kappa statistic was used to determine the concordance between the RTs for syphilis and the gold standard, and the values ​​determined by Altman13 were used as reference. Poor concordance was given for Kappa values ​​of between 0.00 and 0.20; fair (0.21 to 0.40); moderate (0.41–0.60); good (0.61–0.80) and very good (0.81–1.00).13 The Statistical Package for Social Sciences (SPSS software, V.17.0) was used to determine the Kappa values.

Results

Table 1 shows that all the tests admit plasma, serum or whole blood samples as a matrix. All the tests have the same storage temperature range (2°C–30°C), with the exception of the Rapid Check Syphilis test that must be stored at 5°C–30°C.

One of the first steps of this evaluation was to establish a panel with samples duly classified in terms of their reactivity to the syphilis tests. This meant that it was subsequently possible to evaluate the RTs available in the Brazilian market. The initial strategy was to use bags of plasma obtained from blood donors as a sampling source. The size of each bag ensured that there was sufficient plasma to generate aliquots which, when correctly stored, can enable several evaluations to be carried out over the years.

After characterising the total of 493 samples, the first panel consisted of 127 (25.8%) treponemal antibody-positive and 366 (74.2%) antibody-negative specimens for syphilis. There was complete concordance between the results obtained with the three reference tests (FTA-ABS, chemiluminescence immunoassay and TPHA).

Among the evaluated tests, none of the seven RTs required samples to be inactivated. The sensitivity ​​of the RTs evaluated varied from 94.5% to 100%, while their specificity ranged from 91.5% to  100% (table 2). It is important to note that the RT DPP Syphilis test correctly classified all the panel samples. With the exception of Determine Syphilis, which had a specificity of 91.5% (95% CI 88.2 to 94.2) and was therefore below the minimum 93% acceptable to the MoH,7 8 all the other tests had specificity above this value, ranging from 94% to 100%. The accuracy of the tests ranged from 95.8% to 98.5%.

Table 2

Sensitivity, specificity, accuracy and Kappa coefficient of the seven evaluated rapid tests

The Kappa statistic was used to estimate the concordance level between the seven RTs evaluated and the results of the reference samples. Five (5/7) of the RTs presented an excellent Kappa index (0.92–1) and two others presented good concordance.

As for operational performance, all the RTs evaluated obtained a maximum score (five points) with the exception of the RT DPP Syphilis assay, which obtained 4 points due to the number of reagents used (two buffers) (table 3).

Table 3

Evaluation of the operational performance of the test based on five characteristics of the RT

Discussion

Syphilis diagnosis remains a challenge because the tests results need to be associated with the clinical history to be correctly interpreted. Countries with low prevalence of syphilis and well-distributed laboratory coverage do not recognise in the RTs a role to contribute for the improvement of syphilis diagnosis and have not found reasons to replace the regular syphilis laboratory tests.14 15 In addition, Food and Drug Administration has cleared only one syphilis RT in the USA.14 However, low-income countries or those that present specific characteristics have other challenges to face. For example, Brazil’s huge continent-sized land area, with many remote and difficult-to-access places and often restricted internet access, often make it difficult to obtain the results of laboratory tests. As an integral part of the strategies for improving access to diagnosis and therefore care, since 2005, on-site rapid testing has enabled same-day test results and treatment as indicated.7 8 The centralised bulk procurement of RTs has produced substantial economies of scale. In 2014, 2015 and 2017, a total of 3 960 000, 9 298 980 and 12 037 550 tests, respectively, were purchased in this way. In parallel to the introduction of RTs, Brazil implemented strategies to evaluate the quality of the tests as well as an external quality control assessment programme to cover the whole country and control the quality of the services delivered.

Obtaining reactive samples for syphilis is a fairly difficult task, especially in countries where syndromic treatment is widely available in the health services. In these circumstances, blood banks play an important role in providing syphilis plasma samples. From a legal standpoint, plasma bags with reagent results for any serological marker should be discarded as unfit for use in transfusions. They can also, for example, contain anticoagulants that lead to the formation of fibrin clumps that could interfere with certain syphilis test results (eg, VDRL). Furthermore, the plasma freeze–thaw process can facilitate the formation of precipitates. However, Castro et al 6 demonstrated that plasma samples can be converted into serum that can be used for producing quality control panels.6 These authors showed that as a result of the plasma conversion process, there is virtually no loss of sample reactivity, since 85% of the samples retain the titre in the non-treponemal reagin plasma reaction (RPR) test. In the treponemal tests, the reactivity is maintained in 96% in FTA-ABS and in 89% in the TPHA. Since they are free of fibrin, plasma samples converted into serum can be kept frozen for long periods without precipitation and are therefore suitable for the manufacture of reference sera and quality control panels. The worldwide CDC and WHO Joint External Quality Testing Program for Syphilis tests uses plasma samples converted to serum.6 Since the defibrination methodology is very well described, this approach can be followed by other countries.

The panel used in the present study was made with plasma samples converted into serum. We used only true-positive and true-negative samples, since the objective was to verify the performance of the RTs as compared with the conventional tests routinely used in laboratories. The MoH’s diagnosis access policy and the expansion of the diagnostic coverage for syphilis in pregnant women have adopted RTs as a screening test. In this respect, RTs are expected to perform as well as the treponemal tests used in the laboratory. The present study is the first to evaluate RTs for syphilis marketed in Brazil.

The sensitivity of the tests evaluated varied from 94.5% to 100% in relation to the panel and the specificity from 91.5% to 100%. Imuno-Rapido Syphilis, Determine Syphilis, RT DPP Syphilis and RT ABON tests showed 100% sensitivity with substantial variations in specificity values. The RT DPP Syphilis test showed 100% specificity, followed by RT ABON (97.0%) and Imuno-Rapido Syphilis (94.0%). Determine Syphilis (91.5%) failed to reach the minimum values ​​established by the MoH for this parameter (93.0%). Rapid Check Syphilis (sensitivity 99.2% and specificity 99.7%), Syphilis Strip Test Bioeasy (94.5% and 99.5%) and Syphilis Test Biocon (94.5% and 97.8%) showed values ​​of sensitivity and specificity of below 100% but nevertheless within the limits accepted by the MoH. The kappa coefficients were between 0.89 and 1.0, indicating good and excellent concordance between the RTs and the gold standard.

Several studies report the variability of sensitivity values ​​of the Determine Syphilis test. Results consistent with those found in the present study (100.0%) were reported by Lien et al 16 Hernández-Trejo et al 17 and Siedner et al. 18 Other authors reported low sensitivity values when using finger-prick, whole blood samples as 88.0%,18 91.8%,19 85.7%20 and 97.3%.21 A multicentre clinic-based study performed by Mabey et al 22 also found low sensitivity in Tanzania (59.6%), Brazil (88.5%), China (81.9%) and Haiti (72.5%).22

Specificity values for the Determine Syphilis test similar to those found in this study were reported in the study by Brozan et al 20 who found 90.9%. Herring et al 23 in a multicentre study with archived samples, found specificity values ranging from 88.0% to 100.0%.23 Other authors have reported specificity values ​​higher than those found in the present study, varying from 97.7% to 100.0%.17 18 22 24 25 These findings suggest that important differences occur in the results obtained in different studies and raise questions about the variability of the test depending on the sample matrix used in the evaluation, as well as possible methodological modifications in the test, the influence of testing conditions (field or laboratory based) or lack of robustness of the test per se.

In the present study, Syphilis Strip Test Bioeasy showed values ​​of 94.5% sensitivity and 99.5% specificity. The kappa coefficient (0.95) indicates that this test is similar to the other members of this evaluation. Due to the change in the name of the test, the comparison in the literature was carried out with the test previously marketed in Brazil—SD Bioline Syphilis 3.0. It is possible, however, that in addition to the name change there may have been methodological modifications. Several studies have different specificity and sensitivity values ​​with the SD Bioline Syphilis 3.0 test. Sensitivity values ​​very close to those found in the present study were reported by Herring et al 23 who found 95.0%; Benzaken et al 26 90.2%; Dlamini et al 27 90.1%; Causer et al 21 87.8%; Bocoum et al 28 90%; and Mehra et al 29 92.9%. As for specificity values, the same authors found 94.9%, 99.4%, 98.0%, 98.5%, 98.0% and 98.3%, respectively. With the exception of the 94.9% obtained by Herring et al 23 the specificity values ​​obtained by the other authors are in line with the 99.5% found in the present study.23

The RT DPP Syphilis test presented values ​​of 100.0% sensitivity, specificity and kappa coefficient of 1, presenting excellent performance in all the evaluated parameters. However, this was the only test that did not show maximum operational performance (table 3). The high performance of the test cannot be compared with other references since there are no studies on RT DPP Syphilis for a single marker on database platforms.

The tests Imuno-Rápido Syphilis (Wama Diagnostica), Rapid Check Syphilis (NDI-UFES), Syphilis Test Biocon (Biocon Diagnostics) and RT ABON (Abon, Hangzhou) have no validation studies available in database platforms, which rules out discussion here of their results, although they presented, respectively, sensitivity values ​​(100.0%, 99.2%, 94.5% and 100.0%), specificity (94.0%, 99.7%–97.8% and 97.0%) and kappa coefficient (0.89, 0.99, 0.92 and 0.94) above the minimum values ​​established by the Brazilian MoH.

The present study has the following limitations: (1) no clinical data were obtained, and therefore it was not possible to correlate the laboratory results with the clinical presentation, (2) the use of defibrinated plasma as a primary sample for the performance of reference tests and RTs, although it is not the matrix used in the public health services, is appropriate for the making up quality control panels; and (3) although the study was conducted in a laboratory environment with technical staff trained and experienced in performing RTs, it is expected that the results obtained in the health services are comparable with those obtained in the present evaluation.6 Despite these limitations, the results from this work can be extrapolated to the countries that use the same syphilis RTs.

Cost-effectiveness studies of different diagnostic strategies for syphilis indicate that an RT is cost-effective if the sensitivity is higher than 88%. Failing that, the use of RPR is recommended.30

Conclusion

The RT DPP Syphilis, Imuno-Rápido Syphilis, Rapid-Check Syphilis, Syphilis Strip Test Bioeasy, RT ABON and Syphilis Test Biocon tests showed operational performance and sensitivity and specificity values as established in the Ministerial Directive for the diagnosis of syphilis in Brazil.

RTs for syphilis are an excellent alternative for use in the public health sphere because they enable diagnoses to reach populations that were previously excluded from access. Notwithstanding the benefits of RTs in the implementation of public health policies, the tests need to be carefully quality controlled, ideally, on a batch-to-batch basis.

Key messages

  • The seven rapid tests evaluated in the study presented good operational performance.

  • The professionals who performed the rapid tests considered them easy to use and interpret.

  • Most tests reached sensitivity and specificity values as established in the Ministerial Directive for the diagnosis of syphilis in Brazil.

  • Rapid tests for syphilis are an excellent alternative because they enable diagnosis to reach populations that were previously excluded from access.

Acknowledgments

Our thanks to the Department of Surveillance, Prevention and Control of Sexually Transmitted Infections, HIV/AIDS and Viral Hepatitis team, to the Laboratory of Molecular Biology, Microbiology and Serology (LBMMS) of the Federal University of Santa Catarina staff and to John Penney and Leticia Muraro Wildner for the manuscript review.

References

Footnotes

  • Contributors Conception and design of the work: MLB and MF. Laboratory work and data analysis: RCMRdO, LG, AB, PCG and MLB. Drafting the work: MLB, LRdM, OdCFJ, AFNPCP and MF; critically revised the paper: OdCFJ, FM, MF and ASB. Responding to the editorial committee and the referees: MLB, RCMRdO, AB, PCG and AFNPCP. Final approval: ASB.

  • Funding This study was funded by the Fundo Nacional da Saúde (National Health Fund/Brazil).

  • Competing interests None declared.

  • Patient consent Detail has been removed from this case description/these case descriptions to ensure anonymity. The editors and reviewers have seen the detailed information available and are satisfied that the information backs up the case the authors are making.

  • Provenance and peer review Commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.