Article Text

PDF

Alarmingly poor performance in Chlamydia trachomatis point-of-care testing
  1. Laura van Dommelen1,
  2. Frank H van Tiel1,
  3. Sander Ouburg1,2,
  4. Elfi E H G Brouwers3,
  5. Peter H W Terporten1,
  6. Paul H M Savelkoul4,
  7. Servaas A Morré1,2,
  8. Cathrien A Bruggeman1,
  9. Christian J P A Hoebe1,3
  1. 1Department of Medical Microbiology, Maastricht Infection Centre, Maastricht University Medical Centre, School for Public Health and Primary Care (CAPHRI), Maastricht, The Netherlands
  2. 2Department of Pathology, Laboratory of Immunogenetics, Section Immunogenetics of Infectious Diseases, VU University Medical Center, De Boelelaan, Amsterdam, The Netherlands
  3. 3Department of Infectious Diseases, South Limburg Public Health Service, Geleen, The Netherlands
  4. 4Department of Medical Microbiology and Infection Control, VU University Medical Center, De Boelelaan, Amsterdam, The Netherlands
  1. Correspondence to Laura van Dommelen, Department of Medical Microbiology, Maastricht Infection Centre, Maastricht University Medical Centre, School for Public Health and Primary Care (CAPHRI), PO Box 5800, 6202 AZ Maastricht, The Netherlands; lauravandommelen{at}yahoo.com

Abstract

Background Infection by Chlamydia trachomatis (CT) is the most prevalent sexually transmitted infection (STI) world wide. The most frequently used diagnostic test for CT is a nucleic acid amplification test (NAAT), which is highly sensitive and specific. To further shorten time delay until diagnosis has been made, in order to prevent CT spread, the use of point-of-care (POC) tests may be the way forward.

Objectives The diagnostic performance of three POC tests, Handilab-C, Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia test, was evaluated and compared with NAAT.

Methods All women, above the age of 16 years, attending for a consultation at an STI clinic between September 2007 and April 2008, were asked to participate. Women were asked to complete a short questionnaire and to collect six self-taken vaginal swabs (SVS). SVS 2 was used for NAAT and SVS 3 to 5 were randomised for the different POC tests. SVS 1 and 6 were used for determining quantitative CT load to validate the use of successive SVS. All POC tests were performed without knowledge of NAAT results. NAAT was used as the ‘gold standard’.

Results 772 women were included. CT prevalence was 11% in our population. Sensitivities of the Biorapid CHLAMYDIA Ag test, QuickVue Chlamydia and Handilab-C test were 17%, 27% and 12%, respectively.

Conclusions The evaluated POC tests, owing to their very low sensitivities, are not ready for widespread use. These results underline the need for good-quality assurance of POC tests, especially in view of the increased availability of these tests on the internet.

  • Point of care test
  • Chlamydia trachomatis
  • NAAT
  • sensitivity and specificity
  • PCR
  • same day testing
  • screening
  • std control

Statistics from Altmetric.com

Introduction

World wide, Chlamydia trachomatis (CT) remains the most prevalent bacterial sexually transmitted infection (STI), with increased incidence in Europe over the past decade.1 CT infection is, a major cause of reproductive morbidity,2 3 bacterial conjunctivitis in neonates,4 and may facilitate HIV transmission.5 The use of nucleic acid amplification tests (NAAT) with self-taken vaginal swabs (SVS) or urine have made CT testing more sensitive, specific and acceptable.6 Nevertheless, case finding and case recognition is hampered first, by the limited willingness of patients at risk to undergo STI testing because of fear of pelvic examination and stigmatisation, and second owing to the frequently asymptomatic nature of these infections.7 Moreover, with the use of NAAT, there is still a time delay between first consultation and treatment, usually around 1–2 weeks.8 Although some infections may resolve during this period, secondary transmission can take place and infection can progress. Therefore, a point-of-care (POC) test with proven diagnostic accuracy may well help limit the spread of and morbidity associated with CT.

Over the past few years, an increase in the availability of POC tests in drug stores and on the internet has been noticeable. In general, there appears to be a trend towards producing diagnostics, which are faster and easier to use. WHO has formulated criteria for a POC test which is adequate9: a new STI diagnostic test should be affordable by those at risk, sensitive (sensitivity between 43% and 65%), specific (specificity of 98%), user-friendly, rapid and robust, equipment-free and deliverable to those in need (ASSURED criteria; http://www.who.int/std_diagnostics, accessed 14 June 2010). We have selected three widely available POC CT diagnostic tests, which might meet these criteria but which have not yet been evaluated thoroughly. We assessed laboratory performance and the potential acceptability, when used in optimal conditions compared with NAAT, to maximise POC test results before evaluation in non-laboratory and/or less developed settings. Moreover, the use of successive SVS was validated using a quantitative CT NAAT.

Methods

Study setting, specimen collection and population

Women above the age of 16 years applying for STI consultation between September 2007 and April 2008 were included in the study. The medical ethics committee of Maastricht University Medical Centre approved this study (MEC LLL06srs) and all participants signed a written consent form. At the STI clinic, each patient was asked to take six number-marked SVS in the order of number (SVS 1 to 6). Patients were shown how to insert the vaginal swab by approximately 4–5 cm and with 10 s vaginal rotation and rubbing time and how to position the swab into each capped tube. During the consultation, demographic and behavioural data were collected and, if indicated, samples were collected for other STI diagnostics. All data and SVS were anonymised and transported to the hospital while refrigerated. Patients who tested positive for CT were treated with a single dose of 1000 mg azithromycin. CT prevalence was expected to be 11% in this population with no loss to follow-up.6

Point-of-care tests

SVS 3 to 5 were used for the POC tests. The three POC tests that were validated were the Handilab-C (Zonda, Dallas, USA), Biorapid CHLAMYDIA Ag test (Biokit, S.A., Barcelona, Spain) and QuickVue Chlamydia test (Quidel Corporation, San Diego, USA). All POC tests had a CE mark and were commercially available. In order to control for possible differences in CT load in successively taken SVS, the POC tests were randomised before distribution, into SVS groups (named A, B and C) with Handilab-C, Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia tests being performed on SVS 3-4-5 in group A, SVS 4-5-3 in group B and SVS 5-3-4 in group C, respectively. The Handilab-C is an enzymatic test with a detection limit of 16 inclusion bodies/test (package insert). The Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia test are antigen tests; the detection limit of the Biorapid CHLAMYDIA Ag test is 57–570 elementary bodies/test and the QuickVue Chlamydia should have a sensitivity of 81% when <100 inclusion forming units (IFU)/ml are present (package inserts).

All POC tests were stored and performed under optimal conditions in the medical microbiology laboratory, after training provided by the suppliers, and according to the manufacturers' instructions. One exception was the use of an SVS instead of an endocervical specimen with the Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia test. The POC tests were performed in the medical microbiology laboratory, but the Handilab-C test was started at the STI clinic: ‘fluid A’ was allowed to mix with the specimen and left standing for 10 min. After transportation to the laboratory, the swab was pushed through the foil in order to make a short contact with ‘fluid B’. This procedure was discussed and supported by the manufacturer. The Handilab-C cannot be used during menstruation and the second step of the test performance must be completed within 24 h (definition of an ‘on time’ result). Both the Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia test had to be performed within 72 h after collecting the SVS (definition of an ‘on time’ result). POC tests were performed and read by LvD and three fully qualified microbiological technicians. NAAT results and clinical data were linked to the POC test results no sooner than at the end of the study. Stratification by menstruation and time to test performance was therefore done retrospectively.

NAAT tests

The COBAS Amplicor CT/NG (Roche Diagnostics Systems, Basel, Switzerland) on SVS 2 was used as ‘gold standard’ for determining CT presence. Although the COBAS Amplicor CT/NG is not licensed for SVS, previous studies have demonstrated no significant difference in performance between the use of SVS and that of endocervical swabs.10 11 SVS 2 was placed in 1 ml lysis buffer and after rotation for 10 s the swab was squeezed by pressing against the plastic tube and then removed. Next, 1 ml diluent was added, mixed, centrifuged and 50 μl of the supernatant was added to 50 μl PCR Mix. The sample was processed further according to the standard operating procedure for CT PCR. A result of more than 10 000 DNA copies was considered positive. All low positive samples between 2000 and 9999 copies of CT DNA were retested to confirm the presence of CT. Samples with repeatedly borderline (n=1) or inhibited (n=8) NAAT results were excluded from analysis.

For quantitative CT load determination, a real-time PCR (TaqMan assay) targeting the cryptic plasmid of CT (sensitivity of 0.01 IFU as compared with 1 IFU for the COBAS Amplicor and able to detect the recently reported Swedish variant of CT) or the human HLA was developed with Primer Express v2.0 (Applied Biosystems, Foster City, California, USA), described previously by Catsburg et al.12 Real-time PCR reactions were performed in a volume of 30 μl PCR volume, consisting of TaqMan Mastermix (Applied Biosystems), 300 nM of each primer, 150 nM of each probe and 5 μl prepared sample. Amplification and detection was performed with an ABI Prism 7000 sequence detection system (Applied Biosystems) using standard PCR conditions of the manufacturer, with 45 cycles. By using a chlamydial and a human target, the average chlamydial/human cell load ratio, and IFU/swab were calculated. All samples were spiked with an optimal amount of internal control to validate the sample preparation as well as the RT-PCR procedure.

Statistical analysis

Sensitivity, specificity, negative (NPV) and positive (PPV) predictive values of the different POC tests compared with ‘gold standard’ PCR were calculated. Categorical variables were analysed with the Pearson χ2 test for independence and with Fisher's exact test where appropriate. Binary logistic regression was used to determine the influence of different variables (including randomisation) on the outcome of NAAT and POC tests. A p value <0.1 was used for selecting variables and a p value <0.05 was used to determine significant adjusted OR. Quantitative CT results were compared using the t test for paired samples. A p value <0.05 was considered statistically significant. Analyses were performed with the SPSS package version 14.0.

Role of POC test providers

None of the POC test providers had any role in the study design, collection or interpretation of the data or writing of the manuscript.

Results

Population and questionnaire

Between September 2007 and April 2008, 772 women were included with a median age of 23 years (range 16–64). Over 95% of all clients filled in the questionnaire. The median age of first sexual contact was 16 years (range 6–36). The median lifetime number of sexual partners was nine (range 1– >99) and almost half of these contacts were considered as unsafe sexual contact. During the past 6 months, the median number of newly acquired sexual partners was three (mean 4; range 0–>99). Only two out of 772 women were co-infected with Neisseria gonorrhoeae. No cases of syphilis or HIV were detected. In the month before visiting the outpatient clinic, 13% (99/772) of the clients had used antibiotics, five of whom were CT positive with NAAT. The CT-positive clients could not recall which antibiotic they had used.

POC tests compared with NAAT

C trachomatis testing by COBAS Amplicor resulted in a CT prevalence of 11% in our population (84/772 clients). Sensitivities, specificities, NPV and PPV of the different POC tests compared with NAAT are presented in table 1. Results are presented according to time between collecting the SVS and performance of the POC test and subdivided for women with self-reported symptoms. Owing to logistical limitations, 49% of the Handilab-C results were performed in time. On time Handilab-C results are depicted for non-menstruating clients, since this test is not validated in the case of menstruation. Sensitivities of the Biorapid CHLAMYDIA Ag test, QuickVue Chlamydia test and Handilab-C were 17%, 27% and 12%, respectively. The failure rate (meaning an invalid or missing test result) of 5% when including all Handilab-C results is mainly caused by presence of blood on the SVS, which hinders interpretation of the test result; self-reported menstruation was the probable cause of 85% (23/27) of the bloody samples. If all POC tests were included, sensitivity only decreased significantly in the QuickVue Chlamydia test. Binary logistic regression was performed using all POC test results, taking into account factors that might influence diagnostic test results13 14 (details on the binary logistic regression are available in the supplementary online table). This assessment suggested no relevant influences.

Table 1

Performance of the different point-of-care tests

Quantitative CT NAAT results

Quantitative CT NAAT (qNAAT) was used on 70/84 positive CT samples. The qNAAT was inhibited in six paired samples and in a single SVS 6; all other samples tested CT positive. Almost 30% of the bacterial loads were identical between the first and sixth swab taken. Higher bacterial loads were seen in SVS 1 (mean: 445678 IFU/swab, median: 19410 IFU/swab: this is excluding extreme values with the Grubb test for outlier detection15) compared with SVS6 (mean: 29963 IFU/swab, median: 12180 IFU/swab: excluding extreme values). The CT load was <100 (but >20 CFU/ml) in one paired sample and in three single SVS 1 and two single SVS 6. Statistical analysis demonstrated no significant difference in POC test performance in relation to CT load for the different tests (data not shown). On average 14.6×106 HLA targets per swab were seen in SVS 1 (median: 5.0×106 HLA targets/swab), compared with an average of 706.7×106 HLA targets/swab in SVS 6 (median: 167.9×106 HLA targets/swab). The Grubb test was used to detect and remove outliers. The average bacterial load per cell was higher in SVS 1 than in SVS 6, probably owing to mucus removal by the immediately preceding five SVS.

Discussion

The development and marketing of POC tests for CT has taken place in response to the demand for more rapid diagnosis, with the obvious goal of earlier treatment and prevention of secondary cases. In this study, three POC tests were evaluated under optimal laboratory conditions, in a population with a high CT prevalence (11%). Overall, our data show that all POC tests perform alarmingly poorly.

A few limitations in our study should be mentioned. First, choosing patients solely from a western laboratory setting, limits direct translation of our results to other settings. However, the poor performance of POC tests in our setting is unlikely to improve under conditions with lower resources. Second, reproducibility of POC tests could not be assessed, since each swab could only be used for one POC test. Third, the COBAS Amplicor does not detect the Swedish variant of CT (swCT or new variant nvCT) and POC test results might therefore be worse since some CT-positive samples might have been missed. The swCT, however, has been detected in The Netherlands in only one case so far and directly linked to a swCT-positive Swedish women (Morré SA, personal correspondence).16 17 Finally, for the Biorapid CHLAMYDIA Ag test and QuickVue Chlamydia test a SVS was used instead of an endocervical swab as stated in the package insert. As we have shown, the CT loads in the SVS were almost all above the detection limit of the different POC tests and statistical analysis demonstrated no significant influence of CT load on test performance. Moreover, the bacterial loads found in our study using SVS, are comparable to results found for endocervical swabs in a previous study.18

The strengths of our study are the large study population, the comparison of three POC tests in one and the same study, the experiments performed to control for CT load differences in successively taken SVS and, finally, the use of the ASSURED criteria as a reference enabling objective reviewing of results.

In our experience, all POC tests were easy to perform with respect to laboratory handling, but the Handilab-C was difficult to interpret, even after multiple tests had been carried out. In a previously published evaluation, a small-scale Norwegian study19 has already raised questions about the performance of the Handilab-C. In this study, 50% of all participating women, who were asked to perform the test themselves, were not certain how to interpret their Handilab-C result. Sixteen out of 157 participating women were CT positive with NAAT (used as ‘gold standard’). The Handilab-C result was interpreted as positive by only four, and as uncertain by nine clients, which resulted in sensitivity between 25% and 57%. Michel et al recently evaluated the Handilab-C in a group of 231 women (38/231 CT NAAT positive), again demonstrating a low sensitivity, and discussed this in view of the value of a CE mark.20

The QuickVue Chlamydia has been evaluated twice thus far. In a 1997 publication, the QuickVue Chlamydia was evaluated in a population of 724 women divided into two high-risk and one low-risk population.21 Sensitivity and specificity were on average 90.1% and 99.5%, respectively, in the high-risk populations (n=366, CT prevalence 14.1%). Performance of the QuickVue Chlamydia in this study was compared with culture. Samples with false-positive QuickVue Chlamydia results, however, were retested with NAAT and added to the true positive results if found positive with NAAT. In contrast, culture-negative samples, with a negative QuickVue Chlamydia result, were not retested with NAAT despite a sensitivity of culture of only 65%.22 Therefore, false-negative QuickVue Chlamydia test results would not have been detected, and performance of the QuickVue Chlamydia in this study has been overestimated. In 2002, a second evaluation was published comparing QuickVue Chlamydia with NAAT in two groups of 100 women.23 In the high-risk population, sensitivity and specificity were 65% and 100%, respectively, with 16 women being positive with NAAT. In the low-risk population however, the sensitivity was only 25% (1/4). If both groups in the study by Rani et al are taken into account, the CT prevalence in their study is 10% (20/200), which is comparable to the CT prevalence of 11% in our population. Recalculating sensitivity and specificity when using both populations of Rani et al, rendered a sensitivity of 55.0% (95% CI 33.3% to 76.8%) and a specificity of 100%, which is not significantly different from our results.

As can be extrapolated from our results, a POC test with excellent performance may make a difference; assuming a primary CT transmission of 65% (without further transmission) when having sexual contact,24 a treatment delay of 2 weeks8 25 and a POC test sensitivity of 100%, eight additional new CT cases would have been avoided compared with NAAT. In contrast, when applying the same calculation to our data, the result is negative compared with NAAT and owing to false-positive results, participants would have been treated unnecessary, especially in case of the Biorapid CHLAMYDIA Ag and Handilab-C test. In a recent evaluation, the Chlamydia Rapid Test showed promising results26; this POC test primarily would have detected fewer CT cases than NAAT, but owing to instant treatment prevent more CT cases, resulting in equal outcome in our model. A sensitivity of 83.5% is not sufficient to replace NAAT in a setting with minimal loss to follow-up; cost–benefit analysis therefore may determine if combining NAAT and a POC test is beneficial to avert additional CT cases.

In summary, results of this study, performed in a large population, show poorer laboratory performance of the different POC tests than has previously been described. The ASSURED criteria for POC testing including a sensitivity 43–65% and a specificity 98%,9 are not met by any of the POC tests. The poor performance of all POC tests evaluated in our study has implications for public health, since the Handilab-C test remained commercially available via the internet (€29.95) during the entire inclusion period. The distributor has claimed a reliability of 98.15% (not further specified) on his website while, for instance, sensitivity in our study population was only 12%. Our results underline the need for good quality assurance of POC tests, especially in view of their availability on the internet.27 Although excellent guidelines on CT POC test evaluation exist,28 these guidelines are regularly ignored, and thus tighter regulations are urgently needed to prevent unrestrained marketing.9 In our opinion, the CT POC tests we have evaluated, are not ready for widespread use.

Key messages

  • Point-of-care tests (POC) could potentially diagnose Chlamydia trachomatis genital infections rapidly enough to enable the immediate start of treatment.

  • The POC tests evaluated in this study are not ready for widespread use, owing to lack of sensitivity.

  • Our results underline the need for good quality assurance of POC tests, especially in view of internet-based trading possibilities.

Acknowledgments

We thank the staff of the STI outpatient clinic of the South Limburg Public Health Service, The Netherlands for including study participants; Jolein Pleijster (VU University Medical Center, Amsterdam, The Netherlands) for excellent technical assistance while performing the bacterial load assays; and Petra Wolffs (Maastricht University Medical Centre, Maastricht the Netherlands) for her comments on molecular assays.

References

View Abstract

Footnotes

  • Linked articles: 044719.

  • Independence of researchers All authors were independent, and worked independently from the distributors of the tests.

  • Access to data All authors had full access to all of the data in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

  • Data sharing technical appendix, statistical code, and dataset available from the corresponding author at: f.van.tiel{at}mumc.nl

  • Funding This study was not funded. All rapid tests were kindly and unrestrictedly provided by the distributors.

  • Competing interests None.

  • Ethics approval This study was conducted with the approval of the Maastricht University Medical Centre, Maastricht, The Netherlands (MEC-06-4-004).

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles