Objective To evaluate the clinical performance of two chlamydia point-of-care (POC) tests compared with a gold standard nucleic acid amplification testing (NAAT).
Methods Tests evaluated were the Chlamydia Rapid Test (CRT), Diagnostics for the Real World and the ACON Chlamydia Rapid Test Device, ACON Laboratories (ACON). Overall 226 men and 225 women in Port Vila, Vanuatu, participated in this prospective study in 2010. NAAT and POC testing was performed on samples of male urine and female vaginal swabs for 156 men and 223 women (CRT), and 133 men and 75 women (ACON).
Results The sensitivity and specificity of the CRT in men were 41.4% (95% CI 23.5% to 61.1%) and 89.0% (95% CI 82.2% to 93.8%), respectively, and in women 74.2% (95% CI 61.5% to 84.5%) and 95.7% (95% CI 91.3% to 98.2%), respectively; for ACON, they were 43.8% (95% CI 19.8% to 70.1%) and 98.3% (95% CI 93.9% to 99.8%) in men, and in women 66.7% (95% CI 22.3% to 95.7%) and 91.3% (95% CI 82.0% to 96.7%), respectively. Both tests were (absolutely) insensitive at organism loads less than 1000 (log=3) per mL or per swab; the CRT sensitivity was significantly lower at loads less than, compared with those greater than, 100 000 (log=5) per mL or per swab.
Conclusions The performance of both CRT and ACON is well below the levels stated by the manufacturers. The evaluated tests are unlikely to be helpful in clinical settings due to the high proportion of false-negatives that will go untreated and false-positives that will result in overtreatment and potential adverse social consequences.
- Chlamydia Trachomatis
- Chlamydia Infection
- Developing World
- Laboratory Methods
Statistics from Altmetric.com
Chlamydia trachomatis is the commonest notifiable bacterial sexually transmitted infection (STI) in the world, with the WHO estimating 105.7 million cases worldwide in 2008.1 Chlamydial infection is associated with reproductive morbidity, including pelvic inflammatory disease, infertility, chronic pelvic pain, ectopic pregnancy and adverse pregnancy outcomes,2 and is a cofactor in HIV transmission.3 ,4
Port Vila is the capital of the tropical Pacific island nation of Vanuatu, which had a population of 234 000 in 2009.5 Vanuatu has a high prevalence of chlamydia, similar to the region as a whole.6 Overall prevalence of chlamydia in Pacific antenatal populations has been reported at 18% in 6 countries in 20057 and at 19% in 11 countries in 2007and 20086; surveys in Vanuatu have variously found prevalences of 21.5%, 13.2% and 25.1% in antenatal populations in 2000, 2005 and 2008, respectively.8 ,9
Screening is needed to detect chlamydial infections that are often asymptomatic,10 while early diagnosis and treatment is important for patients who present with symptoms. Syndromic management remains the mainstay of diagnosis and treatment in many developing countries. While reasonable sensitivities for male urethral discharge (87–99%) and ulcerative disease (68–98%) have been reported,11 it performs less well for symptoms of vaginal discharge with sensitivities reported from 30 to 80% and specificities from 40 to 80%12; false-positive diagnoses lead to overtreatment and potential adverse social consequences.13
Although DNA-based laboratory testing with high sensitivity and specificity may be available in developing countries, it is often not affordable nor accessible to patients; it requires patients to return to obtain results and treatment, resulting in loss of follow-up and delays.14 Well-performing point-of-care (POC) tests offer a ‘test and treat’ approach in one visit and potentially offer a major advantage over laboratory-based and syndromic management, particularly where the return for treatment is low.10 ,15 ,16
We independently evaluated the diagnostic performance of two chlamydia POC tests, with the manufacturers’ stated sensitivity and specificity for each test above 82% and 96%, respectively. The immunoassay-based tests evaluated were ‘Chlamydia Rapid Test’ (CRT) and the ACON ‘Chlamydia Rapid Test Device’ (ACON), manufactured by Diagnostics for the Real World (DRW) and ACON Laboratories, respectively.
The presented evaluation study was part of a larger intervention study on the utility of POC tests for the management of chlamydia in Port Vila, Vanuatu. The tests for this evaluation study were conducted during the first and second phases of the intervention study.
Independence of the study
The study was independent, with neither manufacturer involved in study design, implementation, analysis or manuscript writing. Test kits were purchased at market price from the manufacturers at their developing country rates.
The KPH Clinic is one of two reproductive health clinics in Port Vila. In total, 4155 clients accessed KPH Clinic services in the calendar year of 2010, 84% being women and 16% men, and 43% were under the age of 25. Most (55%) attended for family planning services, 20% for STI services and 20% for counselling on a variety of reproductive health matters; 44 commercial and opportunistic sex workers attended the clinic.
Ethics approvals for all parts of the study were obtained from the Vanuatu Ministry of Health, Port Vila, Vanuatu and the Ethics Committee of James Cook University, Queensland, Australia.
Recruitment and data collection
Men and women attended the clinic either by self-referral or community outreach. Consecutive patients over the age of 18 were invited to participate in the study between April and November 2010 (the study period) on evaluation days over two phases of testing and treatment. Written informed consent was obtained from literate participants; detailed and standardised information was read to those illiterate. Exclusion criteria were antibiotic treatment within the past month, previous participation and menstruation. Participants were interviewed by a nurse using a fully structured questionnaire. Information was collected on demographic and clinical data.
Training and environmental control
Strict clinical and laboratory handling and decontamination procedures were implemented. Procedures for storage and use of kits between 2 and 30°C, as well as collection and testing procedures, were conducted in accordance with the rapid tests manufacturer's instructions. Nurses were trained in CRT testing prior to the commencement of the study using ‘familiarisation’ panels with samples of varying concentrations of chlamydia antigen, and their competence assessed by successfully analysing samples from similar ‘proficiency’ panels; the panels were supplied by DRW.
Specimen collection and testing
The test evaluations were based on male urine and female vaginal swabs. CRT testing was performed through the first phase of the study evaluating prevalent infection and through the second phase evaluating incident infection. ACON testing commenced towards the end of the first phase and continued through the second phase.
Men collected one or two initial void samples for CRT and/or ACON testing in either the initial 4–5 mL of urine in a ‘First Burst’ device (CRT) or 15–20 mL in a sterile container (ACON) after a non-voiding period of at least 2 h for each sample. The First Burst yields an approximately sixfold higher organism load per mL of urine than a routine cup-collected specimen.17 Where two samples were collected for CRT and ACON, collection order was alternated. A 0.5–1 mL aliquot of urine was taken for nucleic acid amplification testing (NAAT) prior to the centrifugation of aliquots of urine with distilled water for CRT and ACON testing as per the manufacturer's instructions to obtain a centrifuge-generated urine pellet.
All women were instructed and encouraged to self-collect vaginal swabs; in some instances, a vaginal swab was taken by a nurse. Swabs from the rapid test kits were used for female rapid testing, and flocked swabs used for NAAT testing. Women were advised to insert the swab 4–5 cm into the vagina and rotate it 10 times, and immediately place the swabs within the provided containers. Either two swabs (CRT+NAAT) or three swabs (CRT, ACON and NAAT) were taken; the collection order for the swabs taken was (1) CRT, (2) ACON and (3) NAAT (ACON manufacturer's instructions are for a cervical—not vaginal—swab).
Rapid testing commenced within the hour of sample collection. Swabs and centrifuge-generated urine pellets were extracted after adding the supplied reagents and using fixed volume disposable plastic pipettes; the resultant solution was then added to the test cartridges in accordance with the manufacturers’ package instructions.
Reading of results
Rapid tests were interpreted qualitatively as per CRT and ACON manufacturers’ instructions. An additional reading was done by comparing the visual signal of both tests with a signal scale (0.5–5, in increments of 0.5) on a card provided by DRW. Where there was uncertainty in the strength of the visual signal, a second nurse was consulted and consensus reached. Tests were considered invalid if a control line was not visible.
Urine and swab samples were frozen on the same day and sent to the Royal Women's Hospital, Melbourne, for testing. Upon arrival, nucleic acid was extracted from the 200 µL aliquot of urine or swab rotated in phosphate buffered saline on the automated MagNA Pure LC (Roche Applied Science, Mannheim, Germany) using the MagNA Pure DNA I Isolation Kit (Roche Applied Science) according to the manufacturer's protocol with the DNA eluted in 100 µL of Elution Buffer (Roche Applied Science). PCR was carried out with 25 µL of extracted DNA on the COBAS TaqMan 48 PCR assay (Roche Diagnostic, Branchburg, New Jersey, USA) according to the manufacturer's instructions.
All chlamydia-positive samples were tested for chlamydial load by a quantitative PCR (qPCR) system targeting the omp1 gene using published methodology.18 The chlamydial load in each sample was estimated by comparing the crossing threshold of each sample to the crossing threshold of a standard curve constructed by amplifying different known copy numbers of the omp1 gene.
All statistical analyses were conducted using IBM SPSS Statistics for Windows, V.20. Age of participants was summarised using median values and IQR. Categorical data were summarised using absolute and relative frequencies. Comparisons between variables were conducted by χ2 and Wilcoxon bivariate statistical tests as appropriate; the α level was set at 0.05 for all tests. The prevalence of chlamydia, and the sensitivities and specificities for each test, is presented together with exact binomial 95% CIs.
Altogether 379 unique individuals (156 men and 223 women) had valid CRT + NAAT testing, and 207 unique individuals (132 men and 75 women) had ACON+NAAT testing; there were 14 people who had missing rapid or NAAT test results. The median age was 25 years. See table 1 for further details and for patient-reported symptoms.
Prevalence of chlamydia and rapid test performance
There were no invalid rapid tests.
The prevalence by NAAT for men and women tested by CRT was 18.6% and 27.8%, respectively, estimating prevalent infections in the population tested during the first phase of the intervention study; for ACON, it was 12.1% and 8.0%, respectively, estimating incident and untreated chlamydial infections occurring during the second phase.
Comparing the test quality criteria between genders, the CRT male sensitivity of 41.4% is significantly less than the CRT female sensitivity of 74.2%. All other listed gender differences are not significant at an α level of 0.05. Of those truly positive by NAAT, 72.4% and 33.3% of men tested by CRT and ACON, respectively, had self-reported symptoms; similarly 66.7% and 40.0% of NAAT-positive women tested by CRT and ACON, respectively, were symptomatic. See table 2 for details.
The median organism loads (table 3) differed significantly between females and males, with loads in females higher than in males by a (linear) factor of 14 in the CRT population and of 72 in the ACON population (the CRT ‘First Burst’ device concentrates the load by approximately sixfold).
POC sensitivity and organism load
Both tests were absolutely insensitive at organism loads below 1000 (log=3) per mL or swab. Sensitivities of the CRT male and female tests at loads less than 100 000 (log=5) per mL or per swab were 23.8% (95% CI 8.22% to 47.2%) and 58.8% (95% CI 40.7% to 75.4%); at loads greater than 100 000 (log=5), they were significantly different (p<0.05) at 87.5% (95% CI 47.4% to 99.7%) and 92.9% (95% CI 76.5% to 99.1%), respectively.
Figure 1 displays a scatter plot of the relationship between CRT visual signals and organism loads determined by the NAAT test.
By reclassifying the weak positive (0.5) POC visual signals as negative, the CRT specificities increased in men and women to 97.6% (95% CI 93.3% to 99.5%) and 98.8% (95% CI 95.6% to 99.9%), respectively, with a reduction in sensitivity to 27.6% (95% CI 12.7% to 47.2%) and 64.5% (95% CI 51.3% to 76.3%), respectively; similarly in ACON the specificities in men and women were 100% (95% CI 96.9% to 100%) and 100% (95% CI 94.8% to 100%), respectively, and the sensitivities were 31.3% (95% CI 11.0% to 58.7%) and 50.0% (95% CI 11.8% to 88.2%), respectively.
The sensitivity of the male CRT test of 41.4% (95% CI 23.5% to 61.1%) was substantially and statistically significantly lower than the 82.6% (95% CI 75.4% to 89.7%) stated in the manufacturer's package insert. The sensitivity of the female CRT test of 74.2% (95% CI 61.5% to 84.5%) was lower than the manufacturer’s stated 83.5% (95% CI 76.5% to 90.5%) for women, but not significantly so. CRT detected fewer infections in men (41.4% (95% CI 23.5% to 61.1%)) than would have been determined by self-reported symptoms alone (72.4% (95% CI 52.7% to 87.27%)), and similar numbers of infections in women (74.2% (95% CI 61.5% to 84.5%) and 66.7% (95% CI 54.6% to 79.1%)), respectively.
We found a distinct relationship between organism load and CRT performance, which probably resulted in the poorer performance in the male CRT (men having a significantly lower load than women).
Four publications on CRT performance authored between 2007 and 2009 reported CRT sensitivities for male ‘First Burst’ urine between 82% and 82.6%,17 ,19 and for CRT female vaginal swabs between 71% and 86.8%;20 ,21 a systematic review of these studies reported pooled estimates of CRT sensitivity and specificity for male first-void urine specimens to be 77% and 99%, and for female vaginal swabs to be 80% and 99%.22
Our findings, however, are more in keeping with a later, and the only other independent, study of the CRT used for urogenital testing; in 2012, Van der Helm reported a sensitivity of 41.2% in female nurse-collected vaginal swabs, with a higher test sensitivity found with higher organism loads.23
On further examination, our results may be consistent with some findings in the four earlier publications. Wisniewski et al,17 in an evaluation of the male CRT, noted that sensitivity improved from 47% to 82% due to a sixfold increase in organism load obtained from First Burst-collected urine compared with cup-collected urine; they noted that the sensitivity of CRT correlated directly to the chlamydial load.
Mahilum-Tapay et al,21 in a performance evaluation of the female CRT, performed quantitative testing for 33/56 PCR-positive swabs and noted that the rapid test visual signal correlated with organism load. It is also apparent from the graph provided that 30 of these 33 samples had organism loads greater than 100 000 (log=5) plasmids/swab and the median load was greater than 1 000 000 (log=6) plasmids/swab; our female CRT population had a median load of 83 400 (log=4.92) copies/swab. Similarly, Nadala et al19 performed quantitative testing on 80/90 urine samples collected from men by First Burst. It is apparent from the graph provided that the median organism load was greater than 100 000 (log 5) plasmids/mL; our median organism load for First Burst-collected urine was 6010 (log=3.78) copies/mL.
Performance characteristics for NAAT may vary depending on the organism load of patients in the study, the reference standard used and the assay under evaluation. While some differences in the organism loads between the previous evaluation studies and our study may be explained by the different assays used, the data also suggest that the loads found in our population may be lower than those found in those studies. Other authors have found evidence that loads can vary between prevalent and incident infections,24 and between community and clinic populations.25
Furthermore, Michel et al found in 201126 that the sensitivity of a similar DRW-manufactured chlamydia lipopolysaccharide POC test for ocular trachoma was 30.4%, substantially less than the 83.6% published earlier by Michel et al27; the difference was explained by a lower organism load. In an independent study, also published in 2011, Harding-Esch et al28 found that the sensitivities of this test ranged between 33.3% and 67.9%.
The found ACON sensitivity for the male urine of 43.8% (95% CI 19.8% to 70.1%) compared poorly to the manufacturer’s stated 90.9%. The ACON female vaginal swabs sensitivity of 66.7% is not directly comparable to the manufacturer's stated 88.5% for cervical swabs due to a potential twofold to threefold reduction in organism load in vaginal compared with cervical samples.29 The number of truly positive ACON participants with symptoms was too small to make inferences about the sensitivity of symptoms.
Specificities for CRT in men and women were 89.0% and 95.7%, respectively, which are less than the manufacturer's stated 98.5% and 98.9%, but not significantly different; similarly, ACON specificities were 98.3% and 91.3%, less than the manufacturer’s stated 99% and 96.7%, respectively, but not significantly different.
It is possible that temperature affects POC specificity; the independent study of the ocular POC test manufactured by DRW found specificity to decrease markedly with temperatures above 31.4°C and with lower relative humidity.28 While our maximum storage and laboratory temperatures did not exceed the recommended 30°C, they were likely to be higher than those found in the temperate zones where the tests were mostly evaluated.
It is also possible that weak positive visual signals (0.5) were over-read by our staff. By classifying all such weak signals as negative, specificity increased to greater than 97.6% for all POC tests; however, there is a consequent fall in sensitivity, ranging from 27.6% to 64.5% for all tests.
Potential for bias
Our sample was self-selected through attendance at a clinic and through invitation via community outreach, with a potential for selection bias. Consecutive patients were invited to participate, reducing the potential for this type of selection bias; however, it cannot be seen how even a strong selection bias could influence the quality criteria of the tests unless via organism load as discussed previously.
There were a number of potential causes for information bias; our POC tests were read by a single nurse, without a blind reading from a second nurse, which could lead to a classification bias for the POC tests. We believe this would be unlikely to have affected the sensitivity and specificity readings for the tests as all nurses proved competence in interpreting the tests and a second opinion was sought where there was doubt. Any error in under-reading a weak positive signal as negative would tend to increase specificity at the expense of sensitivity; thus, the observed low sensitivities are rather underestimations if this bias did occur. We have also commented on the possibility of over-reading of the weak signals above.
For the male ACON tests, the voiding interval was two or more hours rather than the first void of the morning as per the manufacturer's instruction, potentially slightly reducing organism load; however, this timing is unlikely to make a significant difference to the organism load30 and hence sensitivity.
The collection order for female swabs (CRT, followed by ACON, then NAAT) could be cause for further information bias due to potentially decreasing loads with each swab collected, hence overestimating the sensitivity of the POC tests at given organism loads.
Utility of tests
Both tests had a high number of false-positive results, which is of particular concern when combined with modest sensitivity. In our CRT female population with a prevalence of 27.7%, the positive predictive value (PPV) was 86.9%; in a population with a prevalence of 10%, the PPV would be just 65.6%, which would result in one in every three women receiving a false-positive diagnosis.
The assessed sensitivity and specificity of CRT and ACON are well below the levels stated by the manufacturers. The evaluated tests are unlikely to be helpful in a clinical setting due to the high proportions of false-negatives that will go untreated, as well as false-positives that will result in overtreatment and potential adverse social consequences.
The Chlamydia Rapid Test (CRT) and ACON tests have performance characteristics well below those stated by the manufacturers.
Both tests are insensitive at low organism loads, and the CRT sensitivity is related to the organism load.
Better performing point-of-care tests are required for clinical use.
The authors would like to express their gratitude to the funders the Secretariat of the Pacific Community (Suva, Fiji), the Fonds Pacifique (Ministry of Foreign Affairs, French Republic) and HealthLink (Auckland, New Zealand); to the Vanuatu Ministry of Health; to the nurses and staff of Wan Smolbag and KPH Clinic, Vanuatu; to DRW for providing the training and proficiency panels free of charge as well as technical advice about the CRT; and to Dr Rosanna Peeling, London School of Hygiene and Tropical Medicine for reviewing the manuscript.
Handling editor Jackie A Cassell
Contributors All authors were involved in the design of the study and in the revision and final approval of the manuscript to be published. DSH, SGB, SB and LT were involved with conducting the study. MB-S conducted the statistical analysis. DSH, MB-S, SGB, SB, SNT and RM collaborated in writing the manuscript. DSH was the principal investigator.
Funding This work was supported by a grant from the Secretariat of the Pacific Community, Suva, Fiji (Grant No: 1RVUKPH01), and by funding from Fonds Pacifique, Ministry of Foreign Affairs, French Republic and HealthLink, Auckland, New Zealand.
Competing interests None.
Ethics approval (1) James Cook University; (2) Vanuatu Ministry of Health.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Further information can be requested from the corresponding author.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.