Introduction Sero-epidemiological studies of herpes simplex virus (HSV) type 2 infection in Africa remain difficult to interpret as a result of the high rate of false-positive results observed when using the new recombinant gG2 HSV-2 ELISA tests. The performance of two widely used gG2 ELISA was compared to derive an appropriate testing algorithm for use in South Africa.
Methods Sera from 210 women attending family planning clinics in Johannesburg were tested using HerpeSelect and Kalon HSV-2 gG2 assays. Sera from 20 discordant pairs, 44 concordant positive and 33 concordant negative samples were further tested by HSV Western blot. The sensitivity and specificity of each test and of combination algorithms compared with Western blot were calculated.
Results HerpeSelect had a sensitivity of 98% (95% CI 95 to 100) and specificity of 61% (95% CI 48 to 74). Kalon was less sensitive (89%, 95% CI 83 to 94) but more specific (85%, 95% CI 61 to 100). Seroprevalence may have been overestimated by as much as 14% by HerpeSelect. Specificity was improved by raising the cut-off index for the determination of a positive result for HerpeSelect (to ≥3.5), but not for Kalon. HIV-1 infection reduced the specificity of HerpeSelect to 30%. Improved sensitivity and specificity were obtained by a two-test algorithm using HerpeSelect (≥3.5) as the first test and Kalon to resolve equivocal results (sensitivity 92%, 95% CI 82 to 98; specificity 91%, 95% CI 79 to 98).
Conclusion Newer HSV-2 serological tests have low specificity in this South African population with a high HIV-1 prevalence. Two-step testing strategies could provide rational testing alternatives to Western blot.
- Herpes simplex virus type 2
- HSV serology
- South Africa
Statistics from Altmetric.com
Herpes simplex virus (HSV) type 2 is a primary cause of genital ulcers and is one of the most prevalent sexually transmitted infections worldwide.1 Recent serological studies conducted among populations with no specific high-risk sexual behaviour characteristics in sub-Saharan Africa have shown prevalence rates that exceed those of similar populations in the USA and Europe.2 Up to 70% of high-risk HIV-1-seronegative and up to 85% of HIV-1-seropositive individuals are seropositive for HSV-2 in sub-Saharan Africa.2 3 However, sero-epidemiological studies of HSV-2 in Africa have been hampered by concerns that some of the newer HSV-2 ELISA are associated with high rates of false-positive reactions in African sera. In an evaluation study of 13 HSV-2 type-specific assays, the specificity ranged from 47% to 99%.4 In this evaluation, the HerpeSelect (Focus Technologies Inc, Cypress Hill, California, USA) was shown to have a high sensitivity (100%) but a low specificity (71%), whereas the Kalon HSV-2 gG2 ELISA (Kalon Biologicals Ltd, Aldershot, UK) was one of the best performing tests (sensitivity 93% specificity 98%). Specificity was shown to be lower in HIV-1-seropositive individuals. In another study of sera from populations in South Africa, Zimbabwe, Kenya and Uganda using the HerpeSelect,5 100% concordance with Western blot was observed in sera from Zimbabwe and South Africa, but was lower for samples from Kenya (96%) and Uganda (88%). More recently, a study comparing HerpeSelect and Kalon with Western blot in 120 HIV-1-seronegative men aged 18–24 years in Kenya showed a lower specificity for HerpeSelect (40%) compared with Kalon (79%).6 Another more recent study using 538 Ugandan samples tested with Western blot, two ELISA assays and a rapid test (Biokit) confirmed the lower specificity of HerpeSelect (51%), which was improved by raising the cut-off value for positive results to 3.2. In the same study, the specificity of the Kalon assay was found to be superior to HerpeSelect; this was enhanced further by raising the cut-off for positive results to 1.5, which increased specificity from 88% to 92%.7 This study did not find any significant difference in assay performance by HIV-1 serostatus.
While sensitive tests are more useful for diagnosis, higher levels of specificity are required in epidemiological studies in which associations with other infections such as HIV-1 are explored. Highly specific testing strategies are required to identify individuals who might benefit from HSV treatment interventions currently being evaluated in trials. Large-scale Western blot testing is costly and is not feasible in many settings in Africa. For these reasons, a comparative evaluation of the sensitivity and specificity of two HSV-2 specific ELISA-based serological assays was undertaken in a South African population in which HIV-1 and HSV-2 prevalence are both high.8–10
Materials and methods
A total of 210 women aged 18–46 years was recruited from a family planning clinic in Johannesburg, South Africa, during the period from August to November 2003. Serum samples collected from consenting women of unknown HSV-2 serostatus were tested for HSV-2 using the HerpeSelect ELISA (Focus Technologies) and the Kalon HSV-2 gG2 ELISA (Kalon Biologicals). Optical density (OD) readings for Kalon and the normalised OD readings for HerpeSelect were recorded. Samples with normalised OD readings of less than 0.9 were recorded as negative, those with values greater than 1.1 were recorded as positive, and those with intermediate values (0.9–1.1) were recorded as equivocal as per manufacturer's instructions.
Using a predetermined sampling strategy, a random selection of specimens with concordant results and all discordant results (with serum remaining) were shipped to the University of Washington, Seattle, USA, for evaluation using a gold standard Western blot assay, which has been described previously.11 Samples with remaining serum were tested for HIV-1 using Abbott AxSYM HIV 1/2 gO (Abbott Diagnostics, Wiesbaden-Delkenheim, Germany) in South Africa only when it became apparent that the HIV-1 serostatus might influence HSV-2 ELISA results.4 Indeterminate results were resolved using BioRad Genetic Systems rLAV HIV-1 ELISA (BioRad Laboratories, Redmond, USA).
The sensitivity and specificity of the different tests were calculated, taking into account the sampling strategy, according to methods described by Hawkins et al.12 Only samples with (normalised) OD readings greater than 1.1 were considered positive. Additional analyses were performed to investigate whether the sensitivity and/or specificity of the tests could be improved by changing the cut-off values for positive specimens using receiver operator characteristic (ROC) curves and likelihood ratios. We specifically investigated the sensitivity and specificity of a higher cut-off value for HerpeSelect of 3.5 or greater as has been suggested by other authors.5 13 The effect of age and HIV-1 serostatus on sensitivity and specificity were also explored.
All participants were volunteers who gave written informed consent to participate before any study-related procedures. This study was approved by the Human Research Ethics Committee of the University of the Witwatersrand and the Research Ethics Committee of the London School of Hygiene and Tropical Medicine, and was conducted in accordance with good clinical and laboratory practice guidelines.
Participants had a mean age of 25.6 years (range 18–46). The overall HSV-2 seroprevalence for this population varied by as much as 14%, depending on the test used. Of the 210 specimens tested, 168 (80%) of women were HSV-2 seropositive, 40 (19%) were negative and two results (1%) were equivocal using HerpeSelect. With the Kalon assay, 138 (66%) of the samples were HSV-2 seropositive, 58 (28%) were seronegative and 14 (7%) results were equivocal. HIV-1 results were available for 145 (69%) participants. The overall HIV-1 prevalence was 52%.
The results of testing using HerpeSelect and Kalon were compared. Of the 210 samples tested, 178 (85%) had concordant results for both tests: 138 (66%) were concordant positive, 42 (20%) were concordant negative and none were equivocal on both tests. Thirty-two specimens (15%) had discordant results. In the samples with discordant results, the majority (n=30) were positive on HerpeSelect but either negative (n=16) or equivocal (n=14) on Kalon. In two cases, samples were equivocal on HerpeSelect and negative with Kalon. Overall, HerpeSelect appeared to detect positive specimens more frequently than Kalon (see table 1).
Sensitivity and specificity
A subset of 19 samples with discordant ELISA results, 44 samples with concordant positive ELISA results and 35 samples with concordant negative ELISA results were compared with Western blot (see table 1). Using the data in this table, sensitivity and specificity were calculated using a method that accounts for this sampling strategy (see table 2).12 According to the manufacturer's instructions, the sensitivity of HerpeSelect was 98% (95% CI 95 to 100) and the specificity was 61% (95% CI 48 to 74). The sensitivity of the Kalon assay was 89% (95% CI 83 to 94) and its specificity was 85% (95% CI 61 to 100).
Because both HerpeSelect and Kalon yield continuous results based on OD readings, it was possible to explore the sensitivity and specificity of the test depending on the cut-off value chosen to define a positive test. Initially, we examined the higher cut-off value for HerpeSelect of 3.5 or greater, which has been proposed by others.13 Whereas this resulted in a decreased sensitivity (94%), the specificity was substantially improved (87%). Further exploration using ROC curves showed that 3.3 was the cut-off value for optimal sensitivity (96%) and specificity (87%), correctly classifying 86% of samples. For Kalon, further interpretation of the ROC curve suggested that there was nothing to be gained in terms of sensitivity by changing the cut-off above or below the recommended index of 1.1. We subsequently analysed whether using two ELISA in combination improved sensitivity and specificity when compared with Western blot. The best combination was obtained when using HerpeSelect at increased cut-off (≥3.5) followed by testing of ‘low positive’ and equivocal samples with Kalon, yielding a sensitivity of 92% (95% CI 82 to 98) and specificity of 91% (95% CI 79 to 98%). Using this approach 22 (10%) of the original samples tested by HerpeSelect would have required retesting with Kalon.
Effect of age and HIV-1 serostatus on assay performance
In an exploratory analysis, we investigated the effect of age and HIV-1 serostatus on assay performance (see table 3). The sensitivity of both tests was lower in the age group under 25 years compared with those in the age group 25 years and older. Conversely, specificity was higher for both tests in the under 25 years age group, compared with the older age group. The sensitivity of HerpeSelect in HIV-1-seropositive specimens was high (100%). By contrast, its specificity was substantially lower in specimens of participants co-infected with HIV-1 (30%), but was improved by raising the cut-off to 3.5 or greater (80%). The performance of Kalon was broadly comparable (sensitivity 91%, specificity 72%).
Table 4 summarises the distribution of index values for samples that gave false-positive results by either test when compared with Western blot. When comparing these values by age group, more false-positive samples had index values in the low range (1.1–2.0) in the younger age group compared with the older age group, when tested by HerpeSelect (4 vs 2). This was not true for Kalon. In the false-positive samples for which we had HIV-1-positive results, all four false-positive samples (three HerpeSelect, one Kalon) were HIV-1 positive.
The sensitivity of HerpeSelect and Kalon observed in this study is high and is similar to previous observations from other African settings in which both HSV and HIV-1 prevalence are high,4 13 14 and compares favourably with the results from industrialised countries.15
We found a wide variation in specificity between the two tests, with HerpeSelect demonstrating a high rate of false-positive results, using the cut-off value recommended by the manufacturer. This resulted in an overestimation of seroprevalence in this population by as much as 14%. This is in contrast to observations by Hogrefe et al,5 who found a specificity of 100% in sera from South Africa and Zimbabwe, although this was similar to observations from other studies in Uganda, Kenya, Zambia, Benin and Nigeria, where specificity was as low as 40–70%.4 6 7 13
There are several possible explanations for the higher sensitivity but lower specificity of HerpeSelect compared with Kalon. One explanation is that HerpeSelect is more sensitive than Kalon, and even Western blot, in detecting early seroconversion. A study comparing the median time to seroconversion of the three assays found that this was significantly longer for Kalon (120 days) and Western blot (87 days, p=0.004) than for HerpeSelect (21 days, p<0.001).16 A recent study among African patients with genital ulcer disease also found that rates of HSV-2 seroconversion in cases of documented first episodes of genital HSV-2 were significantly higher by HerpeSelect compared with Kalon (77% vs 23% at day 14).17 The high HSV-2 prevalence in this population suggests that seroconversion is not a rare event. In addition, 50% of the HerpeSelect false-positive tests had readings in the low positive range, which may be suggestive of early infection.13 However, for this to be true, we would have expected to observe higher false-positive rates in the younger age group compared with the older age group. This was not the case in our study. In fact, we observed a lower sensitivity (of both assays) in the younger population compared with the older population.
An alternative explanation for the differences in specificity could be cross-reactivity with other infections, including HSV-1 or HIV-1. Although the glycoprotein-G2 tests are generally quite specific for HSV-2, one study found that, in patients with cultured documented recurrent genital HSV-1 infection, the specificity of Kalon was 100%, whereas the specificity of HerpeSelect was slightly lower (93%).16 Golden et al18 showed the impact of HSV-1 on lowering the specificity of HerpeSelect in male sexually transmitted disease patients. We were unable to test whether there was cross-reactivity with HSV-1 because of the high prevalence of HSV-1 (98% of samples tested by Western blot) in this population. Perhaps a more plausible explanation relates to the presence of circulating non-specific antibodies, which could either be the result of hyperglobulinaemia secondary to immune activation caused by HIV-1, or could even be the antibodies to HIV-1 themselves, which might cross-react with the G2-specific portion of the test.19 20 Specificity was also shown to be lower in HIV-1-seropositive samples in the analysis of samples from the Four African City Study,4 for both tests, but substantially lower for HerpeSelect. This is in contrast to Laeyendecker et al,14 who did not observe any effect of HIV-1 serostatus on test performance, when comparing median index values for HerpeSelect in HIV-1-infected and uninfected individuals. A further study by the same group among Ugandan subjects did not reveal differences in assay performance by HIV-1 serostatus.7 However, HIV-1 prevalence was lower in this Ugandan population (33%) than in our South African population (52%). Higher rates of co-infection with HIV-1 in the older age groups may also explain the higher specificity observed in the younger age group in our study. Although not conclusive, we also noted in our study that all HSV-2 false-positive tests with available HIV-1 results were indeed HIV-1 positive.
A third possible explanation is geographical variation in HSV-2 strains. Although data from Europe suggest that the gG2 epitope is fairly well conserved,21 strains from African populations have not been sequenced. These strains may be more diverse and have different affinities for both the ELISA assays, as well as the Western blot. Certainly, atypical Western blot profiles were observed in this study (data not shown), and have been reported by other investigators.5 6 13
Raising the cut-off value for defining positive results for HerpeSelect appeared to improve the specificity without compromising sensitivity too much and compares well with the Kalon assay. This approach was eventually used as the strategy for identifying participants with HSV-2 for inclusion in two large multicentre HSV-2 suppressive treatment trials.22 23 We showed that the same approach did not yield similar improvements in performance for the Kalon assay. This may be because this test is already fairly specific for HSV-2, and further improvements in specificity result in losses in sensitivity.
Finally, we showed that using a combination of two tests resulted in high levels of sensitivity and specificity being obtained when compared with Western blot. Using HerpeSelect with a higher cut-off and testing all equivocal results with Kalon as the resolver test resulted in a testing algorithm that was suitably sensitive and specific, and only required re-testing of 10% of the original sample. Economic and operational research will be warranted to determine the role of these strategies in other settings. The demand for improved HSV-2 testing strategies is likely to grow with increasing awareness of the high prevalence of HSV-2 in the developing world and its association with HIV-1 transmission.
In conclusion, high rates of false positivity continue to challenge the performance of the HerpeSelect assay in African sera. In particular, the poor specificity of the test in HIV-1-seropositive populations warrants its cautionary use and a larger scale investigation. However, adjusting the cut-off and/or using a two-test testing algorithm resulted in significant improvements when compared with using either test alone. The feasibility and cost-benefit of such approaches should be evaluated further.
An evaluation of two commercial HSV-2 ELISA (HerpeSelect and Kalon), compared with HSV-2 Western blot, found that co-infection with HIV-1 markedly reduced the specificity of the commercial assays.
HSV-2 seroprevalence ranged from 66% with the Kalon ELISA to 80% with the HerpeSelect ELISA.
HerpeSelect had a high sensitivity but lower specificity whereas Kalon was less sensitive but more specific.
HIV-1 infection appeared to reduce the specificity of HerpeSelect.
Specificity was improved by raising the cut-off index for the determination of a positive result for HerpeSelect (to ≥3.5), but not for Kalon.
The authors would like to thank Tim Clayton for additional useful discussion and assistance. In addition, they would like to thank the participants, the study staff and the staff at the City of Johannesburg clinic who facilitated this study.
Appendix I Example of calculation of sensitivity and specificity for HerpeSelect based on Hawkins et al.
Partial presentation of this work: 16th Biennial Conference of the International Society for Sexually Transmitted Diseases Research, Amsterdam, 2005.
Funding The study was supported by grants from the UK's Department for International Development (DFID) office in Pretoria, South Africa, and the UK's DFID-funded Knowledge Programme on HIV/AIDS and STI of the London School of Hygiene and Tropical Medicine. Western blot testing was supported by NIH grant AI30731.
Competing interests RAM has received research grants or contracts, honoraria or consulting fees during the last 3 years from Biokit, Bio-Rad Laboratories, Biovail, Focus Diagnostics and GlaxoSmithKline. The remaining authors report no competing interests.
Ethics approval This study was conducted with the approval of the University of the Witwatersrand Human Research Ethics Committee, South Africa and the London School of Hygiene and Tropical Medicine Ethics Committee, UK.
Patient consent Obtained.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.