Materials and methods

A total of 210 women aged 18–46 years was recruited from a family planning clinic in Johannesburg, South Africa, during the period from August to November 2003. Serum samples collected from consenting women of unknown HSV-2 serostatus were tested for HSV-2 using the HerpeSelect ELISA (Focus Technologies) and the Kalon HSV-2 gG2 ELISA (Kalon Biologicals). Optical density (OD) readings for Kalon and the normalised OD readings for HerpeSelect were recorded. Samples with normalised OD readings of less than 0.9 were recorded as negative, those with values greater than 1.1 were recorded as positive, and those with intermediate values (0.9–1.1) were recorded as equivocal as per manufacturer's instructions.

Using a predetermined sampling strategy, a random selection of specimens with concordant results and all discordant results (with serum remaining) were shipped to the University of Washington, Seattle, USA, for evaluation using a gold standard Western blot assay, which has been described previously.11 Samples with remaining serum were tested for HIV-1 using Abbott AxSYM HIV 1/2 gO (Abbott Diagnostics, Wiesbaden-Delkenheim, Germany) in South Africa only when it became apparent that the HIV-1 serostatus might influence HSV-2 ELISA results.4 Indeterminate results were resolved using BioRad Genetic Systems rLAV HIV-1 ELISA (BioRad Laboratories, Redmond, USA).

The sensitivity and specificity of the different tests were calculated, taking into account the sampling strategy, according to methods described by Hawkins et al.12 Only samples with (normalised) OD readings greater than 1.1 were considered positive. Additional analyses were performed to investigate whether the sensitivity and/or specificity of the tests could be improved by changing the cut-off values for positive specimens using receiver operator characteristic (ROC) curves and likelihood ratios. We specifically investigated the sensitivity and specificity of a higher cut-off value for HerpeSelect of 3.5 or greater as has been suggested by other authors.5 13 The effect of age and HIV-1 serostatus on sensitivity and specificity were also explored.

All participants were volunteers who gave written informed consent to participate before any study-related procedures. This study was approved by the Human Research Ethics Committee of the University of the Witwatersrand and the Research Ethics Committee of the London School of Hygiene and Tropical Medicine, and was conducted in accordance with good clinical and laboratory practice guidelines.

Results

Population characteristics

Participants had a mean age of 25.6 years (range 18–46). The overall HSV-2 seroprevalence for this population varied by as much as 14%, depending on the test used. Of the 210 specimens tested, 168 (80%) of women were HSV-2 seropositive, 40 (19%) were negative and two results (1%) were equivocal using HerpeSelect. With the Kalon assay, 138 (66%) of the samples were HSV-2 seropositive, 58 (28%) were seronegative and 14 (7%) results were equivocal. HIV-1 results were available for 145 (69%) participants. The overall HIV-1 prevalence was 52%.

The results of testing using HerpeSelect and Kalon were compared. Of the 210 samples tested, 178 (85%) had concordant results for both tests: 138 (66%) were concordant positive, 42 (20%) were concordant negative and none were equivocal on both tests. Thirty-two specimens (15%) had discordant results. In the samples with discordant results, the majority (n=30) were positive on HerpeSelect but either negative (n=16) or equivocal (n=14) on Kalon. In two cases, samples were equivocal on HerpeSelect and negative with Kalon. Overall, HerpeSelect appeared to detect positive specimens more frequently than Kalon (see table 1).

Sensitivity and specificity

A subset of 19 samples with discordant ELISA results, 44 samples with concordant positive ELISA results and 35 samples with concordant negative ELISA results were compared with Western blot (see table 1). Using the data in this table, sensitivity and specificity were calculated using a method that accounts for this sampling strategy (see table 2).12 According to the manufacturer's instructions, the sensitivity of HerpeSelect was 98% (95% CI 95 to 100) and the specificity was 61% (95% CI 48 to 74). The sensitivity of the Kalon assay was 89% (95% CI 83 to 94) and its specificity was 85% (95% CI 61 to 100).

Because both HerpeSelect and Kalon yield continuous results based on OD readings, it was possible to explore the sensitivity and specificity of the test depending on the cut-off value chosen to define a positive test. Initially, we examined the higher cut-off value for HerpeSelect of 3.5 or greater, which has been proposed by others.13 Whereas this resulted in a decreased sensitivity (94%), the specificity was substantially improved (87%). Further exploration using ROC curves showed that 3.3 was the cut-off value for optimal sensitivity (96%) and specificity (87%), correctly classifying 86% of samples. For Kalon, further interpretation of the ROC curve suggested that there was nothing to be gained in terms of sensitivity by changing the cut-off above or below the recommended index of 1.1. We subsequently analysed whether using two ELISA in combination improved sensitivity and specificity when compared with Western blot. The best combination was obtained when using HerpeSelect at increased cut-off (≥3.5) followed by testing of ‘low positive’ and equivocal samples with Kalon, yielding a sensitivity of 92% (95% CI 82 to 98) and specificity of 91% (95% CI 79 to 98%). Using this approach 22 (10%) of the original samples tested by HerpeSelect would have required retesting with Kalon.

Effect of age and HIV-1 serostatus on assay performance

In an exploratory analysis, we investigated the effect of age and HIV-1 serostatus on assay performance (see table 3). The sensitivity of both tests was lower in the age group under 25 years compared with those in the age group 25 years and older. Conversely, specificity was higher for both tests in the under 25 years age group, compared with the older age group. The sensitivity of HerpeSelect in HIV-1-seropositive specimens was high (100%). By contrast, its specificity was substantially lower in specimens of participants co-infected with HIV-1 (30%), but was improved by raising the cut-off to 3.5 or greater (80%). The performance of Kalon was broadly comparable (sensitivity 91%, specificity 72%).

Table 4 summarises the distribution of index values for samples that gave false-positive results by either test when compared with Western blot. When comparing these values by age group, more false-positive samples had index values in the low range (1.1–2.0) in the younger age group compared with the older age group, when tested by HerpeSelect (4 vs 2). This was not true for Kalon. In the false-positive samples for which we had HIV-1-positive results, all four false-positive samples (three HerpeSelect, one Kalon) were HIV-1 positive.

Discussion

The sensitivity of HerpeSelect and Kalon observed in this study is high and is similar to previous observations from other African settings in which both HSV and HIV-1 prevalence are high,4 13 14 and compares favourably with the results from industrialised countries.15

We found a wide variation in specificity between the two tests, with HerpeSelect demonstrating a high rate of false-positive results, using the cut-off value recommended by the manufacturer. This resulted in an overestimation of seroprevalence in this population by as much as 14%. This is in contrast to observations by Hogrefe et al,5 who found a specificity of 100% in sera from South Africa and Zimbabwe, although this was similar to observations from other studies in Uganda, Kenya, Zambia, Benin and Nigeria, where specificity was as low as 40–70%.4 6 7 13

There are several possible explanations for the higher sensitivity but lower specificity of HerpeSelect compared with Kalon. One explanation is that HerpeSelect is more sensitive than Kalon, and even Western blot, in detecting early seroconversion. A study comparing the median time to seroconversion of the three assays found that this was significantly longer for Kalon (120 days) and Western blot (87 days, p=0.004) than for HerpeSelect (21 days, p<0.001).16 A recent study among African patients with genital ulcer disease also found that rates of HSV-2 seroconversion in cases of documented first episodes of genital HSV-2 were significantly higher by HerpeSelect compared with Kalon (77% vs 23% at day 14).17 The high HSV-2 prevalence in this population suggests that seroconversion is not a rare event. In addition, 50% of the HerpeSelect false-positive tests had readings in the low positive range, which may be suggestive of early infection.13 However, for this to be true, we would have expected to observe higher false-positive rates in the younger age group compared with the older age group. This was not the case in our study. In fact, we observed a lower sensitivity (of both assays) in the younger population compared with the older population.

An alternative explanation for the differences in specificity could be cross-reactivity with other infections, including HSV-1 or HIV-1. Although the glycoprotein-G2 tests are generally quite specific for HSV-2, one study found that, in patients with cultured documented recurrent genital HSV-1 infection, the specificity of Kalon was 100%, whereas the specificity of HerpeSelect was slightly lower (93%).16 Golden et al18 showed the impact of HSV-1 on lowering the specificity of HerpeSelect in male sexually transmitted disease patients. We were unable to test whether there was cross-reactivity with HSV-1 because of the high prevalence of HSV-1 (98% of samples tested by Western blot) in this population. Perhaps a more plausible explanation relates to the presence of circulating non-specific antibodies, which could either be the result of hyperglobulinaemia secondary to immune activation caused by HIV-1, or could even be the antibodies to HIV-1 themselves, which might cross-react with the G2-specific portion of the test.19 20 Specificity was also shown to be lower in HIV-1-seropositive samples in the analysis of samples from the Four African City Study,4 for both tests, but substantially lower for HerpeSelect. This is in contrast to Laeyendecker et al,14 who did not observe any effect of HIV-1 serostatus on test performance, when comparing median index values for HerpeSelect in HIV-1-infected and uninfected individuals. A further study by the same group among Ugandan subjects did not reveal differences in assay performance by HIV-1 serostatus.7 However, HIV-1 prevalence was lower in this Ugandan population (33%) than in our South African population (52%). Higher rates of co-infection with HIV-1 in the older age groups may also explain the higher specificity observed in the younger age group in our study. Although not conclusive, we also noted in our study that all HSV-2 false-positive tests with available HIV-1 results were indeed HIV-1 positive.

A third possible explanation is geographical variation in HSV-2 strains. Although data from Europe suggest that the gG2 epitope is fairly well conserved,21 strains from African populations have not been sequenced. These strains may be more diverse and have different affinities for both the ELISA assays, as well as the Western blot. Certainly, atypical Western blot profiles were observed in this study (data not shown), and have been reported by other investigators.5 6 13

Raising the cut-off value for defining positive results for HerpeSelect appeared to improve the specificity without compromising sensitivity too much and compares well with the Kalon assay. This approach was eventually used as the strategy for identifying participants with HSV-2 for inclusion in two large multicentre HSV-2 suppressive treatment trials.22 23 We showed that the same approach did not yield similar improvements in performance for the Kalon assay. This may be because this test is already fairly specific for HSV-2, and further improvements in specificity result in losses in sensitivity.

Finally, we showed that using a combination of two tests resulted in high levels of sensitivity and specificity being obtained when compared with Western blot. Using HerpeSelect with a higher cut-off and testing all equivocal results with Kalon as the resolver test resulted in a testing algorithm that was suitably sensitive and specific, and only required re-testing of 10% of the original sample. Economic and operational research will be warranted to determine the role of these strategies in other settings. The demand for improved HSV-2 testing strategies is likely to grow with increasing awareness of the high prevalence of HSV-2 in the developing world and its association with HIV-1 transmission.

In conclusion, high rates of false positivity continue to challenge the performance of the HerpeSelect assay in African sera. In particular, the poor specificity of the test in HIV-1-seropositive populations warrants its cautionary use and a larger scale investigation. However, adjusting the cut-off and/or using a two-test testing algorithm resulted in significant improvements when compared with using either test alone. The feasibility and cost-benefit of such approaches should be evaluated further.

Short summary

An evaluation of two commercial HSV-2 ELISA (HerpeSelect and Kalon), compared with HSV-2 Western blot, found that co-infection with HIV-1 markedly reduced the specificity of the commercial assays.