Article Text
PDF
Abstract
Objectives The availability of point-of-care (POC) tests for infectious diseases has revolutionised the provision of healthcare for remote rural populations without access to laboratories. However, quality assurance for POC tests has been largely overlooked. We have evaluated the use and stability of dry tube specimens (DTS) for External Quality Assurance (EQA) for HIV and syphilis screening in remote indigenous populations in the Amazon region of Brazil.
Methods All healthcare workers (HCWs) participating in the community-screening were trained. We used HIV and syphilis DTS panels developed by the reference laboratory, containing samples with negative and positive results at different antibody concentrations, for both infections. DTS panels were distributed to HCWs in the communities for reconstitution and testing using POC HIV and syphilis tests. The results of testing were sent to the reference laboratory for marking and remedial action taken where necessary.
Results In total 268 HCWs tested 1607 samples for syphilis and 1608 samples for HIV. Results from HCWs showed a concordance rate of 90% for syphilis and 93% for HIV (κ coefficients of 0.74 and 0.78, respectively) with reference laboratories. Most false negatives were in samples of very low antibody concentration. DTS syphilis specimens produced the expected results after storage at 2–8°C or at 18–24°C for up to 3 weeks.
Conclusions The results show that POC tests for syphilis and HIV give valid results in environments where traditional tests do not work, but errors in the interpretation of POC test results were identified by the EQA programme using DTS. EQA using DTS can help to improve the quality of screening programmes using POC tests in remote regions.
- TESTING
- SYPHILIS
- HIV
- SCREENING
Statistics from Altmetric.com
Introduction
The use of rapid testing for HIV and syphilis at the point of care (POC) has recently expanded and successfully allowed the implementation of Same Day Testing and Treatment Strategies.1 Their rapid scale-up is largely attributed to their low cost, flexible transport and storage requirements, and ease of use. The Amazon region is characterised by extreme climate, geographical isolation, fragile physical infrastructure and lack of well-trained laboratory personnel. All of these factors impact the use of laboratory technologies in communities living in difficult to access, remote rural areas, such as the indigenous population of the Brazilian Amazon region. Syphilis and POC tests can help overcome these barriers and bring the laboratory to the patient, and yet have high sensitivity and specificity.2–5
To obtain reliable results and ensure the success of screening programmes involving many healthcare workers (HCWs), it is not sufficient to use high quality reagents and simple methods. It is also necessary to implement an external quality assurance system (EQA) to ensure proper performance in each of the centres where the tests are used and in all the steps involved in the process—from sample collection to reporting of results.6 In Brazil, the quality assurance of laboratory activities is regulated by the National Agency of Sanitary Surveillance (Agência Nacional de Vigilância Sanitária)7 but quality assurance for POC tests has not been available.
In order to increase access and demonstrate the feasibility of a programme for the diagnosis of syphilis and HIV with POC tests in indigenous populations, a research project was developed in the Amazon region of Brazil along with an EQA system through the use of dried tube specimens (DTS).8
This EQA system allows the systematic and periodic assessment of the proficiency of the HCWs implementing POC testing. The results of the EQA enable supervisors to verify if the test and interpretation of results are being done correctly, and to train the HCW in the event of inconsistencies. This control method allows the correct identification of cases of both diseases and the timely provision of treatment.9
We conducted a feasibility study to determine if DTS offered a reliable form of EQA for HIV and syphilis POC testing in remote regions of the Brazilian Amazon. This paper discusses the benefits of the EQA programme and the final evaluation of its potential success.
Methodology
Programme settings and population
Syphilis and HIV POC testing and the associated EQA programme were implemented in six Special Indigenous Health Districts (Distrito Sanitário Especial Indígena, DSEI) in the Alto Río Negro, Alto Solimoes, Leste Roraima, Manaus, Parintins and Yanomami regions. In Brazil, the National Policy for the Health Care of Indigenous Peoples states that the DSEI are the organisational models designated to provide health services to these populations. The DSEI are designed to address a well-defined geographical area and are dynamic ethnocultural spaces that respect the traditional demographics and social relations among different indigenous populations and regional society.10
Screening among indigenous populations began in February 2009 and continued until June 2011. All consenting sexually active individuals were tested in health facilities or laboratories in their communities. Forty-five thousand two hundred and twenty-six individuals who had not previously been screened were tested, representing 54% of the indigenous population in the study region.1 The EQA programme described in this study occurred from March 2010 to March 2011 in the six DSEI participating in the community screening programme. In July 2011, with the purpose of providing complete and effective recommendations to the National Sexually Transmitted Infection (STI)/AIDS Programme in Brazil, the stability testing of HIV and syphilis DTS was conducted.
Site preparation
All local staff (n=268) participating in the community screening were trained in the implementation of POC tests and EQA with DTS. In each DSEI, the distribution of materials followed standard operating procedures. The POC tests for HIV and syphilis were purchased from Bio-Manguinhos/Fiocruz (TR-Bio-Manguinhos-HIV1/2) and Standard Diagnostics (TR Syphilis 3.0-SD-Bioline), respectively, and transported by ship or aeroplane in coolers (to areas with ambient temperatures above 30°C) or at ambient temperature (to areas with temperatures below 30°C), stored according to manufacturer's instructions (up to 30°C) upon arrival at the DSEI, and subsequently distributed to the base points for screening the population.
Preparation of DTS
DTS specimens and proficiency panels were prepared by Alfredo da Matta Foundation's reference laboratory in Manaus. For each assay, a pool of serum specimens was obtained from STI clinic attenders and were well characterised as positive or negative. Rapid plasma reagin (RPR) positive sera (titre 1:128) were used undiluted or diluted 1:10 and 1:100 in negative sera.
For both tests, DTS were prepared according the method of Parekh et al8 with the following modifications: 50 µL of serum or plasma mixed with 0.1% (v/v) of trypan blue (Vetec, Rio de Janeiro, Brazil) were dispensed into a 1.2 mL tube (TPP-Techno Plastic Products AG, Trasadingen, Switzerland). The tubes were left open in a laminar flow hood overnight to dry. They were then capped and sent to each DSEI where they were stored at 4°C until testing.
Each HIV EQA panel consisted of six tubes: five positives with high (2), low (2) and very low (1) intensities, and one negative. Each syphilis EQA panel consisted of six tubes: high positive (2), positive (1), low positive (1), very low positive (1) and one negative. The panel package also included one vial of PT buffer (phosphate buffered saline (PBS)-Tween, 1.5 mL) a transfer pipette and a simple job aid with instructions.
DTS-based proficiency testing in the field
Dried sera were reconstituted with PBS-Tween 20 onsite as described by Pareck et al.8 PBS-Tween was prepared, filtered through a 0.2 µm filter and aliquoted in 1.5 mL volumes to be used as rehydration buffer (or PT buffer). A day before testing, DTS were rehydrated by adding 7 drops (∼200 µL) PBS-Tween with a disposable transfer pipette. This resulted in 1:10 dilution of the original specimen but was treated as undiluted for the purpose of further testing. Tubes were capped and tapped for a few minutes to allow mixing of the specimens. They were left overnight at room temperature to allow solubilisation of dried serum into the PT buffer. The next day, tubes were tapped again to mix the content and used to perform tests according to manufacturer's instructions.
HCWs were provided with a PT panel package and evaluated six samples for HIV and six samples for syphilis, all previously selected at random. Data on test results along with test kit information and quality control data were collected on a simple worksheet developed for the panels. At the end of the EQA programme, all HCWs answered a simple questionnaire to assess the difficulties faced during the implementation of EQA and rapid testing.
Stability tests for syphilis DTS
Stability studies were performed in the Universidade Federal de Santa Catarina laboratory. Parekh et al had shown that HIV-specific antibodies in the DTS specimen were stable at 4°C −37 for 4 weeks.8 Therefore, we focused stability studies on syphilis DTS from Brazil.
The syphilis stability panels were made using samples that had been well characterised by treponemal (EIA, Wama—Brazil) and non-treponemal tests (RPR, Wama—Brazil). Three positive serum samples with RPR titres of 1:64, 1:128 and 1:256 were used. A plasma bag, non-reactive with treponemal chemiluminescent assay (ABBOTT-USA) and non-treponemal test (VDRL-Laborclin-Brazil), was used as the negative sample and for dilution of reactive samples. The results of stability tests were read by two well trained technicians.
Stability DTS panels were prepared using sera of different RPR titres: P1 (1:64), P2 (1:128) and 1:256). Each stability panel was composed of five tubes: one with negative sample, one with undiluted reactive sample and the other three with the following samples dilutions: 1:2; 1:4 and 1:8. The positive samples were diluted in negative plasma (table 1). These dilutions were chosen to provide samples that produced bands with different intensities in the POC tests: very intense (undiluted), moderately intense (1:2), weak (1:4) and very weak intensity (1:8). The syphilis DTS panels were prepared in numbers sufficient to be kept (without reconstitution) under the following conditions: dry environment for 3 days, 1 week, 2 weeks and 3 weeks at 2–8°C (fridge), ambient temperature (18–24°C), 37°C and 45°C. Samples stored at 37°C and 45°C were tested also in a humid environment. The samples were reconstituted before testing. To evaluate the stability of DTS tubes, one set from each temperature was rehydrated at 3 days, 1 week, 2 weeks and 3 weeks and tested for syphilis. After each incubation, the panels were tested. We considered that the DTS specimen was stable when it gave expected results on syphilis rapid test.
Characteristics of the dried tube specimens panels used in Stability tests
POC tests used for stability testing were SD Bioline Rapid Syphilis test (Standard Diagnostics, Seoul, Korea), Determine Syphilis TP test (Alere, Stockport, UK) and DPP SIF—Bio (Bio Manguinhos/Fiocruz, Rio de Janeiro, Brazil).
Statistical analysis
Data were analysed using STATA V.10.0 (StataCorp, College Station, Texas, USA). Participants were asked to interpret a sample on the basis of the intensity of reactivity as one of strong positive, weak positive, very weak positive and negative, and give their final interpretation of the results as positive or negative. Participant interpretation was compared against the expected result and reported as correct or incorrect. For each test, we calculated concordance rate and the κ coefficient between results obtained in the reference lab and in the DSEI.
The Standards for Reporting of Diagnostic Accuracy guidelines have been completed with respect to the following study (see online supplementary appendix 1).
Results
In total 268 HCWs tested 1607 samples for syphilis and 1608 samples for HIV. The overall prevalence of syphilis was 1.6%, and of HIV 0.1%. Seventy-seven of the testing personnel were from Alto Rio Negro (28.7%), 65 from Alto Solimoes (24.3%), 44 from Leste Roraima (16.4%), 42 from Manaus (15.7%), 7 Parintins (2.6%) and 33 from Yanomami (12.3%). Most of the testing personnel were nursing staff (n=129, 48.1%) and nursing technicians (n=120, 44.8%). Other professionals involved in testing included 2 biologists (0.7%), 1 microscopist (0.4%), 12 pathology technicians (4.5%) and 3 nursing assistants (1.1%).
Results of a questionnaire on DTS-PT and POC testing ease of use
In response to the questionnaire, only 11 of the 268 HCWs (4.1%) reported difficulties in performing POC tests during the implementation phase. The highest percentage of difficulties were reported by four technical nursing staff (33.3%) and one pathology technician (8.3%), although in both cases the number of HCWs in these categories was small.
The highest percentage of HCWs who reported having difficulties in performing the tests (28.6%) were located in Parintins. The percentage of HCWs who reported having difficulties with the reconstitution of the DTS was slightly higher (7.1%)—the largest percentages being among HCWs in Yanomami (15.2%). Nursing personnel reported this particular difficulty most frequently (8%).
Results of the EQA for HIV tests
Each of the 268 HCWs who performed the POC tests received a panel comprising six randomly selected HIV sample tubes (DTS). Among these HCWs, 71 individuals (26.5%) made at least one incorrect interpretation of test result. Twenty of these 71 (28.1%) provided an incorrect reading of test results twice. The highest percentage of HCWs who reported an incorrect diagnosis at least in one tube was in Parintins (42.9%).
A total of 1608 sample tubes were supplied to the 268 HCWs, of which 111 samples (6.9%) were incorrectly interpreted, with the largest number of incorrect interpretations of test results being in Leste Roraima (9.1%).
Table 2 shows the percentage of incorrect readings made per category of the DTS of the HIV panel performed by each HCW at the six DSEI. For HIV, there were four possible categories of response: ‘high positive’, ‘low positive’, ‘very low positive’ and ‘negative’. The overall percentages of incorrect interpretation for each category were respectively 3.2%, 9.4%, 11.9% and 5.2%.
HIV: percentage of incorrect interpretation per category and DSEI
Most inaccurate diagnoses corresponded to the category ‘very low positive’, with the exception of HCWs in Parintins, who correctly diagnosed 100% of tubes. For the ‘low positive’ category, the highest percentage of incorrect diagnoses was in Leste Roraima (15.4%). In regards to the ‘positive’ and ‘negative’ categories, the highest percentage of incorrect diagnoses was in Manaus (5.4%).
The concordance rate observed, when comparing the total number of positive and negative outcomes diagnosed by the reference laboratory and the DSEI laboratories, was 93%, and the κ coefficient was 0.78.
Results of the EQA for syphilis tests
For the syphilis tests, each of the 268 HCWs received an EQA panel comprising six sample tubes (DTS) from the reference laboratory. Among these HCWs, 101 (37.7%) incorrectly interpreted test results in at least one of the samples analysed. Sixty-five of the 101 (24.2%) made a wrong diagnosis once, 25 (9.3%) in two of the samples and 5 (1.9%) in three and four of the samples received. The highest percentage of HCWs who incorrectly interpreted test results in at least one tube was in Manaus (35.7%).
In relation to the total number of tubes, 150 out of the 1608 tubes delivered (9.3%) were interpreted incorrectly. HCWs in Alto Solimoes (11.5%) made the highest percentage of misinterpretation of test results.
Table 3 shows the percentage of incorrect interpretation of test results made in each diagnostic category of the syphilis panel, performed by each HCW at the DSEI. For syphilis, there were five possible categories of response: ‘positive’, ‘high positive’, ‘low positive’, ‘very low positive’ and ‘negative’. The overall percentages of misdiagnosis for each of these categories were respectively 3.7%, 4.2%, 21.4%, 40.0% and 6.8%.
Syphilis: percentage of incorrect interpretation of test results per diagnosis category and DSEI
The highest percentages of incorrect reading of test results were in the ‘very low positive’ and ‘low positive’ categories, and in both cases the incorrect test interpretation was more frequent among HCWs working in Alto Rio Negro (66.7% and 27.9%).
The ‘positive’ category showed low rates of incorrect interpretation across all DSEI, while those for the ‘negative’ category were above the average—particularly in Leste Roraima (10.0%).
The concordance rate observed when comparing the total number of positive (including low and very low positive) and negative outcomes diagnosed by the reference laboratory and the DSEI laboratories was 90.4% and the κ coefficient was 0.74.
Results of the stability tests for syphilis DTS
The Syphilis POC tests were performed with all DTS panels (P1, P2 and P3) which were stored under different conditions. Since the panels were reconstituted only before the tests, an expected rapid test performance was achieved when the pellets dissolved completely during the suspension step. In all panels stored for up to 3 weeks at 2–8°C or at ambient temperature (18–24°C), syphilis POC testing gave the expected results. However, at storage temperatures of 37°C, the P1 and P2 panels showed expected results only when stored for up to 1 week in dry conditions. At this storage temperature, the P3 panel showed expected results only when stored for up to 3 days in dry conditions. All panels stored at 37°C with humidity or at 45°C in dry and humid conditions did not show expected rapid test results because the pellet did not dissolve in the suspension step.
Discussion
POC tests can undoubtedly increase access to healthcare in remote populations. The use of POC tests in these indigenous populations in the Amazon region showed that the prevalence of syphilis and HIV are similar to those found in non-indigenous Brazilian populations. Our findings demonstrate the importance of efforts to enhance and, most importantly, give continuity to programmes for the prevention and control of STIs in indigenous populations in order to reduce the transmission of these diseases and their consequences, especially in women and children.
A study by Shott et al11 affirmed the importance of quality control and quality assurance systems when implementing POC devices, particularly when considering the ‘gold standard’. The implementation of our EQA programme allowed us to identify which HCWs needed to be retrained. In this study the concordance rates for both diseases, observed by the HCWs working in indigenous communities, in relation to the reference laboratory were above 90%, and the κ coefficients were between 0.70 and 0.80. This implies that these POC tests can be performed accurately by nurses and other HCWs in remote tropical settings.
We verified the stability of DTS panels for HIV, and this is similar to data reported by Parekh et al8 (data not shown). However, the syphilis stability tests showed some weaknesses related to the time and temperature of storage. It was found that at 37°C in a humid environment, and at 45°C in a humid and a dry atmosphere, the DTS pellets became gelatinous and could not be dissolved. Another weakness identified relates to test results becoming negative (false-negatives)when samples are very dilute (1:8), which indicates the necessity of making the evaluation panels for syphilis with samples diluted up to 1:4. The data, however, showed that the DTS for syphilis can be used for testing in populations living in remote areas under extreme temperature conditions, but the DTS panels should be transported in boxes that enable thermal control determined in the range of 2–30°C. According to the manufacturer's directions, this is also the recommended temperature range for POC test kits.
There are several potential limitations to our study. The EQA occurred in a specific geographical and cultural area, which might influence the generalisation to other regions. Analysis was limited to quantitative data and the opinions of HCWs on the ease of use of DTS and POC testing would have been better examined by open questions. While it is clear that the use of POC tests can reduce the inequity of healthcare provision for remote communities by providing high quality screening programmes of syphilis and HIV, it is important to realise the need for EQA programmes that validate these results. Such validation can be achieved with the use of DTS, which proved to be easy to use (confirmed by over 90% of HCWs), have a low cost, and show excellent stability without refrigeration or special conditions for transportation.8
Conclusions
The DTS method is appropriate for EQA of screening programmes using POC tests for syphilis and HIV in remote indigenous communities, enabling remedial action to be taken when incorrect results are obtained with POC test kits. The use of POC tests for syphilis and HIV is now recommended in the policies of the Brazilian government that aim to increase access to diagnosis and prevent mother to child transmission of these two infections.12 An EQA programme for POC tests for HIV and syphilis is being implemented nationally in Brazil.
Key messages
-
This is the first quality assurance (QA) programme for syphilis and HIV point-of-care (POC) tests in Brazil.
-
This feasibility study led to the introduction of QA of HIV POC tests in voluntary counselling and testing clinics across the country.
-
This study demonstrates that dried tube specimens (DTS) as a method of QA can be implemented in even the most remote regions and extreme temperatures and humidity.
-
The dried nature of DTS means it is safe to mail to laboratories and testing sites around the country, easing distribution.
Acknowledgments
The authors thank Ron Ballard, Ned Hook, King Holmes, Myron Cohen, Maurine Murtagh, Renee Ridzon, Christine Rousseau, and Cathy Wilfert for expert advice, and Izabela Suder and Beth Downe for administrative and organizational assistance throughout the project.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online appendix
Footnotes
-
Handling editor Jackie A Cassell
-
Contributors Study design and conception: RWP, DM, ASB. Carrying out the field and/or laboratory work: CLN, LG, ASB, ICPP. Statistical analysis and data interpretation/review: MLB, ASB, EG. Drafting and reviewing the manuscript: MLB, ASB, NSB, RWP, DM, KAS.
-
Funding Financial support for this study was provided by the Bill & Melinda Gates Foundation through a grant (#47697) to R. Peeling at the UNICEF/UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases and subsequently at the London School of Hygiene and Tropical Medicine. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
-
Competing interests None.
-
Ethics approval This project was reviewed by Fundação Alfredo da Matta (FUAM), Manaus, Brazil ethical review committee and approved on 18 October 2008.
-
Provenance and peer review Not commissioned; externally peer reviewed.