External Quality Assurance Program Oversight Laboratory Research Articles

Sources of variability in Luminex bead-based cytokine assays: Evidence from twelve years of multi-site proficiency testing

Bead array assays, such as those sold by Luminex, BD Biosciences, Sartorius, Abcam and other companies, are a well-established platform for multiplexed quantification of cytokines and other biomarkers in both clinical and discovery research environments. In 2011, the National Institute of Allergy and Infectious Diseases (NIAID)-funded External Quality Assurance Program Oversight Laboratory (EQAPOL) established a proficiency assessment program to monitor participating laboratories performing multiplex cytokine measurements using Luminex bead array technology. During every assessment cycle, each site was sent an assay kit, a protocol, and blinded samples of human sera spiked with recombinant cytokines. Site results were then evaluated for performance relative to peer laboratories. After over a decade of biannual assessments, the cumulative dataset contained over 15,500 bead array observations collected at more than forty laboratories in twelve countries. These data were evaluated alongside post-assessment survey results to empirically test factors that may contribute to variability and accuracy in Luminex bead-based cytokine assays. Bead material, individual technical ability, analyte, analyte concentration, and assay kit vendor were identified as significant contributors to assay performance. In contrast, the bead reader instrument model and the use of automated plate washers were found not to contribute to variability or accuracy, and sample results were found to be highly-consistent between assay kit-manufacturing lots and over time. In addition to these statistical analyses, subjective evaluations identified technical ability, instrument failure, protocol adherence, and data transcription errors as the most common causes of poor performance in the proficiency program. The findings from the EQAPOL multiplex program were then used to develop recommended best practices for bead array monitoring of human cytokines. These included collecting samples to assay as a single batch, centralizing analysis, participating in a quality assurance program, and testing samples using paramagnetic-bead kits from a single manufacturer using a standardized protocol.

Development of an international external quality assurance program for HIV-1 incidence using the Limiting Antigen Avidity assay.

Laboratory assays for identifying recent HIV-1 infections are widely used for estimating incidence in cross-sectional population-level surveys in global HIV-1surveillance. Adequate assay and laboratory performance are required to ensure accurate incidence estimates. The NIAID-supported External Quality Assurance Program Oversight Laboratory (EQAPOL) established a proficiency testing program for the most widely-used incidence assay, the HIV-1 Limiting Antigen Avidity EIA (LAg), with US Centers for Disease Control and Prevention (CDC)-approved kits manufactured by Sedia Biosciences Corporation and Maxim Biomedical. The objective of this program is to monitor the performance of participating laboratories. Four rounds of blinded external proficiency (EP) panels were distributed to up to twenty testing sites (7 North American, 5 African, 4 Asian, 2 South American and 2 European). These panels consisted of ten plasma samples: three blinded well-characterized HIV-1-seropositive samples that were included as replicates and an HIV-negative control. The seropositive samples spanned the dynamic range of the assay and are categorized as either recent or long-term infection. Participating sites performed the assay according to manufacturers’ instructions and completed an online survey to gather information on kit manufacturer, lot of kit used, laboratory procedures and the experience of technicians. On average, fifteen sites participated in each round of testing, with an average of four sites testing with only the Maxim assay, seven testing with only the Sedia assay and five sites utilizing both assays. Overall, the Sedia and Maxim assays yielded similar infection status categorization across the laboratories; however, for most of the nine HIV+ samples tested, there were significant differences in the optical density readouts, ODn (N = 8) and OD (N = 7), between LAg kit manufacturers (p < 0.05 based on mixed effects models. The EQAPOL LAg program is important for monitoring laboratory performance as well as detecting variations between manufacturers of HIV-1incidence assays.

A Robust PCR Protocol for HIV Drug Resistance Testing on Low-Level Viremia Samples.

The prevalence of drug resistance (DR) mutations in people with HIV-1 infection, particularly those with low-level viremia (LLV), supports the need to improve the sensitivity of amplification methods for HIV DR genotyping in order to optimize antiretroviral regimen and facilitate HIV-1 DR surveillance and relevant research. Here we report on a fully validated PCR-based protocol that achieves consistent amplification of the protease (PR) and reverse transcriptase (RT) regions of HIV-1 pol gene across many HIV-1 subtypes from LLV plasma samples. HIV-spiked plasma samples from the External Quality Assurance Program Oversight Laboratory (EQAPOL), covering various HIV-1 subtypes, as well as clinical specimens were used to optimize and validate the protocol. Our results demonstrate that this protocol has a broad HIV-1 subtype coverage and viral load span with high sensitivity and reproducibility. Moreover, the protocol is robust even when plasma sample volumes are limited, the HIV viral load is unknown, and/or the HIV subtype is undetermined. Thus, the protocol is applicable for the initial amplification of the HIV-1 PR and RT genes required for subsequent genotypic DR assays.

Genetic Characterization of a Panel of Diverse HIV-1 Isolates at Seven International Sites.

HIV-1 subtypes and drug resistance are routinely tested by many international surveillance groups. However, results from different sites often vary. A systematic comparison of results from multiple sites is needed to determine whether a standardized protocol is required for consistent and accurate data analysis. A panel of well-characterized HIV-1 isolates (N = 50) from the External Quality Assurance Program Oversight Laboratory (EQAPOL) was assembled for evaluation at seven international sites. This virus panel included seven subtypes, six circulating recombinant forms (CRFs), nine unique recombinant forms (URFs) and three group O viruses. Seven viruses contained 10 major drug resistance mutations (DRMs). HIV-1 isolates were prepared at a concentration of 107 copies/ml and compiled into blinded panels. Subtypes and DRMs were determined with partial or full pol gene sequences by conventional Sanger sequencing and/or Next Generation Sequencing (NGS). Subtype and DRM results were reported and decoded for comparison with full-length genome sequences generated by EQAPOL. The partial pol gene was amplified by RT-PCR and sequenced for 89.4%-100% of group M viruses at six sites. Subtyping results of majority of the viruses (83%-97.9%) were correctly determined for the partial pol sequences. All 10 major DRMs in seven isolates were detected at these six sites. The complete pol gene sequence was also obtained by NGS at one site. However, this method missed six group M viruses and sequences contained host chromosome fragments. Three group O viruses were only characterized with additional group O-specific RT-PCR primers employed by one site. These results indicate that PCR protocols and subtyping tools should be standardized to efficiently amplify diverse viruses and more consistently assign virus genotypes, which is critical for accurate global subtype and drug resistance surveillance. Targeted NGS analysis of partial pol sequences can serve as an alternative approach, especially for detection of low-abundance DRMs.

Variability of the IFN-γ ELISpot assay in the context of proficiency testing and bridging studies

Assays that assess cellular mediated immune responses performed under Good Clinical Laboratory Practice (GCLP) guidelines are required to provide specific and reproducible results. Defined validation procedures are required to establish the Standard Operating Procedure (SOP), include pass and fail criteria, as well as implement positivity criteria. However, little to no guidance is provided on how to perform longitudinal assessment of the key reagents utilized in the assay. Through the External Quality Assurance Program Oversight Laboratory (EQAPOL), an Interferon-gamma (IFN-γ) Enzyme-linked immunosorbent spot (ELISpot) assay proficiency testing program is administered. A limit of acceptable within site variability was estimated after six rounds of proficiency testing (PT). Previously, a PT send-out specific within site variability limit was calculated based on the dispersion (variance/mean) of the nine replicate wells of data. Now an overall ‘dispersion limit’ for the ELISpot PT program within site variability has been calculated as a dispersion of 3.3. The utility of this metric was assessed using a control sample to calculate the within (precision) and between (accuracy) experiment variability to determine if the dispersion limit could be applied to bridging studies (studies that assess lot-to-lot variations of key reagents) for comparing the accuracy of results with new lots to results with old lots. Finally, simulations were conducted to explore how this dispersion limit could provide guidance in the number of replicate wells needed for within and between experiment variability and the appropriate donor reactivity (number of antigen-specific cells) to be used for the evaluation of new reagents. Our bridging study simulations indicate using a minimum of six replicate wells of a control donor sample with reactivity of at least 150 spot forming cells per well is optimal. To determine significant lot-to-lot variations use the 3.3 dispersion limit for between and within experiment variability.

Setting objective thresholds for rare event detection in flow cytometry

The accurate identification of rare antigen-specific cytokine positive cells from peripheral blood mononuclear cells (PBMC) after antigenic stimulation in an intracellular staining (ICS) flow cytometry assay is challenging, as cytokine positive events may be fairly diffusely distributed and lack an obvious separation from the negative population. Traditionally, the approach by flow operators has been to manually set a positivity threshold to partition events into cytokine-positive and cytokine-negative. This approach suffers from subjectivity and inconsistency across different flow operators. The use of statistical clustering methods does not remove the need to find an objective threshold between between positive and negative events since consistent identification of rare event subsets is highly challenging for automated algorithms, especially when there is distributional overlap between the positive and negative events (“smear”). We present a new approach, based on the Fβ measure, that is similar to manual thresholding in providing a hard cutoff, but has the advantage of being determined objectively. The performance of this algorithm is compared with results obtained by expert visual gating. Several ICS data sets from the External Quality Assurance Program Oversight Laboratory (EQAPOL) proficiency program were used to make the comparisons. We first show that visually determined thresholds are difficult to reproduce and pose a problem when comparing results across operators or laboratories, as well as problems that occur with the use of commonly employed clustering algorithms. In contrast, a single parameterization for the Fβ method performs consistently across different centers, samples, and instruments because it optimizes the precision/recall tradeoff by using both negative and positive controls.

Pilot Studies for Development of an HIV Subtype Panel for Surveillance of Global Diversity

The continued global spread and evolution of HIV diversity pose significant challenges to diagnostics and vaccine strategies. NIAID partnered with the FDA, WRAIR, academia, and industry to form a Viral Panel Working Group to design and prepare a panel of well-characterized current and diverse HIV isolates. Plasma samples that had screened positive for HIV infection and had evidence of recently acquired infection were donated by blood centers in North and South America, Europe, and Africa. A total of 80 plasma samples were tested by quantitative nucleic acid tests, p24 antigen, EIA, and Western blot to assign a Fiebig stage indicative of approximate time from initial infection. Evaluation of viral load using FDA-cleared assays showed excellent concordance when subtype B virus was tested, but lower correlations for subtype C. Plasma samples were cocultivated with phytohemagglutinin (PHA)-stimulated peripheral blood mononuclear cells (PBMCs) from normal donors to generate 30 viral isolates (50-80% success rate for samples with viral load >10,000 copies/ml), which were then expanded to 10(7)-10(9) virus copies per ml. Analysis of env sequences showed that sequences derived from cultured PBMCs were not distinguishable from those obtained from the original plasma. The pilot collection includes 30 isolates representing subtypes B, C, B/F, CRF04_cpx, and CRF02_AG. These studies will serve as a basis for the development of a comprehensive panel of highly characterized viral isolates that reflects the current dynamic and complex HIV epidemic, and will be made available through the External Quality Assurance Program Oversight Laboratory (EQAPOL).