Skip to main content
Download PDF

Analytical validation of an mRNA-based urine test to predict the presence of high-grade prostate cancer

Translational Medicine Communications volume 2, Article number: 5 (2017) Cite this article

Abstract

Background

The challenge for prostate cancer (PCa) diagnosis is to improve the ability to distinguish indolent from aggressive PCa. Increased urinary levels of Distal-less Homeobox 1 (DLX1) and Homeobox C6 (HOXC6) mRNA have been associated with high-grade PCa (Gleason Score ≥7). Reverse transcriptase PCR (RT-PCR)-assays for the quantification of DLX1 and HOXC6 mRNA levels were developed, and clinical validation showed that these urinary markers can significantly improve accuracy for detection of high-grade PCa at biopsy. The aim of this study was to validate the analytical performance of these mRNA-based assays, including specimen stability and analytical sensitivity, specificity, precision, repeatability and reproducibility.

Methods

Analytical validation of the RT-PCR assays for DLX1, HOXC6 and KLK3 was performed using in vitro transcribed (IVT) specimens of the target genes, covering all aspects of the analytical method including assay sensitivity, specificity, linearity, precision, repeatability and reproducibility using pre-specified acceptance criteria. To confirm that the mRNA assays are sufficient robust for use in clinical routine laboratories, the performance characteristics were verified in an independent laboratory using post-DRE collected urine samples from 101 men scheduled for prostate biopsy.

Results

A stabilization buffer, developed for urine preservation under standard pre-processing conditions, makes sample shipment from clinics to laboratories easily feasible. The mRNA in urine samples, preserved in this stabilization buffer, is stable at room temperature up to 5 days from collection, resulting in 100% evaluable rate. The long term stability of the mRNAs in the buffer was shown by similar clinical performances using RNA values obtained immediately after urine collection (area under curve (AUC) 0.72(95% CI: 0.61–0.83)) and after 1-year storage (AUC 0.71 (95% CI: 0.60–0.81)). Test performance was not compromised by most common inhibitors and bacterial strains found in urine. However, an inhibitory effect of hemoglobin was observed.

All precision, reproducibility, instrument and inter-laboratory variation data obtained for the mRNA-based assays met the pre-specified acceptance criteria of a standard deviation less than or equal to 0.5 crossing point. The analysis of 99 whole urine samples at two laboratories indicated a very strong positive correlation (r = 0.997, P < 0.001). The test outcome in terms of absolute difference in likelihood for high-grade PCa upon biopsy was less than 2% between the two sites.

Conclusions

This study illustrates the robustness of the mRNA assays, enabling testing in clinical routine laboratories and molecular pathology laboratories where the here described automated RNA extraction and PCR platforms are available.

Background

In the United States and Europe, prostate cancer (PCa) is one of the most common cancers and one of the leading causes of cancer-related deaths [1, 2]. Globally, in 2012, an estimated 1.1 million men were newly diagnosed with PCa and 307.000 men died from the disease [3]. Autopsy on men aged 80 years and older, who died from other causes, revealed that almost 70% had PCa at time of death [4]. This implies that the majority of PCa’s are clinically insignificant and will have an indolent disease course. Less than 3% of men diagnosed with PCa develop invasive, lethal disease [3]. Since the early 1990’s, serum prostate-specific antigen (PSA) testing is widely used to screen for PCa. However, the limited specificity of PSA has resulted in a large number of negative biopsies in healthy males and the diagnosis of indolent cancers, leading to anxiety for potentially aggressive occult PCa and overtreatment [5]. Therefore, the most important challenge is to improve the ability to specifically identify aggressive PCa at the time of diagnosis and in an early stage of the disease.

Novel biomarkers with high specificity for aggressive disease could aid in the decision for prostate biopsy and hereby help to avoid the problem of unnecessary biopsies, overdiagnosis and overtreatment. Previously, a stepwise selection process of mRNA-profiling data was described, in which an mRNA-panel was identified for predicting the presence of high-grade (Gleason score (GS) ≥7) PCa upon biopsy using urinary sediments collected after digital rectal examination (DRE) [6]. The mRNA panel includes the genes Distal-less Homeobox 1 (DLX1) and Homeobox C6 (HOXC6). DLX1 is involved in the acquisition of epithelial-neuroendocrine differentiation, a characteristic associated with aggressive cancer [7, 8]. HOXC6 is an upregulated homeobox gene in primary, metastasized and castration-resistant PCa [9]. These biomarkers form the basis for a urine test that allows the stratification of patients with high-grade disease. However, analytical validation is key for the successful introduction of these biomarkers to clinical practice.

Because of the ease of collection, and the fact that prostate cells are directly released into the urethra through prostatic ducts after DRE, whole urine has become the future for non-invasive biomarker testing. It is a preferable substrate since it does not require labor-intensive, time-consuming urine processing procedures, whereas mRNA yield is not compromised avoiding that differences in handling of urine can affect the outcome of the analytical assay. To minimize the variability, the post-DRE urine collection procedure was standardized and the sample preprocessing protocol was optimized with a stabilization buffer specifically developed to this end.

For clinical validation of the DLX1 and HOXC6 biomarkers, real-time polymerase chain reaction (RT-PCR) assays were developed specifically designed for analysis of whole urine samples from men considered for prostate biopsy. The RT-PCR assays were developed to quantify DLX1 and HOXC6 mRNA, in addition to Kallikrein-3 (KLK3) mRNA for reference purposes. The normalization to KLK3 mRNA is used to control analytical variability as well as prostate-related RNA input variability. An RNA value was calculated by summing the normalized ratios of DLX1 and HOXC6 mRNA. In whole urine the RNA value was shown to be a good predictor for the detection of high-grade PCa upon biopsy [6, 10]. The robust automated RNA extraction and PCR platforms that form the base of these assays have contributed to the success of these clinical validation studies. The inter- and intra-laboratory variability was minimized through the implementation of standardized techniques which were relied upon for both extraction and molecular analysis procedures [11].

Here the analytical validation of the RT-PCR assays for DLX1, HOXC6 and KLK3 is described, covering all aspects of the analytical method including assay sensitivity, specificity, linearity, precision, repeatability and reproducibility using pre-specified acceptance criteria [12]. To confirm that the mRNA assays are sufficient robust for routine use in diagnostic laboratories, the performance characteristics were verified in an independent Clinical Laboratory Improvement Amendments (CLIA) accredited laboratory.

Methods

Clinical sample collection

Voided urine samples were collected from a cohort of 99 men scheduled for prostate biopsies due to elevated PSA levels, an abnormal digital rectal exam (DRE) or a family history of PCa. Subjects were enrolled from six urological outpatient clinics in the Netherlands between July 2014 and December 2014. Men with verified PCa, with prior PCa diagnosis, undergoing medical therapy known to affect PSA levels, or men that underwent prostate biopsy within 3 months prior to enrolment, or invasive treatment for benign prostatic hyperplasia (BPH) within 6 months prior to enrolment were excluded from the study. All participants signed a consent form that was approved by the institutional review board of the participating clinics. In Table 1, the characteristics of the study population are presented.

Table 1 Characteristics of the study population
Full size table

After a six-stroke DRE, 30 ml of first void urine was collected and 8 ml was immediately transferred to a urine specimen transport tube containing 2 ml stabilization buffer (Catalogue number UTXL32SE, MDxHealth B.V., Nijmegen, the Netherlands). Samples were shipped at room temperature to MDxHealth clinical diagnostic laboratory (Nijmegen, the Netherlands) on the day of collection, after which the samples were stored at -80 °C. One ml of each specimen was used for RT-PCR analysis of the biomarkers in February 2015, and the remainder of the sample was stored at -80 °C. In December 2015, each specimen was thawed and distributed into 4 aliquots of 1.1 ml, which were stored at -80 °C. Two aliquots of each sample were shipped on dry ice to an independent CLIA laboratory (MDxHealth Inc., Irvine, California, USA). Throughout this manuscript, the laboratories in Nijmegen and Irvine will be referred to as Lab A and Lab B, respectively.

RNA extraction

Prior to extraction, the samples were thawed at 37 °C for 15 min, during which the tubes were inverted every three minutes. Automated nucleic acid isolation using the MagNA Pure 96 Instrument (Roche Life Science, Indianapolis, IN, USA) was performed with 1 ml of whole urine per analysis using the DNA and Viral NA large volume kit (Roche Life Science, Indianapolis, IN, USA) according to the manufacturer’s instructions. One ml of nuclease-free water was used as negative nucleic acid isolation control (NAI-). The RNA of each extracted specimen as well as the NAI- control was eluted in 50 μl Elution buffer (Roche Life Science, Indianapolis, IN, USA), the elution plate was sealed and stored in a refrigerator at 2 to 8 °C.

Quantification of DLX1, HOXC6 and KLK3

For the analytical validation, a prototype amplification kit (Labo Biomedical Products B.V., Rijswijk, the Netherlands) was used. The reagents for the test include a singleplex Taqman® Fast virus one-step PCR mastermix (ThermoFisher Scientific, Waltham, MA, USA) for each of the 3 targets, a calibrator (Cal) and a reverse transcriptase (RT)-control. The singleplex mastermixes contain TaqMan® probe and primers for specific detection of either DLX1, HOXC6 or KLK3, reverse transcriptase, Taq polymerase, dNTPs and buffer. The calibrator consists of a multi-target plasmid containing the relevant DLX1, HOXC6 and KLK3 sequences. The RT-control is a mix of in vitro transcribed (IVT) RNA of HOXC6, DLX1, and KLK3 target sequences. The calibrator (Cal 104), quantified to 10,000 copies of the multi-target plasmid, and RT-control, quantified to 10,000 copies of IVT RNAs, were used as positive reference controls in each run. Additionally, a NAI- and a no-template control (NTC) were included in each run. KLK3 was included as an endogenous positive nucleic acid isolation control.

Per RT-PCR reaction, 14 μl of RNA specimen or control was mixed with 6 μl mastermix in a single well of a LightCycler® 480 II multiwell plate (Roche Life Science, Indianapolis, IN, USA). At Lab A, a LightCycler 480 II 96-well plate format was used whereas at Lab B a LightCycler 480 II 384-well plate format was used. Regardless of the plate format, 20 μl reaction was reverse-transcribed and amplified in a LightCycler® 480 II system (Roche Life Science, Indianapolis, IN, USA) using the following cycling conditions: 10 min 50 °C and 30 s 95 °C, followed by 50 cycles each consisting of 5 s 95 °C and 30 s 60 °C.

The levels of gene expression were calculated with the absolute quantification analysis using the Crossing point (Cp) as quantification point and the second derivative maximum method implemented by the LightCycler® 480 II software version 1.5.0 (Roche Life Science, Indianapolis, IN, USA). The normalized ratio (NR) of the target gene, either HOXC6 or DLX1, relative to the KLK3 reference gene, was calculated using the delta delta Cp method (ΔΔCp) [13].

An RNA value was calculated by summing the NR of HOXC6 and DLX1. In the clinical setting, the RNA value is combined with traditional clinical risk factors, including PSA, DRE, prostate volume, age and family history using a logistic regression model described by Van Neste et al. [10]. The output of the clinical test is the likelihood of a patient harboring high-grade PCa upon biopsy.

Analytical specificity

The specificity of the mRNA-based assays was determined both theoretically and analytically, by assessing the cross-reactivity. The theoretical specificity was investigated using the NCBI’s Basic Local Alignment Search Tool (BLAST) (version 2.3.0) tool. Sequences of primers, probes and complete PCR amplicons of the targets DLX1, HOXC6 and KLK3 were checked for homology to pseudogenes, non-target related sequences or sequences of other organisms present in the GenBank® database. The individual sequences of DLX1, HOXC6 and KLK3 and the amplicon sequences were tested using search sets “Human genomic + transcript” and “Others”, including bacteria, viruses and other organisms, using programs MEGABLAST and BLASTN. Human genomic sequences and other organism sequences were evaluated looking for potential binding from the target sequences of the primers and probes using the Primer-BLAST program.

Cross-reactivity was tested at least quadruplicate using genomic DNA of eight bacterial and one yeast species known to be commonly involved in urinary tract infections. Bacteriuria as result of acute cystitis or acute pyelonephritis is defined as the presence of 1.103 or more colony forming units (CFU) per mL of urine. Based on the yield of the bacterial cultures, 1.105 copies DNA/PCR were included for Staphylococcus saprophyticus, Enterococcus faecium and Escherichia coli, while for Klebsiella pneumoniae subspecies pneumonia, Pseudomonas aeruginosa, Proteus mirabilis and Enterobacter aerogenes 1.107 copies DNA/PCR were used. Staphylococcus aureus and the yeast strain Candida albicans were tested at both 1.105 copies DNA/PCR and 1.107 copies DNA/PCR. In addition, RNase-treated human genomic DNA was evaluated at 20 ng/PCR.

Analytical sensitivity

For each of the targets DLX1, HOXC6 and KLK3 serial dilution series of IVT RNAs were made in carrier RNA diluent (Labo Biomedical Products B.V., Rijswijk, the Netherlands) (12 ng/μl), resulting in specimens with IVT RNA concentrations of 640, 320, 160, 80, 40, 20 and 10 copies per RT-PCR reaction. To assess the limit of detection (LOD) and the lower limit of quantification (LLOQ) of the DLX1, HOXC6 and KLK3 assays, 12 replicates of each IVT dilution were tested in 4 individual RT-PCR experiments. The LOD was determined as the lowest IVT concentration that could be detected in ≥95% of the samples. The LLOQ was determined as the lowest IVT concentration that could be measured with acceptable accuracy, determined as a standard deviation (SD) <0.5 Cp.

Linearity

To determine linearity, specimens were used with DLX1, HOXC6 and KLK3 IVT RNA concentrations of 107, 106, 105, 104, 103, 102 copies per RT-PCR reaction. The lowest point was defined as 20% below LLOQ resulting in 16 copies/RT-PCR for DLX1 and KLK3 and 32 copies/RT-PCR for HOXC6. For each target, six replicates of each concentration were tested in 5 individual RT-PCR experiments. Linearity was assessed using the polynomial method by fitting quadratic and cubic regression models and comparing whether these models were a better fit than linear regression [14]. The assays were considered linear if the cubic and quadratic variables for a 2nd or 3rd order polynomial were not statistically significant (P > 0.05) [15]. If either one of these higher-order terms failed to meet the linearity criterion, the deviation from linearity (DLi) was assessed using a weighted linear regression of the normalized Cp values versus the logarithm of the copy-numbers for each input concentration. If the absolute maximum DLi across concentrations was ≤ 0.5 Cp, the deviation was considered minimal and linearity was established for the assay.

Precision

Precision assessment included PCR instruments, inter-run, operator and intra-run variation. The overall precision of the assays was determined for DLX1, HOXC6 and KLK3 using three dilutions of IVT RNA’s with carrier RNA diluent (12ng/μl) (Labo Biomedical Products B.V., Rijswijk, the Netherlands). The lowest copy numbers were defined as 2x (LLOQ + 3x SD), the other concentrations were based on 5x and 20x LLOQ. This resulted in specimen with DLX1 concentrations of 400, 100 and 76 copies per RT-qPCR, KLK3 concentrations of 400, 100 and 67 copies per RT-PCR and 800, 200 and 125 copies of HOXC6 per RT-PCR. Eight replicates of each IVT concentration were tested using three LightCycler 480 II instruments with a 384-well plate format (Roche Life Science, Indianapolis, IN, USA), by 3 operators on 3 consecutive days. Specific sub-analyses of this dataset allowed the analysis of the intra-run (repeatability), inter-run precision (reproducibility), inter-instrument and inter-operator precision, with the other factors being fixed.

Precision assessment of the combination of MagNA Pure 96 Instrument and LightCycler® 480 II instrument (Roche Life Science, Indianapolis, IN, USA) was done using urine samples of anonymous healthy males, stabilized in stabilization buffer as previously described. For this purpose, 2000 LNCaP cells/mL were added to the urine after which the samples were distributed into 4 aliquots of 1.1 mL and stored at -80 °C. RNA extraction and PCR amplification were performed in triplicate by 2 operators on 4 days.

All assays were considered precise if the SD ≤ 0.5Cp. The reproducibility and robustness of the assays were further assessed during an external precision experiment, in which a total of 99 post-DRE urine specimens were tested side-by-side study in two independent laboratories (Lab A and Lab B). Beforehand, it was defined that between laboratories the urine test outcome in terms of absolute difference in likelihood for high-grade PCa upon biopsy was not allowed to be more than 2%.

Interference testing

To study the potential loss of assay sensitivity due to interfering endogenous substances that can occur in urine, e.g. uric acid, hemoglobin, white blood cells, red blood cells, albumin, bilirubin and human IgG, high concentrations of these potential interfering substances were added to urine (Table 2). For this purpose, mock urine samples were prepared using pooled healthy donor urine spiked with the biomarker-positive cell line LNCaP at two concentrations. In total, two urine samples containing 20,000 cells/mL and two urine samples containing 10,000 cells/mL were tested in duplicate.

Table 2 Test concentrations versus normal concentration of endogenous substances in urine
Full size table

Westgard et al. defined guidelines for the interpretation of a control result that is 1SD, 2SD, or 3SD from the mean [16]. The average SD of 0.3 used in this analysis was obtained from the precision experiments done for the instrument combination. The Westgard rules were used for data analysis of the interference study with a difference in Cp value of less than 1 SD scored 0 (within the 67% confidence interval (CI)), less than 2 SD scored 1 (within the 95% CI), less than 3 SD (within the 99% CI) scored 2 and differences larger than 3 SD scored 3. When using a specific candidate interfering substance, multiple scores of 2 and 3 were considered inhibition.

Effect of the stabilization buffer on clinical performance

The performance of the stabilization buffer was validated through a direct comparison of the RNA value obtained from urine samples collected in the stabilization buffer from a cohort of 99 men scheduled for prostate biopsies and the RNA value obtained from samples in Progensa® urine transport medium (Hologic Inc., San Diego, CA, USA). The latter data were obtained from two published, independent prospective multicenter studies, clinical study A (n = 519 samples) and B (n = 386 samples) [10].

Stabilization buffer stability

The stability of urine specimens in the stabilization buffer was evaluated by processing urine of five patients incubated at 4 °C, room temperature and 30 °C up to 15 days after collection. From the start, all samples were split into two aliquots and analyzed in duplicate after which RNA extraction, RT-PCR and RNA value calculation were performed.

The influence of transit time on RNA value was determined using the 99 samples processed in the stabilization buffer. The number of samples received on Day 1 (day of collection) and Days 2, 3, 4–6 were 56, 16, 17 and 10 respectively. Furthermore, the effect of decline in RNA value due to RNA degradation on the likelihood score for high-grade PCa was studied. The 56 samples received at the day of collection (Day 1) were used for this analysis as these samples were considered to be least affected by RNA degradation. Additionally, the RNA stability was tested after 1-year storage at -80 °C using the 99 samples processed in the stabilization buffer.

Statistical analysis

Statistical analyses were performed with SPSS v.20.0 (IBM Corp., Armonk, NY, USA) and R v.3.2.1 (R Foundation for Statistical Computing, Vienna, Austria). Because calculated RNA values are continuously increasing with patient risk, the performance was assessed and evaluated as area under the curve (AUC) of the receiver operating characteristic (ROC). The 95% confidence intervals and comparisons of AUCs were determined using DeLong’s method as implemented in the R package pROC [17]. The non-parametric Wilcoxon signed-rank test was used to compare the results obtained for 99 urine samples tested in Lab A and Lab B. Analysis of Variance (ANOVA) and the Kruskal-Wallis test were used to test for differences between the medians of RNA values over transit days at room temperature. For each comparison between two groups, if P <0.05, the conclusion is that they differ significantly from each other.

Results

Analytical specificity

BLAST searches revealed that the target sequences of DLX1, HOXC6 and KLK3 showed no homology to the DNA of bacteria, viruses, other organisms or other human DNA or RNA sequences. Cross-reactivity experiments were conducted to determine the practical specificity, using human genomic DNA and genomic bacterial DNA as template for the mRNA-based assays. No cross-reactivity was observed for human genomic DNA and most bacterial strains and yeast strain as obtained Cps were below the LOD. Some genomic bacterial DNA’s resulted in a very low signal for KLK3 only, with the highest result for Enterobacter aerogenes (max. 113 copies). However, the amplification curves of this bacterial strain had an aberrant pattern and analysis of the PCR products using a TapeStation instrument (Agilent Technologies Inc., Santa Clara, CA, USA) confirmed that the signal obtained was non-specific (data not shown). Even if the signal cannot be separated from a true positive one, it would account for only <1% of the reference gene KLK3 and the impact on test results can be considered negligible.

Analytical sensitivity

The analytical performance of the mRNA-based assays was assessed and the established limits of detection and quantification are depicted in Table 3. The LLOQ for KLK3 and DLX1 was determined at 20 copies/RT-PCR and the LLOQ for HOXC6 was determined at 40 copies/RT-PCR. However, repeat experiments to verify this difference of LLOQ between HOXC6 versus KLK3 and DLX1, showed that an LLOQ of 20 copies/RT-PCR could also be established for HOXC6. All assays had LODs ranging from Cp 35.97–38.26, corresponding to 10 copies of target RNA. The LLOQs ranged from Cp 35.20–36.01, indicating that all assays can be accurately quantified from 20 copies of target RNA upward. For all RT-PCR assays the ULOQ was 107 copies/RT-PCR, as determined in the linearity experiment, establishing a working range of 10 to 107 copies for all genes.

Table 3 The analytical measurement range of the assays for DLX1, HOXC6 and KLK3
Full size table

Linear range

Both the DLX1 and HOXC6 mRNA-based assays were linear over the entire range of dilutions as established by the non-significant coefficients of the cubic and quadratic variables (P > 0.05) in the second and third order polynomial regression analyses (Table 4). The Cp values ranged from 16.74 to 36.49 (corresponding with 107–16 copies/RT-PCR) for DLX1, and from 16.40 to 36.30 (corresponding with 107–32 copies/RT-PCR) for HOXC6.

Table 4 P-values associated with the polynomial regression coefficients for the dilution series
Full size table

Since the cubic and quadratic variables were statistically significant for KLK3 (P < 0.001), this assay was not considered linear when all seven concentrations were taken into account (Table 4). If only five dilutions with ≥1000 copies KLK3/RT-PCR were included, the assay was linear. To assure sufficient input material for the clinical assay, a minimum of 10,000 copies of KLK3 need to be present. Hence, the assay is linear in the range of its clinical performance.

The deviation from linearity (DLi) was assessed for the KLK3 mRNA-based assay. When all seven concentrations were taken into account, the DLi was within the preset limits and hence, the KLK3 assay was considered sufficiently linear, even when fewer than 1000 copies were present (Fig. 1).

Fig. 1
figure 1

DLi determined for the KLK3 mRNA assay is illustrated by this plot in which the average DLi (red asterix) and 95% CI are plotted against the RNA concentration for each sample

Full size image

In summary, all mRNA-based assays can accurately quantitate their RNA target sequence over the entire evaluated range of concentrations, encompassing those that can occur in whole urine specimens.

Precision

Precision of the mRNA-based assays was evaluated under various conditions covering inter-run, PCR instrument and operator variation. The average Cp and SD Cp of the results are shown in Table 5. The overall precision was determined by testing the reproducibility of a sample tested at three concentrations in 8 replicates between three operators, three instruments and 3 days resulting in SD of the Cp ranging from 0.1 to 0.3.

Table 5 Precision of DLX1, HOXC6, and KLK3 mRNA-based assays
Full size table

Intra-run precision was based on 8 data points per concentration, generated by one operator on the same day and on the same machine resulting in SD of the Cp ranging from 0.1 to 0.5. Inter-run precision was based on 24 data points per concentration, generated by one operator on three different days, with the SD of the Cp ranging from 0.1 to 0.4. Inter-instrument precision was based on the 24 data points per concentration, generated on three different PCR instruments by one operator on the same day, resulting in SD of the Cp ranging from 0.1 to 0.4. Finally, inter-operator precision was based on 24 data points per concentration, generated by three different operators on the same day and on the same PCR instrument, resulting in SD of the Cp ranging from 0.1 to 0.3.

The precision of MagnaPure and LightCycler® 480 II Instruments combination was based on 33 data points generated by two operators on four different days using aliquots of urine mixed with 2000 LNCaP cells/mL. An average Cp of 22.7 with an SD of 0.3Cp was obtained for KLK3, a Cp of 28.2 with an SD of 0.3Cp for HOXC6 and a Cp of 31.9 with an SD of 0.3Cp for DLX1. All precision measurements demonstrated a robust analytical performance, with overall precision SD of maximum 0.3Cp being within the acceptance criteria of ≤0.5.

Interference testing

The most common urinary substances were evaluated for potential interference with the actual assay outcome, i.e. hemoglobin, uric acid, bilirubin, albumin, white blood cells, red blood cells and monoclonal immunoglobulins such as human IgG and the Westgard scores are shown in Table 6.

Table 6 Westgard scores for interference testing
Full size table

Most constituents were shown to be no inhibitors of the assays. However, in half of the samples studied, hemoglobin resulted in Westgard scores of 2 for DLX1 and HOXC6. While there is typically no free hemoglobin present in urine, it appears in urine of patients with hemoglobinuria due to e.g. acute glomerulonephritis, kidney cancer, malaria or pyelonephritis, giving urine a purple color. Red blood cells will be absent in urine of patients with hemoglobinuria, in contrast to hematuria, as a result of urinary tract infections or bladder cancer. Although intact red blood cells do not impact the test results, free hemoglobin due to damaged red blood cells could act as an inhibitor.

Clinical performance bridging study

The clinical performance of the mRNA-based assays was validated through an external precision experiment, performed in an independent laboratory. To this purpose, RNA was extracted from 99 whole urine samples collected post-DRE. Sample validity was based on KLK3 in the sample, which can be considered as a measure for the presence of cellular or exosome content originating from the prostate, and the LOD of the DLX1 and HOXC6 mRNA assays. The decision tree to determine sample validity is shown in Fig. 2.

Fig. 2
figure 2

Decision tree to determine sample validity

Full size image

The number of valid samples was 99 resulting in an informative rate of 100%. The inter-laboratory variation was determined for each mRNA-based assay. Per laboratory, the Cp value of the sample was subtracted from the Cp value obtained for the calibrator of each target. This was only done for Cp values ≤ LLOQ, since these can be accurately quantified. For each sample the difference between the Cp values obtained at Lab A and Lab B was calculated per target (Fig. 3).

Fig. 3
figure 3

Box plot showing the absolute difference in Cp for each target gene obtained for 99 urine samples analyzed at Lab A and Lab B. The box represents the inter-quartile range and the line in the box represents the median. The whiskers represent the minimum and maximum values observed. The open circles represent outliers

Full size image

Non-inferiority testing showed that the absolute difference between the Cp values was not significantly higher than the 0.5Cp defined in the precision experiments as the maximum variation of the mRNA-based assays (P = 0.8 for DLX1, P = 0.1 for HOXC6 and P = 1.0 for KLK3), showing robust performance of the test between laboratories.

The diagnostic performance of the mRNA-based assays for DLX1, HOXC6 and KLK3 was determined using the RNA value obtained for these 99 samples analyzed independently at Lab A and Lab B. The ROC was used to visualize the diagnostic efficacy of the RNA value for the detection of high-grade PCa. The AUC of the ROC obtained at Lab A (AUC = 0.71 (95% CI:0.60–0.81)) was not statistically significant different (P = 0.2) from the AUC obtained at Lab B (AUC = 0.73 (95% CI:0.62–0.83)).

To further validate the robustness of the assays between both laboratories, the outcome of the previously published logistic regression prediction model comprising the RNA values and traditional clinical risk factors i.e. PSA, prostate volume, DRE and family history, was used as validation [10]. A strong correlation coefficient (r = 0.997) was obtained between both laboratories as shown in Fig. 4, and this correlation was shown to be significant (P = 2.2.10-16). The clinical performance of the likelihood scores for the detection of high-grade PCa at Lab A (AUC = 0.92 (95% CI:0.84–0.99)) was not statistically significant different (P = 0.3) from the clinical performance at Lab B (AUC = 0.92 (95% CI:0.85–0.99). These results are in line with the clinical performance (AUC = 0.90 (95% CI:0.85–0.95)) obtained in the validation study recently described by Van Neste et.al. [10].

Fig. 4
figure 4

Scatter plot showing the correlation between the likelihood profiles for high-grade PCa upon biopsy obtained for 99 urine samples analyzed at Lab. A and Lab. B with y = 0.982, r = 0.997 and P < 0.001

Full size image

The absolute difference between the results in terms of likelihood for high-grade PCa upon biopsy was shown not to be significantly higher than 2% (P = 1.0), showing robust performance of the test between laboratories.

Clinical performance stabilization buffer

To avoid RNA degradation, the samples should be stabilized immediately after urine collection and for this purpose a stabilization buffer was developed. Assay performance for the prediction of high-grade PCa upon biopsy using samples collected in this buffer (Clinical trial C: AUC 0.71 (95%CI: 0.60–0.81) was comparable to the performance using samples collected in Progensa® urine transport medium (Clinical Trial A: AUC 0.76 (95%: 0.71–0.81) and Clinical trial B: AUC 0.73 (95%: 0.67–0.78)), as indicated by the non-significant differences in AUC’s (Trial A versus C; P = 0.4, Trial B versus C; P = 0.8).

Stabilization buffer and assay robustness

To determine the maximum allowable transport conditions for sample transfer from clinic to the laboratory, the robustness of the RNA value was evaluated over a period of 2 weeks at 4 °C, room temperature and 30 °C. The RNA values were calculated relative to the average RNA value obtained on Day 1 (Fig. 5). Until Day 5 from collection, there was no significant signal degradation for samples stored at room temperature and 4 °C (P = 0.5 for storage duration and P = 0.6 for storage temperature).

Fig. 5
figure 5

Influence of temperature and time on RNA value using samples collected in stabilization buffer. The influence of RNA degradation on RNA level should not be greater than 15% as indicated by dotted line at 85%

Full size image

Post-DRE collected whole urine in stabilization buffer resulted in max. 15% decrease of RNA value up to 4 days after collection at either 4°C or room temperature. At 30 °C, the 15% decrease in RNA value was already obtained on Day 2 from collection. Temperature had a significant effect on the RNA value for longer storage periods (P < 0.05). However, up to Day 4 from collection, neither temperature (P = 0.1) nor storage duration (P = 0.3) had a significant effect on the RNA value.

The influence of transit time on RNA value was also determined using the 99 evaluable urine samples in stabilization buffer (Fig. 6) and was shown to be minimal as no significant differences (P = 0.9) were observed between the medians of RNA values obtained for Day 1 (56 samples) through Day 4 (10 samples).

Fig. 6
figure 6

Boxplot summarizing the influence of time on RNA value using samples collected in stabilization buffer. The box represents the inter-quartile range and the line in the box represents the median. The whiskers represent the minimum and maximum values observed

Full size image

The influence of RNA degradation on the likelihood for high-grade PCa was determined. Using the 56 samples collected on Day 1, the influence of 15% RNA value decrease on the likelihood for high-grade PCa was calculated and resulted in ≤2% change in likelihood score for high-grade PCa.

Finally, the long-term stability was tested before and after a storage period of 1 year at -80 °C. The clinical performance using RNA values obtained immediately after urine collection (AUC 0.72(95% CI: 0.61–0.83)) and after 1-year storage (AUC 0.71 (95% CI: 0.60–0.81)) was not affected, as illustrated by the non-significant difference between the AUC’s (P = 0.5).

Discussion

The major challenge in PCa detection and management is to differentiate aggressive from non-aggressive cancer at an early stage of this disease. Currently, PCa diagnosis is driven by PSA, fostering overdiagnosis and overtreatment of indolent tumors. A test that would better separate men with high-grade PCa from those with indolent or no disease is urgently needed. Previously, it was shown that high-grade PCa could be detected using a urinary biomarker-based risk score [10]. This test, based on two mRNA assays for the DLX1 and HOXC6 biomarkers, was developed on the LightCycler 480 II platform for the specific detection of high-grade PCa in urinary samples. KLK3 transcripts, considered as a measure for the presence of content from prostatic origin, i.e. cells or exosomes, was used as reference marker to normalize the DLX1 and HOXC6 signals. In men considered for biopsy, e.g. due to elevated PSA, the RNA value combined with other clinical risk factors led to an improved patient risk stratification for high-grade PCa [10].

Although clinical validity is of obvious importance, analytical validity and ease of use are essential for the successful implementation of novel diagnostic tests into clinical practice. In this report, we have described the stability testing of mRNA using a stabilization buffer and a simplified urine sample collection method, and demonstrated the robust analytical performance of the quantitative mRNA assays.

An important aspect of this study is the comparison of the clinical performance characteristics of the mRNA-based assays between two independent laboratories using whole urine collected post-DRE from a cohort of 99 men considered for prostate biopsy. For each target it was shown that the absolute difference in Cp per sample was not significantly higher than the maximum variation obtained in the precision studies for the assays (0.5 Cp). This is an important aspect assuring assay robustness, as it implicates that the variation introduced by different MagNa Pure 96 instruments and different laboratories does not significantly affect the overall precision or influence the patient’s result. Whole urine collected post-DRE from a cohort of 99 men considered for prostate biopsy showed no statistical difference in clinical performance between two laboratories. Furthermore, an absolute difference between the results in terms of likelihood for high-grade PCa upon biopsy was shown to be significantly lower than 2%, demonstrating the robust performance of both the standardized automated RNA extraction and RT-PCR assays.

Overall, the mRNA-based assays for DLX1, HOXC6 and KLK3 exhibited a precise and robust behavior, with the natural variation or experimental noise being the main factor influencing the outcome. The variation introduced by the operator, different runs, PCR instrument or even the combination of automated RNA extraction and PCR platform (96 well versus 384 well format), RNA extraction instrument and different laboratory settings was limited and considered well within the allowable limits. The assay results were accurate and reproducible for the quantification of expression levels of the three targets over a wide range of concentrations, covering the typical content of a post-DRE whole urine sample. This illustrates that the mRNA-based assays are suitable for clinical testing, not only in routine or reference laboratories, but also in general molecular pathology laboratories.

High quality urine samples are key to minimize potential bias in results. Therefore, no more than 30 ml first catch urine should be collected after a six-stroke DRE in order to prevent false negative results due to a dilution effect. To avoid RNA degradation, a stabilization buffer was developed allowing temporary preservation of the mRNA in whole urine and resulting in a high rate of evaluable samples under routine collection conditions. Easy sample shipment from clinics to laboratories is feasible as the RNA of urine samples in this stabilization buffer is stable at room temperature up to 5 days from collection, assuring standardized, reproducible and reliable results. Using this buffer, the clinical performance was comparable to the test performance shown in two independent clinical studies using Progensa® urine transport medium. Furthermore, it was demonstrated that RNA in whole urine collected in this stabilization buffer was stable for 1 year at -80 °C, as the same clinical performance in terms of AUC was obtained.

The influence of inhibitory substances that may occur in urine on the assays was also evaluated. Although intact red blood cells in urine were not inhibitory, free hemoglobin was indicated as a possible inhibitor of the mRNA-based assays for HOXC6 and DLX1. Hemoglobin is known to potentially act as an inhibitor to PCR reactions [18]. The inhibition of hemoglobin on these assays was surprising, since the manufacturer of the PCR master mix specially recommended it for use with whole blood samples. However, the results should be interpreted with caution as the evaluated hemoglobin concentration hemoglobin (2 mg/ml) is the equivalent of ~69,000,000 lysed red blood cells containing 29 pg hemoglobin per cell [19]. This high concentration even caused clotting of the compound in urine sample 1 (Table 6) showing no inhibition. Such high concentrations of hemoglobin are only representative for patients with gross hematuria and patients with hemoglobinuria presenting with purple urine as result of the presence of free hemoglobin. Gross hematuria only occurs in 4 per 1000 urological patients, and has not been observed in over 1000 urine samples that were tested in clinical validation studies [10, 20]. In case of gross hematuria, intact red blood cells will contain most of the hemoglobin, hence free hemoglobin is not likely to pose a problem. Testing inhibition in urine samples of patients instead of inhibitory compounds, will yield a more accurate reflection of diagnostic test performance.

The automated RNA extraction and PCR platform used in this validation study contribute to the robust performance of the mRNA assays. Easy-to-use and efficient handling is provided by both platforms allowing standardized extraction and analysis of large numbers of samples. Since platform or reagent change might affect the clinical performance of the assays, it is recommended to routinely monitor the analytical characteristics using appropriate quality controls for each batch of samples analyzed, guaranteeing reliable results for the likelihood of high-grade PCa.

Conclusion

The analytical validation of the mRNA-based assays for DLX1, HOXC6 and KLK3 indicate a precise and robust performance. A standardized whole urine sample collection procedure, a widely used automated RNA extraction and PCR platform contribute to the overall good performance of the assays. Results of this study illustrate the robustness of the assays, enabling testing in routine laboratories but also in general molecular pathology laboratories that have both platforms available. Independent clinical studies have shown that the RNA value can complement and improve upon the existing pre-biopsy risk assessment tools, such as PSA, to predict the likelihood of finding high-grade PCa upon biopsy. The combination of these mRNA-based assays with simple specimen processing will provide physicians with test results from a robust and reliable mRNA-based urine test.

Abbreviations

AUC:

Area under curve

CLIA:

Clinical Laboratory Improvement Amendments

Cp:

Crossing point

DLi:

Deviation from linearity

DLX1:

Distal-less Homeobox 1

DRE:

Digital rectal examination

GS:

Gleason score

HOXC6:

Homeobox C6

IVT:

In vitro transcribed RNA

KLK3:

Kallikrein 3

NAI-:

Nucleic acid isolation control

NTC:

No-template control

PCa:

Prostate cancer

PSA:

Prostate-specific antigen

RT-PCR:

Reverse transcriptase-polymerase chain reaction

ΔΔCp:

Delta delta Cp method

References

  1. Ferlay J, Steliarova-Foucher E, Lortet-Tieulent J, Rosso S, Coebergh JWW, Comber H, Forman D, Bray F. Cancer incidence and mortality patterns in Europe: estimates for 40 countries in 2012. Eur J Cancer. 2013;49:1374–403.

    Article  CAS  PubMed  Google Scholar 

  2. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2016. CA Cancer J Clin. 2016;66:7–30.

    Article  PubMed  Google Scholar 

  3. Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A. Global cancer statistics, 2012. CA Cancer J Clin. 2015;65:87–108.

    Article  PubMed  Google Scholar 

  4. Yatani R, Chigusa I, Akazaki K, Stemmermann GN, Welsh RA, Correa P. Geographic pathology of latent prostatic carcinoma. Int J Cancer. 1982;29:611–6.

    Article  CAS  PubMed  Google Scholar 

  5. Postma R, Schröder FH, van Leenders GJLH, Hoedemaeker RF, Vis AN, Roobol MJ, van der Kwast TH. Cancer detection and cancer characteristics in the European Randomized Study of Screening for Prostate Cancer (ERSPC)--Section Rotterdam. A comparison of two rounds of screening. Eur Urol. 2007;52:89–97.

    Article  PubMed  Google Scholar 

  6. Leyten GHJM, Hessels D, Smit FP, Jannink SA, de Jong H, Melchers WJG, Cornel EB, de Reijke TM, Vergunst H, Kil P, Knipscheer BC, de Kaa CAH-V, Mulders PFA, van Oort IM, Schalken JA. Identification of a Candidate Gene Panel for the Early Diagnosis of Prostate Cancer. Clin Cancer Res. 2015;21:3061–70.

    Article  CAS  PubMed  Google Scholar 

  7. Altintas DM, Allioli N, Decaussin M, de Bernard S, Ruffion A, Samarut J, Vlaeminck-Guillem V. Differentially expressed androgen-regulated genes in androgen-sensitive tissues reveal potential biomarkers of early prostate cancer. PLoS One. 2013;8:e66278.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Merlo GR, Zerega B, Paleari L, Trombino S, Mantero S, Levi G. Multiple functions of Dlx genes. Genes Dev. 2000;44:619–26.

    CAS  Google Scholar 

  9. Hamid ARAH, Hoogland AM, Smit F, Jannink S, van Rijt-van de Westerlo C, Jansen CFJ, van Leenders GJLH, Verhaegh GW, Schalken JA. The role of HOXC6 in prostate cancer development. Prostate. 2015;75:1868–76.

    Article  CAS  PubMed  Google Scholar 

  10. Van Neste L, Hendriks RJ, Dijkstra S, Trooskens G, Cornel EB, Jannink SA, de Jong H, Hessels D, Smit FP, Melchers WJG, de Reijke TM, Vergunst H, Kil P, Knipscheer BC, de Kaa CAH-V, Mulders PFA, van Oort IM, Van Criekinge W, Schalken JA: Detection of High-grade Prostate Cancer Using a Urinary Molecular Biomarker-Based Risk Score. European Urology 2016;70:740–748.

  11. Espy MJ, Uhl JR, Sloan LM, Buckwalter SP, Jones MF, Vetter EA, Yao JDC, Wengenack NL, Rosenblatt JE, Cockerill FR, Smith TF. Real-time PCR in clinical microbiology: applications for routine laboratory testing. Clin Microbiol Rev. 2006;19:165–256.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Mattocks CJ, Morris MA, Matthijs G, Swinnen E, Corveleyn A, Dequeker E, Müller CR, Pratt V, Wallace A. A standardized framework for the validation and verification of clinical molecular genetic tests. Eur J Hum Genet. 2010;18:1276–88.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Schmittgen TD, Livak KJ. Analyzing real-time PCR data by the comparative C(T) method. Nat Protoc. 2008;3:1101–8.

    Article  CAS  PubMed  Google Scholar 

  14. Krouwer JS, Schlain B. A method to quantify deviations from assay linearity. Clin Chem. 1993;39:1689–93.

    CAS  PubMed  Google Scholar 

  15. Tholen DW, Kroll MH, Astles JR, Caffo AL, Happe TM, Krouwer JS, Lasky F. EP06-A: Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline. Clin Lab Standards Inst. 2003;23:1.

    Google Scholar 

  16. Westgard JO, Groth T, Aronsson T, Falk H, de Verdier CH. Performance characteristics of rules for internal quality control: probabilities for false rejection and error detection. Clin Chem. 1977;23:1857–67.

    CAS  PubMed  Google Scholar 

  17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45.

    Article  CAS  PubMed  Google Scholar 

  18. Al-Soud WA, Rådström P. Purification and characterization of PCR-inhibitory components in blood cells. J Clin Microbiol. 2001;39:485–93.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Sarma PR: Red Cell Indices In: Walker HK, Hall WD, Hurst JW, editors. Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd edition. Boston: Butterworths; 1990. Chapter 152.

  20. Bruyninckx R, Buntinx F, Aertgeerts B, Van Casteren V. The diagnostic value of macroscopic haematuria for the diagnosis of urological cancer in general practice. Fam Pract. 2003;53:31–5.

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Shannon Dempsey, Gabriel Alvarado, Jane Pham and Dagoberto Gomez of MDxHealth Inc. for their contribution to the practical work and data analysis and Dr Jack Groskopf of MDxHealth Inc. for critically reading the manuscript, the urologists of the participating clinics and all patients that gave their consent.

Funding

None.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on request.

Authors’ contributions

DH participated in research design, participated in data analysis and writing of the paper. HdJ participated in research design, performance of the research and participated in data analysis. SJ participated in research design, performance of the research and participated in data analysis. MC participated in research design, performance of the research and participated in data analysis. MK participated in research design, performance of the research and participated in data analysis. WVC participated in research design and data analysis and was supervisor of the study. LVN participated in data analysis and writing of the paper. JS participated in research design, writing of the paper and was supervisor of the study. All authors read and approved the final manuscript.

Competing interest

The authors (DH,HdJ,SJ,MC,MK) declare that they are employees of MDxHealth. WVC, LVN, JS are consultants for MDxHealth.

Consent for publication

Not applicable.

Ethics approval and consent to participate

All participants signed a consent in accordance with the Declaration of Helsinki -based form that was approved by the institutional review board of the participating clinics (Radboud University Medical Center Nijmegen, CWZ Hospital Druten, St. Elisabeth Hospital Tilburg, Scheper Hospital Emmen, ZGT Hospital Hengelo, Slingeland Hospital Doetinchem).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations

  1. MDxHealth B.V., Nijmegen, The Netherlands

    Daphne Hessels, Hans de Jong & Sander A. Jannink

  2. MDxHealth Inc., Irvine, CA, USA

    Michael Carter & Manuel Krispin

  3. Department of Mathematical Modelling, Statistics and Bioinformatics, Ghent University, Ghent, Belgium

    Wim Van Criekinge

  4. Department of Pathology, Maastricht University Medical Center, Maastricht, The Netherlands

    Leander Van Neste

  5. Department of Urology, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands

    Jack A. Schalken

Authors
  1. Daphne Hessels
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hans de Jong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sander A. Jannink
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Carter
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Manuel Krispin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wim Van Criekinge
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Leander Van Neste
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jack A. Schalken
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jack A. Schalken.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hessels, D., de Jong, H., Jannink, S.A. et al. Analytical validation of an mRNA-based urine test to predict the presence of high-grade prostate cancer. transl med commun 2, 5 (2017). https://doi.org/10.1186/s41231-017-0014-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41231-017-0014-8

Keywords

Translational Medicine Communications

ISSN: 2396-832X

Contact us