Using PRAiS to monitor outcomes after paediatric heart surgery
*** UPDATE APRIL 2014: Version 2.2 of the PRAiS software is now available for download from UCLB E-Lucid ***
As described in a paper published in Heart in April 2013, CORU have developed an Excel file “Monitoring Outcomes with PRAiS” that is designed to allow UK paediatric cardiac centres to
routinely monitor their short term surgical outcomes. This uses data that UK centres collect for the Central Cardiac Audit Database (CCAD), run by the National Institute for Cardiovascular Outcomes Research (NICOR). To purchase this software, please visit the University College London Business E-lucid website.
Outcomes are adjusted for risk using a model (Partial Risk Adjustment in Surgery: PRAiS)
that estimates the risk of death within 30 days of a surgical procedure
based on CCAD specific procedure, age, weight and a patient's recorded diagnoses and
co-morbidities. The original risk model has been published in Crowe
et al. JTCVS 2013 (online July 2012) and was developed on a random 70% subset of ten years’ of UK national audit data and used the 2010 CCAD specific procedure algorithm. It was validated in the remaining 30% of the
national data set which had not been used for model development. The full final report on the risk model development has been published by NIHR in Health Services and Delivery Research and is open access.
Version 1.0 of the PRAiS software used an updated version of the PRAiS risk model which was recalibrated on the full 2007-2010 CCAD data set using the 2012 CCAD specific procedure algorithm. In response to more complete data entry and an apparent improvement in UK outcomes since the 2007-2010 period, we have recalibrated the PRAiS risk model again on cleaned 2009-2012 UK data, with the agreement of the NICOR congenital steering committee, using the 2013 version of the Specific Procedure algorithm (Specific_Procedure_Coding_V4.50 22_05_13.xlsx). All other risk factors are as in the validated version of PRAiS, except small amendments to the allocation criteria for the broad diagnosis risk groups following improvements in diagnosis code completeness. The parameters for these risk factors within the model have changed. After recalibrating in June 2013, it was realised that there were some errors in the national dataset which have since been corrected. We recalibrated the PRAiS risk model on the revised 2009-2012 dataset in April 2014 and have incorporated this calibration into the PRAiS software (version 2.2). (NOTE: there is no version 2.1 of the PRAiS model calibration, but there is a version 2.1 of the software used to implement PRAiS.)
We note that this recalibrated version has not been tested in a pristine data set. Cross validation of a set of 10 parameterisations using 70% samples of the 2009-2012 data in the remaining 30% subsets showed reasonable performance (range of ROC Area under the Curve in 30% subsets (N=3640): 0.76-0.80). As with any risk model, its performance among later data cannot be guaranteed.
The VLAD charts do not incorporate statistical thresholds, but in
response to requests from UK units we have added functionality to estimate
prediction intervals for observed survival in version 2 of the software. A summary of our thoughts on the role of our software in quality assurance and improvement initiatives is given in the Heart paper and in our document called "Use of PRAiS for risk adjustment". We have also published an open-access editorial in Heart discussing the benefits and risks of monitoring risk-adjusted mortality in paediatric cardiac surgery.
Before implementing the risk model, we exclude certain records: duplicated records, records for patients over 16, records which do not pertain to a cardiac procedure or have no procedure codes, records which are for patients who only had catheter procedures or other procedures which are neither "bypass" or "non-bypass". 30-day episodes of care are then generated as follows: Each patient’s first episode starts with their first surgical procedure and the patient’s vital status at 30 days is assigned as a primary outcome. Any further procedures within this 30 day episode constitute a secondary outcome. A surgical procedure more than 30 days after the first procedure constitutes the beginning of a new episode.
Finally, to implement the risk model as given in the specification, the relevant risk factors need to be generated from the raw data. This is done by using mappings from the European Paediatric Cardiac Code Short List (EPCC short codes) for diagnosis and co-morbidity. We also map ten CCAD specific procedures to a single "Low Volume" Category. You can download all the relevant mappings for version 2.
The Excel file “Monitoring Outcomes with PRAiS v2” performs all of this processing for the user on raw data in the format needed for UK national audit, including basic error checking and generation of the CCAD "specific procedure" category. The user can then use the software to generate estimates of risk for all 30 day episodes of care and produce a Variable Life Adjusted Display (VLAD) chart covering the period of the data. VLAD charts allow units to examine their own programme-level outcomes and quickly identify trends in outcomes (positive or negative) that might warrant further investigation. An illustrative VLAD output of the software is shown below. Note that this does not correspond to actual data.