Have you ever wondered how hospital scores are created? As we have explored in previous posts, there are a number of organizations and companies that publish hospital scores and ranks for the consumer, in an effort to help individuals make educated choices about their healthcare while also making medical facilities more transparent about their successes and challenges (to varying degrees of success). But where do these scores come from?
The quick answer is that these scores are generated when the scoring organization pulls publicly-available data (and/or their own proprietary data) about hospitals and runs it through their unique methodology of measures, weights, and exclusions.
The long answer is a bit more complicated.
First, the data. There is a vast amount of publicly-available data about U.S. hospitals and healthcare facilities. This information is aggregate, that is, taken from various sources, using a variety of measurements and then is combined into a summary. (There is no personal or protected health information, such as patient names, included in this data.) The main sources of this public data, which anyone can access, are:
Second, the measures. All this data is overwhelming to the point of being useless to the average consumer. To create a meaningful summary of this data, agencies and organizations have created summaries that take select pieces of data that align with their intended focus and present them in a user-friendly version. These select pieces of data can be called “measures,” “criteria,” “indicators,” or other similar terms, but basically mean a list of specific attributes they will measure using the data. For example, one organization rating hospitals may come up with a list of 15 attributes of a good hospital and then pull specific data to see how each hospital measures on those 15 attributes. However, another organization rating hospitals may come up with 27 attributes, or 5, and those attributes may be completely different from each other, depending on the focus of the score or ranking.In addition to these sources, there are professional associations, such as the American Hospital Association, which collect proprietary data that can then be purchased by third-party consumers.
Reputable organizations will have a panel of experts who determine these attributes based on published literature and established work by healthcare organizations. For example, The Leapfrogs Group's Hospital Safety Score's 28 measure include seven measures from AHRQ’s 20 Patient Safety Indicators and eight of the NQF’s 34 Safe Practices for Better Healthcare. Selecting the measures is a critical component of the scoring methodology (which should be made available to any consumer on the agency’s or organization’s website) and these measures, along with their relative weights, can dramatically influence how a hospital scores.
Third, the weights. Now that the attributes, or measures, have been determined, the relative weight of each measurement must be established if an overall score is to be determined. Just as a teacher weighs a test score more heavily than a quiz score in coming up with a final grade, hospital scoring attributes are given individual weights. Again, these weights are determined by an expert panel based on published research.
If no overall score is given, then the performance on each measure is provided. This performance can be in comparison to national averages, predicted outcomes, or other comparisons.
Finally, the exclusions. In the world of statistics, some data points, while true, can influence the analysis of a data set so much as to make that analysis not accurate. These data points should not be ignored, but they can be excluded from analysis in order to generate a more accurate “big picture.” For example, the scoring methodology for HealthGrades excludes patients who left the hospital against medical advice, a decision that would ideally be backed up by research that demonstrates that patients in this category experience confounding factors that muddy the statistical waters.
Another important aspect of statistical analysis of data is the idea of “risk-adjustment.” To compare hospitals using the same scoring methodology, the context of the hospital and the patient is very important in order to make those comparisons fair. In the case of a hospital-to-hospital comparison of mortality rates, for example, it is important to take into consideration if a hospital performs more high-risk operations. When it comes to statistics about a particular procedure, the age and health of patients before the procedure also influence an analysis of outcomes. How a group determines risk-adjustment is a component of their methodology, and should be defended with research – but it is not always disclosed to the public.
The final score. Just as the results of each hospital score will vary according to how they choose to design their methodology, the way they present the results varies as well. For example,
- HealthGrades calculates a predicted score for each hospital based on cohorts of patients for specific conditions or procedures to which it compares the actual outcomes, giving a rating of “Better than Expected,” “As Expected,” or “Worse than Expected.”
- Hospital Safety Score converts data from a variety of scales (percentage, percentile, # of incidents, degree of completion, etc.) into one scale, derives the Z-Score, and presents the result as “Above Average,” “Average,” or “Below Average.”
- CMS’s Hospital Compare presents all data in comparison to the state and national numbers as “Better than,” “No different than,” or “Worse than” the given number.
- Consumer Reports and US News and World Report use scores from 1 – 100 for overall scores and for specialty areas.
No hospital score is created alike. Each organization designs a methodology that they believe best suits the consumer within their limits of funding, time, and mission focus. Some organizations are influenced by commercial considerations such as hospitals who pay to use their ranking in advertisements, to be included in their database or accredited by their institution, or consumers who pay to access the reports.
Our advice is to not only look at scores, but to always check out the methodologies while considering those scores to make sure you know exactly what they are communicating. Here are some direct links to get you started!
CMS Hospital Compare (main data page)
The Joint Commission Quality Report
US News and World Report Best Hospitals
Consumer Reports Health Safety Score
