6-Measuring Success

MODULE 10 | Section 6 of 11

Measuring Success

IMPLEMENTING YOUR PLANS

After defining your problem, setting a SMART goal for improvement, and structuring a framework to meet that goal, you are ready to move forward with implementing, measuring, and evaluating the impact of your intervention. A common approach for structuring this portion of a quality or value improvement project in health care is the PDSA (Plan-Do-Study-Act) cycle we introduced in Section 3.

M10s6p2

But before we delve further into PDSA and tools used with it, let’s briefly revisit the concept of different types of measures. For most projects, you will have at least one of each of these measure types. Defining your measures early before starting an intervention is important as it helps you focus on what you are trying to impact.

DESIGNING WITH SUCCESS IN MIND: DEFINING MEASURES

Recall from Module 2 that there are four categories of measures: Structure, Process, Outcome, and Balancing. Quality improvement projects should generally evaluate a mix of measures across each of these categories.

DESIGNING WITH SUCCESS IN MIND: DEFINING MEASURES

M2s5icon1
PROCESS MEASURES

What is actually done in giving and receiving care. For a process measure to be valid, it must previously have been demonstrated to produce a better outcome.

For example, whether or not a patient is given an influenza vaccine during a hospitalization could be considered a process measure.
M2s5icon2
STRUCTURE MEASURES

The material, human, and organizational resources available in the settings in which care is delivered.

For example, the number of MRI machines, hospital beds, and available staff may be important structural aspects to consider and measure. If a project seeks to deliver influenza vaccines to all eligible patients, then the number of nursing full-time equivalents (FTE) required to deliver influenza vaccines could be a structural measure.
M2s5icon3
BALANCING MEASURES

Efforts to ensure changes do not result in other unintended consequences or effects.

For example, if efforts to discharge patients before noon actually resulted in increased length of stay because some patients were being kept until the next day to be discharged in the morning, this would be an important balancing measure that may not be captured if one is only measuring the percentage of patients discharged prior to noon.
M2s5icon4
OUTCOME MEASURES

The effects of care on the health status of patients and populations.

For example, if fewer people die after receiving influenza vaccines, then this mortality benefit would be an outcome measure.

FINDING AND USING DATA

Defining your measures will help you to understand what data you will need to track and evaluate your project, so it’s a good idea to do so early on in your project. Tracking data over time is key to driving improvement by allowing you to measure the impact of small changes.

 

There are many sources of data in health care. Some common sources you can use are detailed below. However, you will likely need to gather your own data as well. These can be obtained through reports from your electronic health record (EHR), chart review, or otherwise tracking and collecting on your own.

FINDING DATA IN YOUR HOSPITAL

FINANCIAL DATA

Hospitals keep close records on all services that are billed for in the hospital.

 

Direct costs: This is typically how much it costs your hospital to provide a particular service. It incorporates all the materials, labor, and expenses related to the production of a product.

Facility charges: This is what a hospital charges to a self-pay patient. This is the charge that would be listed on a hospital bill.

Facility payment: This is how much an insurer pays your hospital for a given service. Payment rates for Medicare are publically reported. One example is the website below which provides Medicare payments to physicians for certain services:https://www.cms.gov/apps/physician-fee-schedule/overview.aspx

Decision Support: Many hospitals have a “decision support” team that provides hospital departments with the tools, technology, and applications to access clinical, financial, and administrative information on a timely basis. A decision support analyst may conduct various types of specialized analyses, including cost-benefit analyses, financial analyses, and feasibility studies.

Service Line Director: Other hospitals employ “service line directors” whose job is to ensure the financial success and productivity of a specific set of services in the hospital. They serve as a bridge between physician needs and hospital administration. They often have access to current data on the use and profitability of services in their domain.

Administrator/Billing Department: If your hospital does not have a decision support department, it is probably best to approach an administrator who can connect you to a friendly analyst in the billing department.

QUALITY DATA

This is a loosely defined term to encompass publicly reported data like core measures and hospital-acquired complications as we all other frequently measured performance data.

Core measures: Hospital core measures are CMS recommended treatments that the scientific evidence shows produce the best results. All hospitals are required to collect and report on these measures. These include treatment of perinatal care, stroke, VTE, Heart Failure, ED care, etc. A complete list of core measures and the requirements are found onhttp://www.jointcommission.org/core_measure_sets.aspx

 

Hospital Acquired Conditions: In early 2010, Medicare implemented 11 categories of HACs for which they would no longer be paying hospitals. Hospitals had to develop a way to monitor, track, and report these conditions. Some states have additional penalties. For a list of conditions go to: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/HospitalAcqCond/Hospital-Acquired_Conditions.html

 

Readmissions: Similarly, as part of the Affordable Care Act, payments to hospitals are reduced for hospitals that have excess readmissions in the category of heart failure, pneumonia, acute myocardial infarction. Penalties for COPD and large joint replacement started in 2014. In addition to tracking these rates, many hospitals are now tracking 7-, 14-, and 30-day readmissions for a variety of other groups. For post-acute care facilities, there are no readmission penalties as of yet and there is a lot of variation in the way this data is collected and monitored.

 

Mortality: Hospitals keep close track of mortality cases that occur in the hospital. Many hospitals even have a mortality review committee, responsible for reviewing the quality of care, documentation, and coding of in-hospital deaths. Mortality can be tracked as a raw percentage as well as a ratio of observed/expected (O/E). The O/E Mortality Index can be helpful in determining patients who had unexpected deaths based on their presenting severity of illness.

 

Sepsis: Given that many large hospitals have sepsis initiatives, many track prevalence of sepsis patients in their organization, mortality rates, and compliance with national best practices such as sepsis recognition and timely administration of antibiotics.

Director of Quality: The structure in every organization is different, but most hospitals have a quality department that is responsible for providing both the government and the organization with data regarding core measures and HAC performance. This person can be a wonderful source for your hospital’s data.

Chief Quality Officer: A good link to the Quality Department will be the facility’s chief quality officer. This is often a physician champion working with the quality department to monitor data and drive improvement.

Publically Reported Hospital Data Websites: The following websites contain publicly reported data from multiple hospitals and can be a great way to compare your facility to others.

  • WhyNotTheBest.org: was created and maintained by The Commonwealth Fund, a private foundation working toward a high performance health system.
  • The Leapfrog Group: compares hospitals’ performance on the national standards of safety, quality, and efficiency.
  • Hospital Compare: was established to promote reporting on hospital quality of care.

Public Data for Post-Acute Facilities: Information regarding Skilled Nursing Facilities is obtained in cooperation with SNF Data Resources, an online source for SNF cost report data and Medicare survey findings.

Healthcare Collaboratives: Your hospital may belong to a healthcare collaborative to which a group of similar hospitals submit data. These can be a rich source of data on your hospital’s performance and comparisons to other hospitals. Some examples are University Health Consortium (UHC), NSQIP (National Surgical Quality Improvement Program), BEACON (Bay Area Patient Safety Collaborative). Ask about healthcare or patient safety collaboratives your hospital may belong to as a source of data.

PATIENT SATISFACTION DATA

Increasingly patient satisfaction is being publicly reported and tracked by hospitals.

Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS): HCAHPS is a standardized instrument designed for measuring patient perspectives on hospital care. All hospitals are required to participate in HCAHPS data collection and public reporting. More information can be found here: http://www.hcahpsonline.org

MGMA, Press Ganey, SullivanLuallin Group: Three of the largest survey distributors and patient satisfaction consultants. These consultants will distribute the HCAHPS survey as well as other mandatory outpatient surveys, custom surveys, and provide dashboards, reports, and analytics with which to analyze the reports. All have a website that facilities use to access their data and can create custom reports that may be helpful in driving improvement.

Source: Adapted, with permission, from Michelle Mourad, MD, UCSF.



There is power in collecting your own data for improvement! Not only do you gain more insight in the process you are trying to impact (think going to Gemba!) you can often collect the data with a quick turnaround. You may also be surprised to find you need less data than you think to test small changes.

“Data in health care are often not straightforward or easy to obtain for frontline clinicians, but they are essential for driving improvement. The key is to develop a data infrastructure that advances clinical inquiry by enabling the translation of clinical questions into datasets that can be used to track and measure improvement. The Data Core at Dell Med helps with this translation and works with clinicians to get them the data they need for quality improvement”

Anjum Khurshid, MD, PhD
Co-Chief, Data Integration, Department of Population Health
Assistant Professor, Department of Population Health

Leave a Comment