7- Evaluating Data

MODULE 10 | Section 7 of 11

Evaluating Data

THE POTENTIAL POWER OF SMALL SAMPLE SIZES

The key to utilizing small sample sizes for improvement is to set a goal ahead of your data collection. Let’s work through an example.

Step 1 - SCENARIO


You believe that there is a high level of satisfaction with wait times for getting blood drawn in your hospital lab clinic.  You’d like to measure satisfaction rates to confirm your belief .   You think that the patients in this clinic will be satisfied 85% of the time . You decide to start surveying the patients in this clinic beginning tomorrow.  You sample for one day with a validated scale and collect surveys from all patients visiting the clinic totalling 15 surveys .   You find that 6 of the 15 patients were satisfied with the wait time in the clinic .


Objective

Select on the Objective in the above scenario

Plan

Select on the Plan (or prediction) in the above scenario

Do

Select on the Do (or action) in the above scenario

Study

Select on the Study (or result) in the above scenario

ADDITIONAL RESOURCES

Value of Small Sample Sizes in Rapid-Cycle Quality Improvement Projects

Etchells E, Ho M, Shojania KG. BMJ Qual Saf. 2016;25:202-206.

VISUALIZE AND EVALUATING IMPACT

So let’s say you’re utilizing The PDSA cycle to plan and implement your intervention as it is widely used in business and medicine for the continuous improvement of processes. The Plan, Do, Study, Act cycle is based on continuous improvement, where each cycle ends at the start of the next. The key is to use the PDSA cycle for small, iterative tests of change.

M10s6p2 (1)

While simple to describe, the PDSA cycle can be difficult to use in practice. Too often, improvement teams go through the motions of PDSA cycles without applying the core concepts of the methodology behind PDSA. Rarely is an improvement project linear and straightforward. Just as health care problems are complex, their potential solutions likely become complex too. If you do not hit blocks in the road, you probably aren’t on the right road at all! Truly applying the PDSA cycle will allow the improvement team to have increasing confidence that their change will result in measurable improvement.

 

Below is a checklist from a paper in BMJ Quality and Safety to help you determine if you are authentically applying PDSA in your project. If you are checking most of the boxes on the left, you should pause and reassess your approach.

M10s7p2

In order to help you evaluate, it can be extremely helpful to create data visualization so that you can strategically pinpoint implementation effects and verify that the process change is responsible for any outcomes. Two common methods of creating data visualization are via Run Charts and Statistical Process Control (SPC) Charts. For this module, we will mainly focus on SPC charts; however you are encouraged to learn about both and choose the best visualization methods for your needs.

ADDITIONAL RESOURCES

Read this article from the BMJ to learn more about implementing change with the PDSA

M10s7p3

HOW TO MEASURE SUCCESS

STATISTICAL PROCESS CONTROL CHARTS

Statistical process control charts are a powerful tool for tracking improvement over time. The SPC chart has more detail than a run chart. It includes horizontal lines to represent sigma lines, standard deviation from the mean value which is also plotted. The key difference between run charts and SPC charts is that SPC charts allow you to detect periods of special cause variations–statistically significantly different trends in the data that indicate a departure from the common variation around the historical mean. Run charts are simpler; while they will give you an idea about changes over time, they will not be able to show whether these changes are statistically significant or whether the process you are visualizing is in or out of control.

The animation discussed rules used to interpret SPC charts. Check out this brief document created by the NHS Scotland that discusses interpreting SPC charts using Nelson rules, which are a common ruleset. We’ll discuss these more in the SPC activity in Section 10.

“I have often seen run charts used in health care settings where “trends” are discussed as urgent needs for change when really this is just normal variation inherent in the process. SPC charts allow you to focus only on significant trends on your data–when your efforts are actually worth investing.”

Victoria Valencia, MPH
Associate Director of Value Improvement at Dell Medical School

Learn More

ARTICLE

This article analyzes surgeon competence through statistical process control (SPC) charts and provides an interesting look into their application in health care.
Mohammed MA. BMJ Quality and Safety.

ARTICLE

This review studies the authentic application of PDSA and provides a framework for measuring its success.

Taylor MJ, McNicholas C, Nicolay C, et al. BMJ Quality and Safety.
ARTICLE
There is a high demand for quality reporting and an increasing struggle to collect the data to satisfy that demand. Read about this more in this article.
Penso J. STAT.

REFERENCES

  1. 1- Implementing Value-Based Initiatives: A New Challenge for Clinicians and Healthcare Systems. In: Moriates C, Arora V, Shah N. eds. Understanding Value-Based Healthcare New York, NY: McGraw-Hill. Accessed August 28, 2018.

 

  1. 2-Etchells E, Ho M, Shojania KG. Value of small sample sizes in rapid-cycle quality improvement projectsBMJ Qual Saf. 2016;25:202-206.

Leave a Comment