Training Decay Selection for Medical Product Usability Validation: A Roadmap for Empirical Training Decay Curve Calculation

In this blog post, we will provide an update on the UserWise and San Jose State University joint research grant sponsored by the Food and Drug Administration (FDA) about training decay of medical devices. This content was presented as a poster at the 2019 International Symposium on Human Factors and Ergonomics in Health Care held in Chicago, March 2019.

Background

The Food and Drug Administration (FDA) requires that medical devices undergo usability validation testing and include a “comparable” training decay as is experienced by users during real-life use of the product. Training decay is defined as the elapsed time between the end of the user training and the start of when the user actually starts using a medical product for the first time. Over time, the user’s retention of the material covered during training decays. Training decay implementation in usability validation studies varies depending on the type of medical product being tested, and can vary in length between 1 hour and up to 7 days. The longer training decays present challenges for industry that include:

  • Participant management if the user population is unique or requires teams

  • Large time involvement may cause participant drop out

  • 10-30% increased costs according to an analysis previously conducted by UserWise

Training decay was a topic of interest at the symposium with two posters and one presentation on various aspects of training decay.

Overall Research Goals

The main research goals of this study are to:

  • Identify generalizable training decay curves for task types relevant to medical product use.

  • Develop a methodology for training decay selection among medical product sponsors and manufacturers.

Below are example training decay curves we expect to see for different task types (i.e., physical and cognitive) involved in medical product use. The left graph would represent a steeper training decay curve with a higher loss of performance over time in comparison to the right graph. The right graph shows a shallower curve with minimal loss of performance after 1 hour. Medical products with tasks represented in the left graph might require a training decay of 7 days in the usability validation study, where as medical products with tasks represented in the right graph might only require a training decay of 1 hour in the usability validation study.

 
Training Curves UserWise Training Decay Research Medical Devices.PNG
 

Overall Approach

The intended medical product users in our research study will be lay users in an at-home setting. Training will be standardized for a group of participants with a structured review of the product’s Instruction for Use (IFU) and hands-on experience or practice. Then participants will be subject to a specified training decay period of non-practice between 1 hour and 7 days. Training decay periods were selected after feedback from the FDA and industry consortium. After the training decay period, participants will be asked to perform simulated use tasks with the selected medical product in a representative home use environment. Currently, 3 medical products are potential candidates for this study.

Method

To calculate effect sizes and estimate the sample size of retention intervals for the proposed study, a systematic review of skill retention and skill decay literature was conducted for retention intervals between 1 day to 7 days. Coding of task characteristics, retention intervals, and sample size were performed.

Results

In result, 15 articles provided 48 data points and a minimum average effect size of 0.316. To guarantee a statistical power of 0.8 or larger with significance level of 0.05, 4 groups with 29 participants each (at least 116 participants total) can be tested as shown in Table 1.  To have 2 medical devices with separate participant groups, the future experimental design would be 2x3 groups with 24 participants each and a larger total sample size of 144 (Table 2).

Table 1. Example Study Design with 1 Medical Device

 
Table 1. Example Study Design with 1 Medical Device
 

Table 2. Example Study Design with 2 Medical Devices

 
Table 2 Example study design with 2 medical devices for UserWise Training Decay Research.PNG
 

Future Research

Experimental data on taxonomies or categories of tasks that have distinctly different decays has not been found. Further research to understand existing experimental task taxonomies would improve future analysis.

Similarly, interference from multiple device training and testing is expected, but the exact impact is not known. Finally, the time required to perform a pre-training-test, training, post-training-test, and post-decay-period-test is unknown for devices being considered, but will be studied in a pilot study.

How You Can Get Involved?

If you are interested in getting involved with this study to provide input or donate medical devices, contact UserWise to discuss next steps for being invited to our next industry consortium meeting or visit our FDA Grant page.  

Sources

1. Clark, S. E. (2016). Training decay selection for usability validation.

2. Paper presented at the Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, , 5(1) 76-83.

3. MiniMed Paradigm 512/712 Insulin pump User Guide | Medtronic Diabetes. (n.d.). Retrieved March 21, 2019, from https://www.medtronicdiabetes.com/download-library/minimed-512-712

4. Mix2Vial® Reconstitution System and Needle-Free Transfer Device - West Pharma. (n.d.). Retrieved March 21, 2019, from https://www.westpharma.com/products/reconstitution-and-transfer-systems/mix2vial-and-needle-free-transfer-device

5. Workflow-Process-Service. (n.d.). Polymers for Better Injection

See full poster below:

Training Decay 2019 HFES Healthcare Symposium Poster UserWise SJSU.jpg