Highlights:

• Psychometric science provides evidence for your client’s improvement or deterioration.
• The Reliable Change Index (RCI) and Minimally Important Difference (MID) can help you determine if client change is meaningful.
• NovoPsych uses various data-driven methods to provide nuanced interpretations for change scores.

Trusting your intuition can be empowering, but like a tightrope walker needing a safety net, it thrives with the support of evidence. How strong is your trust in intuition when it comes to assessing progress or deterioration in your clients? A meta-analysis by Miller et al. (2015) suggests that clinical decisions regarding client improvements based solely on intuition are not as accurate as those based on psychometric assessments.

Intuition can be enhanced by the rigour and precision of statistical analysis, potentially picking up important trends. Relying solely on intuition can overlook blind spots and hinder symptom reduction, especially for clients who initially do not show improvement (Lambert et al., 2003). Measurement-based care in psychotherapy, which involves the regular administration of standardised assessments, complements clinical intuition and is increasingly recognised as essential (Meyer et al., 2001).  At NovoPsych, we strive to accurately assess changes—improvement or deterioration— in a client’s mental health by integrating various empirical methods and providing clinicians with a solid sense of clinically meaningful change.

Measuring Change- Not So Simple

A rudimentary and sometimes misleading method to determine change scores is to review the severity descriptors to quantify whether an individual has changed severity category. We know from experience that many clinicians use this method. For example, the Depression Anxiety and Stress Scale (DASS-21) manual delineates distinct categories such as “mild,” “moderate,” “severe,” and “extremely severe” (see Figure 1) (Lovibond & Lovibond, 1995). Some clinicians then simply determine if a client has improved (or worsened) by assessing if the client has moved to a different severity classification category. For example, we could determine that a client has improved if they were ‘Severe’ when they first started seeing us but they are now classified as ‘Moderate’.

Figure 1. DASS-21 severity classification table showing score ranges for total and subscale measures across five categories.

However, a crucial issue with these severity descriptors is that some of the defined ranges are quite broad, which hinders the systematic tracking of improvement and deterioration within these categories. Conversely, a client could demonstrate minimal change and improve by only 1 point but change from one severity classification to another. These are crucial limitations. Therefore, a method for determining meaningful score changes instead of focussing upon severity descriptor changes is required.

Reliable Change Index (RCI)

One such method is the Reliable Change Index (RCI), which assesses whether changes in an individual’s psychological scale scores before and after treatment are statistically significant, adjusting for natural fluctuations, to determine genuine improvement or deterioration (Jacobson et al., 1984). RCI is a rigorous and empirical standard to assess change, and is often described as the gold standard.

NovoPsych exemplified the effective use of the RCI by analysing over 90,000 DASS-21 assessments to determine the level of change needed to be considered “Reliable Change” and included this information in our reports to enhance the interpretation for clinicians. We calculated RCIs using the Jacobson-Truax method and internal reliabilities (Jacobson & Truax, 1991) in order to categorise change with meaningful descriptors, such as significant improvements (i.e., a reduction in over DASS-21 score of 7 or more points), aiding clinicians in evaluating treatment effectiveness. These cut-offs for improvement (or deterioration), are based upon differences between initial and most recent scores and provide clear benchmarks for assessing psychological well-being.

NovoPsych’s outcome data analysis package called Insights, uses the RCI to determine the proportion of clients that have changed. In this way, a practice can determine in a statistically rigorous way the change in a group of client symptoms pre and post treatment.

The figure below shows that for a typical practice, 54.8% of clients have significant improvement in DASS-21 scores, based on the RCI (see Figure 2).

Figure 2. Client Outcomes Using Reliable Change Index (RCI)

This highlights that a large proportion of people (36.3%) are considered to have no significant change, as determined by the RCI threshold of 7 points. Significant improvement was achieved by 54.8% of clients at this practice. On the other hand, if using a simplistic valuation of change being determined by just one point change on the DASS-21, the same data shows 75.1% improvement (Figure 3). This highlights how different methods of calculating change at a group level can impact overall interpretation.

Figure 3. Client Outcomes Using One-Point Change Criterion

By adding this to NovoPsych’s interpretations, we save clinicians from needing to learn about the actual maths behind RCI (Formula 1):

Formula 1. The Jacobson-Truax method of clinical significance classification (Jacobson & Truax, 1991). Note. x_pre= pre-test score of clients; x_post = post-test score of clients; s₁ = standard deviation of the pre-treatment group; r_xx = internal reliability of the scale (as per Bauer et al., 2004).

Example 1. Consider the case of a hypothetical client whose score on the DASS-21 has decreased from 59 to 40 (see Figure 4). Despite both the initial and new total scores falling within the “Extremely Severe” range, there was clinically significant improvement of 19 points. Although this improvement is clearly evident in the NovoPsych plot of scores over time (Figure 4), if a clinician was only looking at the severity descriptor, it would appear that the client has not improved. In addition to the plot, NovoPsych will highlight in the Interpretive Text (see Figure 5) that this client has improved significantly on their overall distress.

Figure 4. Line graph visualising DASS-21 score decrease from 59 to 40 over time.

By utilising this calculation approach, grounded in robust statistical models, we offer clinicians more detailed insights into changes in mental health.

Figure 5. NovoPsych’s Interpretive Text highlighting significant improvement despite unchanged severity category.

Example 2. Despite minimal score changes, the graph in Figure 6 demonstrates that severity descriptors can shift due to predefined cut-off thresholds. NovoPsych’s advanced statistical methods ensure that slight score variations, despite a change in descriptor, are accurately recognised in the Interpretive text as indicating no significant change (see Figure 7).

Figure 6. Graph demonstrating how small score changes can shift severity descriptors due to cut-off thresholds.

Figure 7. NovoPsych’s interpretation accurately identifying no significant change despite descriptor shift.

Effective evaluation of treatment outcomes hinges on clear and standardised classifications that guide clinicians in assessing client progress. NovoPsych integrates these classifications seamlessly into tools like the DASS-21, leveraging the RCI to establish thresholds for meaningful changes.

The Minimally Important Difference (MID)

An RCI is not always available either because research has not determined it or because there might not be enough NovoPsych data to calculate it (if we are adding a new assessment). Therefore, an alternative to the RCI could be beneficial for clinicians.

While several alternatives to RCI exist for outcome monitoring, the Minimally Important Difference (MID) stands out as a simple yet empirically sound method (Turner et al., 2010). MID represents the smallest change in a treatment outcome that clients perceive as beneficial or clinically meaningful. According to a review by Norman et al. (2003), which examined 38 studies across various conditions and instruments, changes approaching half a standard deviation (about 0.5 SD) consistently signalled meaningful change. Their research underscores the reliability and relevance of MID in clinical settings, advocating for its widespread adoption.

Using the MID to track mental health outcomes offers several advantages. Firstly, MID establishes a definitive threshold for meaningful change using the mean and variance of a representative sample, eliminating the need for research to longitudinally track outcomes to produce an RCI. Secondly, it enhances sensitivity, allowing detection of subtle yet important changes in client status (Toussaint et al., 2020). Finally, MID promotes a client-centred approach by aligning treatment goals with changes that are perceptible and meaningful to clients, improving engagement and satisfaction (Jayadevappa et al., 2017).

Example 3. NovoPsych utilises the MID to assess score changes in the Oldenburg Burnout Inventory. This approach ensures that even when clients remain within the same severity category (e.g., “High Burnout,” as shown in Figure 8), the interpretation highlights meaningful shifts in their scores (see Figure 9). It provides clinicians with a precise assessment of client progress that extends beyond conventional severity descriptors.

Figure 8. Oldenburg Burnout Inventory scores chart, showing change within “High Burnout” category

Figure 9. NovoPsych’s interpretation of Burnout scores, highlighting meaningful shifts using the MID approach.

Psychometric science, exemplified by methods such as the Reliable Change Index (RCI) and Minimally Important Difference (MID), plays an important role in interpreting client change in mental health treatment. The RCI and MID are NovoPsych preferred methods for determining change thresholds and are used on many assessments designed for outcome monitoring. The RCI is a more stringent and conservative metric for change compared with the MID, whereby the RCI will show fewer clients have changed compared to the MID. For example, NovoPsych data shows that based on the DASS-21 RCI, 49% of clients in therapy have significant improvement on Total Distress, but using the MID on the same data shows 56% of clients improved. Put another way, the RCI requires a 7-point difference to be classified as meaningful improvement, whereas the MID requires only a 5-point difference.

NovoPsych builds these rigorous statistical methods into our assessment interpretation so clinicians can combine it with their knowledge of their patient’s treatment journey to help paint a picture of recovery based on statistics and clinician intuition. NovoPsych supports clinicians in making informed decisions that enhance the effectiveness and personalisation of client care. This approach strengthens assessment reliability and emphasises measurable and meaningful changes in mental health outcomes.

References

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. https://doi.org/10.1037/0022-006X.59.1.12

Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15(4), 336–352. https://doi.org/10.1016/S0005-7894(84)80002-7

Jayadevappa, R., Cook, R., & Chhatre, S. (2017). Minimal important difference to infer changes in health-related quality of life—a systematic review. Journal of Clinical Epidemiology, 89, 188–198. https://doi.org/10.1016/j.jclinepi.2017.06.009

Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology (New York, N.Y.), 10(3), 288–301. https://doi.org/10.1093/clipsy.bpg025

Lovibond, S.H. & Lovibond, P.F. (1995). Manual for the Depression Anxiety Stress Scales (2nd ed.). Sydney: Psychology Foundation (Available from The Psychology Foundation, Room 1005 Mathews Building, University of New South Wales, NSW 2052, Australia.

Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues. The American Psychologist, 56(2), 128–165. https://doi.org/10.1037/0003-066X.56.2.128

Miller, D. J., Spengler, E. S., & Spengler, P. M. (2015). A meta-analysis of confidence and judgment accuracy in clinical decision making. Journal of Counseling Psychology, 62(4), 553–567. https://doi.org/10.1037/cou0000105

Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality of life: The remarkable universality of half a standard deviation. Medical Care, 41(5), 582–592. https://doi.org/10.1097/01.MLR.0000062554.74615.4C

Toussaint, A., Hüsing, P., Gumz, A., Wingenfeld, K., Härter, M., Schramm, E., & Löwe, B. (2020). Sensitivity to change and minimal clinically important difference of the 7-item Generalised Anxiety Disorder Questionnaire (GAD-7). Journal of Affective Disorders, 265, 395–401. https://doi.org/10.1016/j.jad.2020.01.032

Turner, D., Schünemann, H. J., Griffith, L. E., Beaton, D. E., Griffiths, A. M., Critch, J. N., & Guyatt, G. H. (2010). The minimal detectable change cannot reliably replace the minimal important difference. Journal of Clinical Epidemiology, 63(1), 28–36. https://doi.org/10.1016/j.jclinepi.2009.01.024