Month: February 2020

Overview of the Rorschach

Posted on by Shawn Wygant

Irving Weiner, who has written extensively regarding the Rorschach, contends that it endures unwarranted criticism.  Weiner insists:  

“Current literature reflects a persistent inclination in some quarters to denigrate the Rorschach Inkblot Method as an invalid and useless instrument for assessing personality functioning.  Although perhaps warranted to some extent in years past, such harsh criticism of the Rorschach runs counter to abundant contemporary data demonstrating its psychometric soundness and practical utility.”i 

 

There are some data demonstrating the Rorschach’s psychometric soundness.  A 1999 study, for example, found that the validity coefficients of the Rorschach and the MMPI2 are approximately equal. ii  These data, however, do not necessarily support Weiner’s claims of “practical utility” for the Rorschach.  The 1999 study reviewed Rorschach data obtained from studies published in peerreviewed journals.  As we at PsychLaw.net pointed out in prior sections, without satisfactory levels of interrater scoring consistency, peerreviewed journals will not publish Rorschach research.   

Research teams working diligently to achieve satisfactory levels of Rorschach scoring consistency can succeed in doing so.  Extrapolating from these data to how practicing psychologists score and interpret the Rorschach is illadvised.  As we at PsychLaw.net also pointed out in prior sections of this Blog, it is necessary to discriminate between the “research” utility of the Rorschach, and its “field” utility.  Compared to research psychologists, practicing psychologists typically do not have sufficient time available to hone their Rorschach scoring skills. We at PsychLaw.net feel that consequently, the “field utility” of the Rorschach most certainly does not even approximate its “research utility.”  Claiming otherwise is tantamount to the first author contending that because we use the same make of clubs, his golfing prowess equals that of Tiger Woods! 

Psychologists using the Rorschach may claim that their specialized training with this technique allows them to administer, score, and interpret it accurately.  Supporting these claims would necessitate training that uses a criterion level for certifying Rorschach competence.  A criterion level would define how sample Rorschach responses should be scored and interpreted.  After reaching a predetermined level of scoring and interpretive accuracy, a psychologist could claim Rorschach competence.  To the best of our knowledge, there are no Rorschach training programs designed to bring trainees to a defined level of accuracy.  Claims of accuracy for scoring and interpreting therefore amount to totally unsubstantiated claims. 

CrossExamining the Clinical Use of the Rorschach 

  1. You claim that there are numerous studies published inpeerreviewed journals supporting your use of the Rorschach  Correct? 
  2. Without satisfactory levels of interrater scoring consistency,peerreviewed journals will not publish Rorschach research  Correct? 
  3. And we know that research teams working diligently to obtain satisfactory levels of Rorschach scoring consistency can succeed in doingso  Correct? 
  4. Compared to the research psychologists who publish Rorschach data, practicing psychologists do not have the time available to hone their Rorschach scoringskills  Correct? 

[THE FOLLOWING QUESTIONS ARE TAKEN FROM PRIOR SECTIONS] 

  1. The psychologists participating in Rorschach research studies underwent specialized training to reach acceptable levels of interrater scoringconsistency  Correct? 
  2. What specialized training have you undergone to assist you in scoring the Rorschach in an acceptable manner? 

[The vast majority of psychologists have not undertaken this kind of training.   

Prior to crossexamination, check the psychologist’s c.v. for postdoctoral  

training in the Rorschach] 

  1. And the“field reliability” of the Rorschach refers to how practicing psychologists such as yourself use it  Correct? 
  2. You cannot cite any data published in apeerreviewed journal reporting the field reliability for the TRACS  Correct?   [or whatever other system the professional used]. 
  3. In other words, the extent to which practicing psychologists carefully comply with the TRACS scoring procedures [or any other scoring procedure] remainsunknown  Correct? 
  4. If the extent to which practicing psychologists carefully comply with the TRACS scoring procedure remains unknown, we do not know how carefully you complied with that scoringprocedure  Correct? 
  5. And if you resorted to your own idiosyncratic scoring method, your interpretations of the Rorschach in this case could differ from anotherprofessional  Correct? 
  6. Because assumptions about the interpretive accuracy of the Rorschach assume that it was scored properly and interpreted in a uniformmanner  Correct? 
  7. Other than accepting your claims, we have no way of knowing if you scored the Rorschach accurately in thiscase  Correct? 
  8. And if you scored the Rorschach inaccurately, you could misinform and mislead thisproceeding  Correct? 
Posted on by Shawn Wygant

The “Improved” DSMIV? 

There is no available evidence indicating that the procedures of DSMIV have reduced the subjective biases associated with the diagnostic work of mental health professionals.  Diagnoses too often reflect the ethnic and social class prejudices of diagnosticians and the social stereotypes they associate with a particular disorder.i At PsychLaw.net we take for example, black patients are diagnosed as schizophrenic far more frequently than white patients.ii  This outcome corresponds more to the unfamiliarity of white diagnosticians with black culture.  Diagnostic labels also involve simple considerations of whether diagnosticians experience an affinity for their patients.  Patients who direct negative attitudes toward diagnosticians are labeled more seriously disturbed than patients who express deferential attitudes.iii  

The concept of “inter-rater reliability” was described in previous posts. The critically important question of interrater reliability asks: If two or more mental health professionals evaluate the same client, to what extent will they agree in their diagnostic conclusions?  Low levels of interrater reliability related to any classification procedure indicates that the procedure frequently leads to mistaken findings.   Surprising as it may seem, there are no interrater reliability data to be found in DSMIV.  DSMIII contained interrater reliability data for its various diagnostic categories, but the supposedly improved DSMIV neglected to report this information. The unavailability of interrater reliability data for the many diagnostic classifications of DSMIV profoundly undermines its evidentiary value. 

The 1999 Standards for Educational and Psychological Testingiv published by the American Psychological Association, clearly define the necessity of reporting interrater reliability data for any procedure relying on clinical opinion.  In particular, Standard 2.10 of these Standards states: 

“When subjective judgment enters into test scoring, evidence should be provided on both interrater consistency in scoring and withinexaminee consistency over repeated measurements.”v 

 

At PsychLaw.net we note that the use of DSMIV relies almost entirely on judgmental processes.  Neglecting to report interrater reliability data for its diagnostic categories therefore amounts to an egregious error.  Though DSMIV is not a psychological test per se, Standard 2.10 is nonetheless applicable to it.  The 1999 Standards specifically state:  

“The applicability of the Standards to an evaluation device or method is not altered by the label applied to it (e.g., test, assessment, scale, inventory).”vi 

 

DSMIV is an assessment technique designed explicitly for the assessment and classification of psycho-pathology. 

Despite the overwhelming importance of interrater reliability to diagnostic classification, most mental health professionals neglect to think about this issue carefully. At PsychLaw.net we consider, for example, how a doctorallevel psychologist responded to the following crossexamination. 

Attny: Do you know the research concerning the reliability of diagnoses? 

Psych: I don’t know what research you’re referring to. 

Attny: I’m referring to the scientific literature concerning the accuracy of diagnoses. 

Psych: I don’t know what you’re referring to, but if you could tell me, I’d be interested. 

Attny: Do you know what interrater reliability is? 

Psych: Yes, I do. 

Attny: Are you familiar with the research on the interrater reliability of DSMIV diagnoses? 

Psych: I am not. 

 

From the time of its first edition published in 1952, through its most recent 1994 fourth edition, each DSM has progressively increased the number of diagnosable disorders.vii The 1952 edition specified 60 categories of mental disorder.  The second edition, published in 1968, contained 145 diagnoses.  DSMIII, published in 1980, expanded to 230 disorders.  DSMIIIR, published in 1980, listed more than 300 diagnostic categories.  Most recently, the 1994 DSMIV has grown to more than 400 disorders.  Rather than lead to improved levels of interrater reliability, this increasing number of diagnosable disorders can only reduce the extent of diagnostic agreement between clinicians.  Increasing the number of category choices inevitably reduces the level of classification agreement between two or more raters. 

In addition to its many diagnostic classifications, DSMIV also uses a “Global Assessment of Functioning Scale” (GAF) to assess how effectively clients function in their daytoday lives.  Curiously, research data demonstrate no significant relationship between patient ratings of their own psychological symptoms, and clinical assessments of those patients using the GAF scale.viii  In other words, this 1995 study by Piersma and Boes, published in the Journal of Clinical Psychology,  demonstrated that GAF ratings are unrelated to how patients assess themselves. We at PsychLaw.net feel that quite obviously, this research sorely undermines the evidentiary value of GAF as a method for assessing the adequacy of a patient’s general functioning. 

Psychologists and laypeople alike typically assume that diagnosis is a necessary prerequisite to treatment.  Nevertheless, the relevant data fail to support this assumption.  In particular, two critics of the various DSM editions have sharply criticized its role in graduate training: 

“As clinicians, students may be given misinformation in graduate school.  They will be taught DSMIIIR or the fourth edition of the DSM as if it were science and not politics; but such diagnoses as are found in these texts predict only 5% to 10% of the outcome of psychotherapy.”ix (p. 216). 

 

CrossExamining The “Improved” DSMIV 

  1. Diagnoses too often reflect the ethnic and social class prejudices ofdiagnosticians  Correct? 
  2. BlackAmericansare diagnosed as schizophrenic more often than WhiteAmericans  Correct? 

3.But this outcome corresponds more to the unfamiliarity of whiteAmerican diagnosticians with blackAmerican culture  Correct? 

  1. Diagnostic labels can also involve considerations of whether diagnosticians experience an affinity for theirpatients  Correct? 
  2. The issue of interrater reliability is also critically important to any procedure for diagnosticclassification  Correct? 
  3. The critically important issue of interrater reliability asks: 

– [read] – 

“If two or more psychologists evaluate the same client, to what extent will they agree in their diagnostic conclusions”  Correct? 

  1. Low levels of interrater reliability related to any procedure for diagnostic classification indicates that the procedure frequently leads to mistakenfindings  Correct? 
  2. DSMIII, published in 1980, contained interrater reliability data for itsdiagnostic categories  Correct? 
  3. But the supposedly improved DSMIV neglects to report any interrater reliability for its diagnosticcategories  Correct? 
  4. Therefore, we do not know how often DSMIV leads to mistakenconclusions  Correct? 
  5. In other words, we do not know what the error rate is for any DSMIV diagnosticcategory  Correct? 
  6. And mental health professionals cannot claim that DSMIV is generallyaccepted  Correct? 
  7. Claiming that DSMIV is generally accepted necessitates that mental health professionals agree in their diagnostic conclusions when usingit  Correct? 
  8. But, we do not have interrater reliability data for the diagnostic categories ofDSMIV  Correct? 
  9. Mental health professionals cannot claim that DSMIV is generally accepted without knowing the extent to which theyagree  or disagree  in their diagnostic conclusions premised on DSMIV  Correct? 
  10. The 1999Standards for Educational and Psychological Testing were published as a cooperative undertaking between the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education  Correct? 
  11. The 1999Standards for Educational and Psychological Testing define standards for a broad range of assessment procedures  Correct? 
  12. And the 1999Standards for Educational and Psychological Testing are generally recognized and accepted as defining appropriate standards of practice regarding a broad range of assessment procedures  Correct? 
  13. The 1999Standards for Educational and Psychological Testing indicate they can be applied to the following: 

– [ read ] – 

    “The applicability of the Standards to an evaluation device or method is not altered by the label applied to it (e.g., test, assessment, scale, inventory).” 

Now my question:  DSMIV is explicitly designed as Aa evaluation device” for the assessment and classification of psychopathology  Correct? 

20.Please consider what Standard 2.10 of the Standards for Educational and Psychological Testing state regarding interrater reliability: 

– [ read ] – 

“When subjective judgment enters into test scoring, evidence should be provided on both interrater consistency in scoring and withinexaminee consistency over repeated measurements.” 

Now my question:  The Standards for Educational and Psychological Testing therefore clearly define the necessity for interrater reliability data  Correct? 

21.DSMIV’s failure to report interrater reliability data consequently amounts to neglect of Standard 2.10  Correct? 

22.DSMIV also uses a “Global Assessment of Functioning Scale” (GAF)  Correct? 

23.And the GAF scale relies in part on what the patient tells the mental health professional  Correct? 

[USE THE 22 FOUNDATIONAL QUESTIONS FROM PREVIOUS POSTS 

IF YOU HAVE NOT YET DONE SO] 

24.The Journal of Clinical Psychology is a generally recognized and accepted, peerreviewed journal in your field  Correct? 

  1. And a 1995 article byPiersma and Boes published in the Journal of Clinical Psychology  titled “Agreement Between Patient SelfReport and Clinician Rating: Concurrence Between the BSI and GAF Among Psychiatric Patient”   might be related to your opinions in this case  Correct? 
  2. And in their 1995 study,Piersma and Boes found no significant relationship between patient ratings of their own psychological symptoms and clinical assessments of those patients using the GAF  Correct? 
  3. Have you published any data in apeerreviewed journal necessitating that we reconsider Piersma and Boes’ 1995 study? 
  4. Can you cite any data published in apeerreviewed journal necessitating that we reconsider Piersma and Boes’ 1995 study? 

29.Without any data necessitating reconsideration of Piersma and Boes’ 1995 study, their results  published in a peerreviewed journal  should be generally recognized and accepted by your profession  Correct? 

  1. In other words, your profession generally recognizes and accepts that there is no significant relationship between patient ratings of their own psychological symptoms and clinical assessments of those patients using theGAF  Correct? 

Cross Examination of Projective Techniques

Posted on by Shawn Wygant

Overview of Projective Techniques

In previous posts, we discussed psychological tests that obtain objective data.  Because of their objectivity, these tests can assist the legal system when used and interpreted properly.  In this post, we review projective tests.  Some of these instruments do not even qualify as standardized tests, and all of them lack the necessary validity and reliability for admissibility in court.  Because the proponents of these projective instruments tend toward pontification, we at PsychLaw.net feel that the cross examiner must be well prepared.  A quick and cogent cross examination is necessary to aid the court in understanding the shortcomings of these procedures.

Subjectivity of Projective Techniques

In comparison to objective tests, projective tests are significantly more subjective.  The procedures for collecting test data are not as well standardized for projective tests.  In other words, there are substantial variations in how psychologists administer the same projective test.  Responses to projective tests also vary enormously between people taking those tests.  Rather than rely on a true‑false or multiple choice options, projective tests obtain more open‑ended responses.  For example, people taking projective tests are asked: (1) “look at the designs on this card and tell me what they look like, or remind you of”; or (2) “look at this picture and tell me a story about the people you see.”

The scoring procedures for projective tests are also far less than objective.  As a result, two or more psychologists scoring the same projective test data, obtained from the same person, can report very different scores.  At PsychLaw.net we note that given these variations in scoring, two or more psychologists can arrive at exceedingly different interpretations of the same test data from the same person.  Despite these administrative and scoring problems, practitioners continue to use projective tests with considerable frequency.[1]

The subjectivity of the administrative and scoring procedures for projective techniques effectively undermine their evidentiary value.[2]  As a result, we will present a cross‑examination procedure directed at projective techniques in general.  The remainder of this section will then address four specific projective techniques: the Rorschach technique, projective drawings (especially the Draw‑a‑Person technique), the Thematic Apperception Test (TAT), and the projective use of Bender‑Gestalt.

As previously pointed out in other posts, psychological tests and other related procedures are designed for use by psychologists.  Non-psychologists have neither the education, nor the training in psychological assessment, to use these standardized procedures.  Nevertheless, various mental health professionals do use them. As a result, we recommend using the 1992 Ethical Standards of the American Psychological Association, and the 1999 Standards for Educational and Psychological Testing, as guidelines for cross examination. At PsychLaw.net we teach that a non-psychologist can also be asked: “Though you are not a psychologist, wouldn’t it be preferable for you to comply with the relevant ethical and practice standards related to psychological testing?”

Cross‑Examining the Subjective Nature of Projective Techniques

  1. Testing approaches for personality assessment can be divided into broad categories of “objective” and “projective” instruments ‑‑ Correct?
  2. The MMPI‑2 would be an example of an “objective” instrument ‑‑ Correct?
  3. And compared to objective instruments such as the MMPI‑2, projective techniques are more subjective in their scoring ‑‑ Correct?
  4. Hold up your hand with thumb and first finger separated by approximately six inches, and ask:

“Dr. X, please show me your estimate of six inches.”

  1. But I say this [wave your hand with thumb and first finger separated by approximately six inches] is six inches, but you say that your [point] estimate is six inches. Therefore, we have your subjective estimate, and we have my subjective estimate ‑‑ Correct?
  2. Which estimate is more accurate is a matter of conjecture and speculation ‑‑ Correct?
  3. That’s what subjective often means ‑‑ resorting to conjecture and speculation ‑‑ Correct?
  4. And there are inevitable differences in the conjecture and speculation between two or more mental health professionals ‑‑ Correct?
  5. And there are inevitable differences between the conjecture and speculation of two or more professionals because of their relying on subjective impressions ‑‑ Correct?
  6. And because of the inevitable differences in conjecture and speculation between two or more professionals, their subjective interpretations of projective test data may not agree ‑‑ Correct?
  7. Mental health professional A can score and interpret the projective test data obtained from someone in one way, but professional B can score and intepret those same data from the same person quite differently ‑‑ Correct?
  8. And when we have those variations between two or more professionals, we cannot know for sure who is accurate ‑‑ Correct?
  9. And when we have those variations in scoring and interpretation between two or more professionals, they may all be mistaken ‑‑ Correct?
  10. The variations in scoring and interpreting projective techniques could therefore misinform and mislead this proceeding ‑‑ Correct?

__________________________________________________________________________________________________________________________

[1].       The Rorschach has a long been blocked from use and discounted in forensic settings.  See for example: People v Jenko, 410 Ill. 478, 481, 102 NE 2d 783 (1952) [Rorschach testimony barred];  State of Utah v Phillip Rimmasch, 775 P 2d 388 (Utah Sup., 1989) [Rorschach testimony is violative of Rule 702]; McCarty v McCarty, (WL 259363 Neb App 1993) [Trial court properly disregarded Rorschach testimony].

[2].       Faigman, D.L. (1995). The evidentiary status of social science under Daubert: Is it “scientific”, “technical”, or “other” knowledge? Psychology, Public Policy, and Law, 1, 960-979. [ For the most part, psychologists who maintain rigorous scientific standards such as those described in the current edition of the APA’s Standards for Educational and Psychological Testing (1985) will find the effect of the Daubert standard to be more liberal than the old Frye standard.  But psychologists wyho cannot justify the validity & reliability of their measures, will find the Daubert standard far more severe].  See, also: Faigman, D.L. (1992). Struggling to stop the flood of unreliable expert testimony. 76 Minnesota Law Review 877-889; Faigman, D.L., Porter, E. & Saks, M. (1994). Check your crystal ball at the courthouse door please: Exploring the past, understanding the present and worrying about the future of scientific evidence. 15 Cardozo Law Review 1799-1835.