A basic and simple assessment for bias in psychometric tests: The 4/5 Rule

Psychometric tests should have been validated by a test publisher prior to being published. However, the test user should also pay due diligence and ensure that the test cannot be accused of having unjustifiable adverse impact for one group of people over another.

There is a simple guideline known as the four/fifths (4/5) rule.

Here we seek to ascertain the percentage of one group that is selected or passes through to the next round of a selection process based on test scores, compared to another group.

Let’s say that we are comparing red people with green people so that we don’t unwittingly offend anybody! If we find that more than 4/5 of reds to greens are reaching the mark, we have little to worry about.

On the other hand, if we discover that 4/5 or less of reds to greens are making it, we have cause for concern. In fact, in some parts of the world, we would be required by law to take action.

The next question of course is: If more reds are making the grade than greens, does this translate into work performance? That is, are red people better on the job than greens?

If so, the adverse impact is said to be justifiable because our test is accurately predicting differences in job performance.

If green people do just as well in terms of job performance as red people however, the adverse impact is unjustifiable. In this case, we must take immediate action and work with the test publisher to assess why the test is not accurately predicting job performance for both groups.

To calculate using the 4/5 rule is simple.

We ask: What percentage of group one are passing compared to group two? The groups can be red or green people (unlikely!), people of different cultural backgrounds, males and females and so on.

E.g., let’s assume 80 red people and 50 green people make the grade:
So, take the lower number and divide it by the higher number: 50/80 = .625 – this is almost 63%. 4/5 is 80% and so our test would be said to be violating the 4/5 rule.

The next step as noted above would be to assess whether or not this difference translates into job performance. Then we can conclude whether our test is producing a justifiable or an unjustifiable adverse impact.

As always, the best advice is to start off with a thorough job analysis and to allow the choice of psychometric test to be informed by the job analysis.

Assuming the test has been chosen by a trained and competent professional and it is a reliable and valid assessment of the construct of interest, we are unlikely to encounter violations of the 4/5 rule. Even where violations may occur, if the test has been chosen based on this firm grounding, our process is highly defensible.

Personality and job performance
(above text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Job performance concerns the knowledge, skills and attitudes that are required to enable an individual to perform the activities listed in the job description as per the competency profile that a human resource or similar professional may have developed through job analysis. Performance appraisal is used in organisations worldwide as a means to ensure the (at least) adequate performance of employees. It is linked to the entire human resource cycle in that it informs the training and development agenda, is a factor in the remuneration calculation and is inextricably coupled with and derived from job analysis, which itself feeds into the recruitment and selection process. Ultimately, assuming both reliability and validity of performance appraisal systems (some have postulated that appraisal is inherently flawed: Derven, 1990; others have described it as a crucial aspect of organisational life: Lawrie, 1990), organisations require solid evidence that candidates for job vacancies will score highly on appraisal dimensions once incumbent in the job. That is, the incumbent will consistently demonstrate competent and high levels of performance as defined by the organisation within the appraisal system.

On the basis of a review of articles appearing in the Journal of Applied Psychology and in Personnel Psychology between 1952 and 1963, Guion and Gottier (1965) concluded that personality questionnaires were not useful in the prediction of job performance and that they should not be used in selection decisions unless their validity has been specifically and competently determined for the specific situation (p.160). The authors did note that too few studies were available in the literature to allow for a thorough review of the criterion-related validity of personality assessments. More recently, with the development of meta-analytical techniques and the “general, if not quite universal” (McCrae, 2004, p.4) acceptance of the FFM, research in the West has, in the main, given support to the limited utility of personality assessment, alongside other reliable and valid forms of assessment, as an aid to the selection and development of employees (Barrick, Mount & Judge, 2001; Tett, Steele, & Beauregard, 2003; Thoresen, Bradley, Bliese, & Thoresen, 2004; Waldman, Atwater & Davidson, 2004). The majority of researchers tend to report correlations and predictions based upon the FFM labels. Despite some criticisms of the FFM (e.g., Block, 1995a) this does provide a unifying ground in which theorists and practitioners may study, communicate and utilise personality as a tool in the workplace.

Van den Berg and Feij (2003) used Structural Equation Modelling to reveal links between personality as measured by four Dutch questionnaires assessing Neuroticism, Extraversion, Achievement Motivation and Experience Seeking and outcome measures such as satisfaction, intention to quit and job performance. In their discussion they interpreted their findings within the Big-Five taxonomy. Among a number of important findings, they discovered that Extraversion predicted work self-efficacy and job satisfaction whilst work stress mediated the relationship between Neuroticism and job satisfaction. They also found that feedback mediated the relationship between achievement motivation and job performance but there were no direct links between the personality scales and job performance. Potentially, had the researchers used a combination of the Big-5 factors they may have observed higher validities. It is also possible that, had the researchers used narrow-band (personality trait) predictors, more significant findings between personality and job performance may have emerged.

Personality has been found to account for unique variance in performance, after partialling out the effects of cognitive ability. McHenry, Hough, Toquam, Hansom, and Ashworth (1990) found that the competencies of personal discipline and physical fitness were better predicted by personality and temperament than by ability. In a study of 284 New Zealand police force recruits, Black (2000) discovered that the NEO-PI-R’s Conscientiousness factor added predictive ability above and beyond cognitive testing. Conscientiousness within the NEO-PI-R was also implicated in Blickle’s (1996) study of 231 university students with this factor being associated with “learning discipline”.

Barrick, Mount and Judge (2001) discuss various ‘phases’ in research assessing the predictive validity of personality instruments in performance. Their paper provides a strong argument for the lack of significant findings in this domain up to the mid-1980s and implicitly cautions researchers not to simply enter all independent variables into an analysis in an attempt to find correlations. They note that, in recent times, the findings have been more positive and that researchers appear to be heeding another of Barrick, Mount and Judge’s assertions – to use different levels of personality measurement (see Section 2.8.4). For example, Timmerman (2004) found significant correlations between NEO-PI-R Conscientiousness (r=.16), Agreeableness (r=.16) and supervisor’s performance ratings in call-centre staff in the USA. He then went on to examine correlations at the facet level and found that a number of Conscientiousness facets, but only one Agreeableness facet, were significantly correlated with performance. Salgado (1997) reported meta-analytic findings from 36 studies carried out in the European community. Conscientiousness and Emotional Stability were valid predictors across occupational groups, whereas Openness and Agreeableness were more likely to show as valid predictors of training success. This finding was confirmed by Barrick, Mount and Judge (2001). Barrick, Mount & Judge (2001) also noted that Extraversion was related to success in specific jobs such as sales or management, but was less related to performance for skilled workers. From the accumulation of the evidence, it does appear that Conscientiousness is the most consistent predictor of performance. This assertion has been supported by Matthews and Deary (1998) in their assessment of Barrick and Mount’s (1991) data (Van den Berg & Feij, 2003).

(above text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Newsflash:
Here’s the abstract and reference from a very recent study that linked the Big-5 to job performance in Thailand:

Personality Traits and Job Success: An investigation in a Thai sample

The p
urpose of this study was to examine the predictive power of each facet of the five-factor model of personality on job success in a Thai sample. The sample consisted of 2518 persons from seven occupations. The research found that for all occupational groups neuroticism was significantly negatively correlated with job success, while extraversion and conscientiousness were significantly positively correlated with job success. Moreover, conscientiousness was the only personality trait that consistently predicted job success of persons across occupations.

Chuchai Smithikrai (2007)
Personality Traits and Job Success: An investigation in a Thai sample
International Journal of Selection and Assessment 15 (1), 134–138.

International Journal of Selection and Assessment Volume 15, Issue 1, Page 72-81, Mar 2007.
The three reliability and two construct validity studies presented here represent the first psychometric research on the commercially successful and popular Hartman Color Code Personality Profile. The reliability studies found high reliability after 3 and…

Occupational tests may be classified in numerous different ways. The use of each term provides a language by which test administrators are able to understand what the test measures, how it measures, what scores are compared against and conditions under which the test is sat (e.g., timed or getting more difficult as test progresses). Different aspects of this form of communication are listed below.

The two major categories of psychometric test are:

TYPICAL PERFORMANCE TESTS
The candidate is asked about their preferences, behaviours, typical response styles and attitudes to a range of things. There are no right or wrong answers and these tests are not usually timed.

Tests may also be classified as:

Power Vs Speed Test
Items on power tests get more complex as the test progresses and usually, not everyone will be able to accurately answer all questions. Speed tests are timed and usually not everyone will finish the test.

Ipsative Vs Normative Test
Scores on one scale are compared against scores on another scale in Ipsative Tests, these tests use forced-choice type questions. In normative tests, candidates are given a number of response choices, scores are compared against an appropriate reference group.

Personality Test
Measures the personality characteristics of the individual. Rather more a questionnaire than a test.

Ability/Aptitude Test (specific & general)
Measures how well an individual is able to perform in specific (e.g., secretarial skills) or general areas (e.g., mental ability) and therefore how well they will likely perform on the job.

Attainment Test
A measure of what level an individual has reached in a specific area – e.g., post-course examination.

Interest Inventory
Used in career counselling to assess an individual’s interests/preferences.

Work Sample Test
A sample of work is given to a person and the person is assessed on their ability to complete the work as required.

With continued technological advancements and growing unemployment, the world of work is becoming an employer’s market. However, with a greater choice of available and interested potential employees, this does not mean that an employer is bound to select the most appropriate one for a given job. In fact, given such an increased selection bank, it is even more important than ever that reliable and valid tools that are able to discriminate between the most potentially-effective employee and the one who will not fit the job or who will be unlikely to stay or will upset the current team atmosphere etc. An employer, on the other hand also needs to remain attractive, both internally and externally to potential and current employees. One way of doing this is to have a well designed system of continual development that benefits both the employee and the organisation. For both of these scenarios, personality assessment is an objective tool that is able to contribute to the validity and successful outcome of the process.

Contrarian Views
(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Despite the support for the FFM noted in our other articles, not all psychologists and personality researchers have been as positive. For example, Block (1995a) provides a “Contrarian View” of the FFM. With the exception of Hogan’s (1983) contribution, the FFM is descriptive (reports relationships between traits) rather than theoretical (explaining those relationships) and this gave Block (1995a) reason for criticism. Much of Block’s critique is based upon the use of factor analysis as a theoretical decision-making model. Indeed, Horn (1967) provided cause for concern by demonstrating that apparently meaningful factors could be defined through using subjective rotational procedures with random, nonsensical data. However, in their reply, Costa and McCrae (1995, p.217) rebuff Block as having committed “sins of omission”, accusing him of neglecting to cite empirical evidence that was critical to the questions he raises. Costa and McCrae were especially critical of Block’s omission of a review of independent confirmations of the FFM. Goldberg and Saucier (1995) also submitted a response to Block. One of their concerns regarded the lack of analytical alternatives available to personality researchers if factor analysis did possess the purported limitations. Block’s (1995b) reply to these points was brief, however, one may conclude, given the comments presented by all parties above, that although factor analysis does have drawbacks and a number of decisions need to be made in a less objective manner than may be expected of a science, awareness of these issues will reduce the chances of subjectivity and error. Moreover, as alluded to by Costa and McCrae, if one were to accept all of Block’s arguments, potentially, there would be no adequate taxonomy of personality to use as a basis in personality research, clinical assessment and workplace performance prediction.

Block (1995a) has not been the only critic of the FFM. If a theory of personality and the assessment that operationalises that theory are to be useful, they need to apply universally, across cultures. Cross-cultural researchers who have not replicated the FFM in its entirety have been at least somewhat critical of the model and so too have indigenous psychologists who have started to argue (as anticipated by Costa & McCrae, 1992a) that although the FFM may be useful cross-culturally, there do exist indigenous personality factors that this model has ignored. Moreover, claims have been made that some such indigenous characteristics are applicable across cultures and although derived through regional studies in China, they can inform Western psychology (Cheung, Cheung, Leung, Ward & Leong, 2003; Lin, 2003).

Despite these claims though, the University of Queensland and PsyAsia International have found that a test based on the FFM and Cattell’s 16 factor model, the 15FQ+, when properly localised to the Chinese context actually had superior technical properties than a locally developed indigenous test:

"The data indicates that Western personality tests that have been translated into Chinese can have acceptable levels of internal consistency reliability. Although still indicating a need for further refinement, these can be higher than the levels of internal consistency reliability achieved by tests developed on Chinese samples. Clearly, the reliability and validity of such tests is dependent upon both the test’s construction, and the quality of its translation. Furthermore, factor analyses have demonstrated that the Five Factor Model of personality holds up on a Chinese sample, adding further support to the universality of this model. Most notably, the current analyses have indicated that the translated versions of the 15FQ+ and NEO-FFI have, in the main, greater internal consistency reliability and criterion-related validity than does the CPAI-2’s indigenously-developed scales, despite this test having been developed on a Chinese population" (Tyler, 2005; Tyler & Newcombe, 2006)

(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Contextual versus Task Performance and personality as a predictor of each
(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Task performance relates to transforming raw materials into the goods and services which are specific to the job, the core technical skill. Contextual performance concerns aspects of an individual’s performance which maintains and enhances an organisation’s social network and the psychological climate that supports technical tasks. Borman and Motowidlo, (1997) contend that it is the contextual dimensions of job performance, rather than the technical components, that can be predicted from personality constructs and Borman, Penner, Allen and Motowidlo (2001) conclude that personality (Conscientiousness and Dependability) correlates more highly with organisational citizenship behaviours than with task performance. It appears that, conversely, cognitive abilities are more relevant for the prediction of task performance (Arvey & Murphy, 1998). Other research asserts that when the personality variables used in the prediction of performance are derived through job analysis, they predict task-related performance, as well as contextual job performance (Cue, 2002; Jenkins & Griffith, 2004). Jenkins and Griffith (2004) go on to state that in order to predict narrow aspects of performance, narrow traits (more specific, primary or facet level) should be used rather than broad global traits such as the FFM’s Openness to Experience, Conscientiousness, Extraversion, Agreeableness and Neuroticism.