Some tests on the market provide a handful of scales when assessing a respondent, whilst others provide notably more scales. Often and if produced based on the greatest amount of research evidence, these tests relate to the five-factor model of personality. Our 15FQ+ has 5 global/secondary scales and 16 primary scales and this, according to the commentry below, sets it in a good position when predicting performance at work based on personality traits.

The bandwidth fidelity dilemna
(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

The point above relates to a debate that is often referred to as the bandwidth-fidelity dilemma (Cronbach & Gleser, 1965); that is the assessment of gain or loss in analytical and predictive power from using broad-band versus narrow-band personality assessments. Goldberg’s (1972) study, using scales developed from the California Psychological Inventory (Consulting Psychologists Press, 1969) item pool with a sample size of 179, led him to conclude that five or six factors could predict a series of 7 criteria (including Grade Point Average, dating success and years spent at college) as well as could 11 narrower factors. Then more recently, Ones and Viswesvaran (1996) advocated the use of broad personality factors such as those within the FFM, rather than narrower traits such as those of the 16PF primary scales in the prediction of behaviours. Based on a large meta-analysis and using broad factors, Salgado (2003) found that personality measures developed within the FFM showed higher criterion-related validity than those based on alternative theoretical viewpoints. This only held true for Conscientiousness and Emotional Stability however with the other three FFM scales showing no differences. Conversely, Hogan and Roberts (1996) argue that the use of narrow personality traits accounts for variance that is situation specific and that broad trait measures are unable to tap into this variance. They argue that there is no evidence to suggest that the fidelity-bandwidth trade-off issue has become a crisis, suggesting that the nature of the criterion should dictate the choice of predictors in order to enhance validity. In support of Hogan and Robert’s first contention, Paunonen (1998) demonstrated that the Personality Research Form (Jackson, 1984; a narrow-band trait measurement) was able to account for more variance than the NEO-PI-R broad-based measure and concluded that the aggregation of narrow personality traits into broad factors may lead to decreased predictive ability due to a loss of trait-specific variance. Additionally, Mershon and Gorsuch’s (1988) meta-analysis found that the 16 factors of the 16PF were able to explain at least twice as much variance in the criterion (of which there were various) as would a 6-factor approach. They discovered a 110% median increase in the proportion of variance accounted for when moving from six factors to sixteen.

Black (2000) suggests that the broad five factors of the NEO may limit their usefulness in selection settings and both Saville, Nyfield, Sik, and Hackston (1991) and Driskell, Hogan, Salas and Hoskin (1994) found that specific facets of the Big-Five constructs were better predictors of performance than global level measures such as the five factors themselves. The Driskell et al. study however found that, although personality was able to predict academic criteria in Naval (electronics) trainees, it contributed no additional variance in academic performance to that offered by the Armed Force’s own cognitive assessment — despite personality being associated with attitudinal and motivational factors that were implicated in training success.

In summary, and although the evidence regarding exactly what types(s) of performance can be predicted from what type(s) of personality dimensions may be in dispute, it is clear that personality does have utility in the performance prediction arena. Given the evidence and more specifically, Barrick, Mount and Judge’s (2001) meta-analysis findings, one is able to conclude that, when used responsibly and in a standardised manner by appropriately trained personnel, personality assessments based on the FFM add an element to the prediction of an individual’s workplace performance that is not accounted for by other human resource tools and methods.

When reviewing literature on this topic it is notable that the correlations that are reported between personality and performance are typically not strong (Robertson & Smith, 2001) given the complex interplay between all predictor variables and job performance. This may lead one to critique that although relationships found in many studies may have been statistically significant, they ultimately remain meaningless given the small size of the coefficient. Meyer et al. (2001) provided a review and extensive tables of correlation coefficients from psychological testing research. The reason for the low coefficient is often simply due to the fact that there is a very small relationship between the two variables. However, on many occasions, criterion reliability and validity issues mean that an observed correlation between predictor and dependent variable would in reality have been higher if correction for attenuation was made (Salgado, Moscoso & Lado, 2003; Salgado, Ones & Viswesvaran, 2001). Nunnally (1978) provided an equation that corrects for such attenuation, although Muchinsky (1996) noted that such an equation cannot be subjected to statistical significance testing. Moreover, it is of greater importance and necessity to work to increase criterion reliability and validity and thus to eradicate the necessity to apply such corrective measures.

(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

McCrae (2004) stated “…trait structure, age and gender differences, and cross-observer agreement are all universal…” (p.3). Earlier, Passini and Norman (1966) were less categorical with a hint at the possibility of a universal conception of personality and Digman and Inouye (1986) noted the possibility that the FFM was not only applicable to “American populations” and to the “English language” (p.116) but also to non-Western cultures such as China, Japan and the Philippines. However, Guthrie and Bennett’s (1971) work with Philippine participants led them to dispute Passini and Norman’s (1966) claim that the dimensions tapped by Norman’s (1963) measuring instrument (basic dimensions of human perception) were universal. These researchers found distinct differences in personality structure when comparing Philippine participants with an American norm (sample size of 100). More specifically, the Philippino sample tended to hold a similar conception of Extraversion to the Americans, but Culture and Conscientiousness factored down into a Sophistication factor (Guthrie & Bennett, 1971). On the other hand, Emotional Stability broke down into two factors labelled Worry/Anxiety and Somatic Symptoms. Finally, Agreeableness was found to include more behavioural manifestation in the Philippine sample than the American sample. With this in mind, Bond, Nakazato and Shiraishi (1975) carried out research using a Japanese sample and Norman’s (1963) original measuring instrument and procedure. Their research showed that although three of the five factors were clearly replicated, the factors of Emotional Stability and Culture were less congruent when compared with Norman’s (1963) US sample. This, according to Bond et al. (1975), indicated that these two factors are construed in different ways in Japan in comparison to the USA.

Jackson’s (1984) 352 item Personality Research Form (PRF) measures 22 scales related to Murray’s list of Needs (Murray, 1938). The PRF has been widely viewed as a “model of scale construction” (Costa & McCrae, 1995, p.217). Factor analysis following administration of this questionnaire in a variety of cultures (Canada, England, the Netherlands, Norway, and Israel) has shown that five factors emerge which can be interpreted within the FFM (Paunonen et al., 2000). McCrae and Costa (1997) found that the factor structure of personality as measured by the NEO-PI-R (Costa & McCrae, 1992a) was similar across 7 different cultural and 5 diverse language groups (American, German, Portugese, Hebrew, Chinese, Korean and Japanese). Factors A (Agreeableness) and E (Extraversion), however, showed less congruence with the Japanese sample when compared to an American normative sample. This is interesting given Saucier and Goldberg’s (2001) observation that it is typically factors N (Neuroticism) and O (Openness to Experience) that do not always appear in cross-cultural lexical studies of the Big-5. However, returning to the FFM (as opposed to the Big-5 conceptualisation), Agreeableness and Openness showed the lowest internal consistencies in McCrae et al.’s (2000) study of a translated/locally modified NEO-FFI in the UK, Germany, Spain, the Czech Republic and Turkey. Costa and McCrae (1992a) make the important point that the finding of congruence between American normative samples and other cultures does not imply that each of those cultures does not possess other traits outside of the FFM, that is, indigenous traits.

In another comparison of the factor structure of the NEO-PI-R, McCrae et al. (1996) found that the personality factor loadings of a Chinese sample of 352 Hong Kong university students displayed a very close congruence to an American normative sample. This provided further support for the cross-cultural replicability of the NEO-PI-R (and FFM), although, facet O4: Actions, had a weaker congruence with the American data. McCrae himself was initially ‘astounded’ that a number of studies showed the factor structure of the NEO-PI-R replicated “almost perfectly” across cultures (McCrae, 2002), although he goes on to state “…I have come to expect that all basic features of trait psychology are universal…” (McCrae, 2002: Sect. About the Author). This is not surprising given that overall, studies of the factor structure of the NEO-PI-R in 40 languages/dialects from over 30 cultures have found at least reasonable approximation to the questionnaire’s intended factor structure (McCrae & Allik, 2002). Allik and McCrae (2004), conducting a secondary analysis on data from 36 cultures found that geographically proximate cultures often have similar NEO-PI-R profiles. They found that American and European profiles were in contrast to African and Asian profiles, with the former being higher in Extraversion and Openness to Experience and lower in Agreeableness. They explain that the findings may be due to differences in the gene pool or have to do with acculturation effects.

McCrae and Terraciano (2005) provide further evidence for the universality of the human trait structure in research carried out in fifty cultures representing six continents. This most recent work has examined not only self-report data, but also peer/acquaintance (3rd person NEO-PI-R questionnaire) reports of the participant. Results showed that the American self-report structure was replicated in most cultures and was recognisable in all cultures. With total congruence coefficients greater than .90 being generally regarded as indicating good factor replication (Yang et al., 1999), the Nigerian sample was the least congruent overall (.71), followed by the Botswana (.82), Moroccan (.85) and Indian (.89) samples. All remaining 46 samples were congruent with the USA normative sample with total congruence coefficients at or above .90. The most problematic single factor in terms of congruence was Factor O (Openness to Experience); this was also the least reliable factor in terms of internal consistency across cultures (see Section 2.7.2 for further discussion). Although the majority of participants in this research were students, the evidence for universality of personality traits cannot be ignored.

Narayanan, Menon and Levine (1995) also studied the cross-cultural robustness of the FFM. Unique to other studies, their research was carried out in India with 221 university students. Employing an emic strategy, they used the free-descriptor method (in which the pool of items is generated by the participants themselves, thus avoiding experimenter-imposed variables: see John, 1990) as a quantitative method of personality assessment and the critical incident approach (Flanagan, 1954) as a qualitative exploration of personality. Analysis of their data collected from both methodologies “strongly supported the five-factor model, whilst also revealing certain culturally based departures” (p.51). One of these “culturally-based departures” was a sixth dimension, outside of the five major factors of personality, that the authors named “Miscellaneous”. It accounted for 4.1% of the variance and included critical incidents involving morality, conservatism and nationalism.

Finally, similarities in the age-related development of personality across cultures have been reported. With a sample of 5085 adolescents and adults in the UK, Germany, Spain, the Czech Republic and Turkey, Costa and McCrae (2001) found that Neuroticism, Extraversion and Openness to Experience declined with age while Agreeableness and Conscientiousness tended to increase. This was partially supported in a Chinese study by Xiu, Wu, Wu and Shui (1996) who found small but significant age effects in a sample of 593 men and women of ages 20-84 using a Chinese version of the NEO-FFI. Neuroticism and Openness to Experience declined with age whereas Agreeableness increased with age.

Small personality-based gender differences across cultures have also been found in a sample of over 23,000 individuals taken from 26 cultures. For example, females tend to score higher on Neuroticism, Warmth, Agreeableness and Openness to Feelings and men tend to score higher on Assertiveness and Openness to Ideas (Costa, Terracciano & McCrae, 2001).

In summary, a large body of research has demonstrated that the FFM has utility outside of the USA, although congruences and similarities may, at times, not be as high as desired. Likewise, there is a possibility that FFM models do not capture the complete depth of human experience. Gender and age-related similarities in profiles across cultures have however provided more evidence for the cross-cultural applicability of the FFM.

(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

The labels Big-Five and Five Factor Model (FFM) are often used interchangeably when considering the trait approach to personality theory. De Fruyt, McCrae, Szirmák and János (2004) clarify that the Big-Five is derived from the lexical approach associated with Allport and Odbert (1936), Fiske (1949), Norman (1963), Tupes and Christal (1961) and Goldberg (1981), whereas the FFM is essentially associated with the emergence of personality factors through the questionnaire approach as in the work of McCrae and Costa (1985).

Big-Five research dates back to Galton (1884) and Baumgarten (1933), although is most often associated with Allport and Odbert (1936). Goldberg (1990) credited Galton (1884) with possibly being “…among the first scientists to explicitly recognize the fundamental lexical hypothesis – namely that the most important individual differences in human transactions will come to be encoded as single terms in some or all of the world’s languages” (Goldberg, 1990, p.1216). Allport and Odbert (1936) reviewed the Webster’s New International Dictionary (2nd Edition) of the time and arrived at a lexicon of 17,953 terms that were “descriptive of personality or personal behaviour”. They grouped these words into four columns: “neutral terms, designating possible personal traits”, “terms primarily descriptive of temporary moods or activities”, “weighted terms conveying social or characterial judgements of personal conduct” and “miscellaneous terms”. Pervin (1993) concluded that Allport is more likely to be remembered for the issues that he raised, rather than for a particular theory, given that, for example, although he believed that many traits were hereditary, he did not substantiate this with research evidence.

The empirical evidence for trait theory began with psychologists such as Thurstone, Cattell and Eysenck. Thurstone (1934) provided the first recorded attempt made at factoring personality adjectives to arrive at a five-factor solution. Spearman (1937b) used the then developing tool of factor analysis in the establishment of the ‘g’ or “General Intelligence” factor. Cattell (1943), who was a student of Spearman, also applied factor analysis to trait psychology (see below). Eysenck (1947) announced the successful isolation of two distinct personality factors following research with 10,000 “normal” and “neurotic” participants and factor analysis of the intercorrelations among 39 trait ratings made by psychiatrists on 700 “neurotic” individuals. Eysenck developed his model from Cattell’s Sixteen-Factor Model.

In the early days of trait theory, the results of factor analyses of personality were inconsistent. Cattell (1943) noted that this was largely due to the use of different measures, biases of researchers, limited sampling of participants and aspects of personality, and differences in how traits were named. Cattell suggested that results could be improved by factor analysing the complete “sphere” of trait names. He grouped synonyms and opposites within Allport and Odbert’s list of trait names and reduced this to 150 categories. Cattell then added the names of ten special abilities and eleven special interests resulting in a total of 171 descriptions of behaviour. Cluster analysis of these descriptions reduced them to 60 observable patterns of behaviour which Cattell named “surface traits”. Factor analysis of the surface traits reduced them to 16 “source traits” (something within the person, but not directly observable, considered to be the causal influence of observable behaviour). Cattell used orthogonal rotation in his original factor analysis, thus allowing him to carry out a further factor analysis of the 16 source traits, producing a final four (Cattell, 1956) or five (Cattell, Eber & Tatsuoka, 1970) or even eight (Cattell, 1973) second-order factors. The issue of how many factors to extract followed a familiar theme within trait psychology for some years and even recently, there has been disagreement as to the actual number of personality traits, be they first or second order (Mershon & Gorsuch, 1988).

With reference to Cattell’s body of work cited above, Digman and Takemoto-Chock (1981) noted a number of clerical errors in Cattell’s data, including incorrect signage (use of positive instead of negative). Further, they noted that options available within the factor analytical technique have the effect of producing inconsistent results among researchers. Cattell was subject to a number of other criticisms given the inability of researchers to replicate his findings (Eysenck & Eysenck, 1969; Howarth & Brown, 1971; Levonian, 1961). Initially, Cattell claimed this was due to researchers not following the exact Cattell methodology (although this methodology has of itself been subject to criticism — see, for example, Howarth, 1976). To address Cattell’s concern, Barrett and Kline (1982) strictly followed the Cattell methodology and, even so, were not able to confirm the sixteen factors on a group of 491 undergraduates, instead finding between seven and nine factors with sufficient reliabilities and factor validities. More inconsistencies emerged from a Swiss study that confirmed a four-factor solution on a sample of 386 general population participants (Rossier, Meyer de Stadelhofen, & Berthoud, 2004).

Despite the criticisms and non–replication of factors within Cattell’s approach, he and his colleagues were responsible for the Sixteen Personality Factor Questionnaire (16PF: Cattell et al., 1973) and Cattell did pave the way for the development of the FFM. The next major player in the development of trait theory was Fiske (1949). Fiske’s factor analysis of peer, self and psychologist ratings of 128 clinical trainees rated on 22 scales of surface behaviour was found to reveal four major factors: Social Adaptability, Emotional Control, Conformity, and Inquiring Intellect. Following this, Norman (1963), working with male university students, found through peer nomination rating methods using twenty paired behavioural descriptions, that there was evidence for the existence of five relatively orthogonal personality dimensions. These dimensions were labelled: Extraversion, Good-naturedness, Conscientiousness, Emotional Stability and Culture. However, it was Tupes and Christal (1961) and Goldberg (1981) who actively sought to confirm the existence of the five factors and later work by McCrae and Costa (1985, 1987) resulted in interpreting the Culture factor as “Openness to Experience”. Hogan (1982) put forward his socioanalytic theory, based on the five-factor model. This theory places importance upon both the “actor” and the “observer” in the assessment of personality and its implications in the workplace. Furthermore, it considers that social situations exist only within an individual’s subjective understanding and not within the physical environment. Hogan’s theory is often positively cited as being the only “theory” within the five-factor model. From this he developed the Hogan Personality Inventory (HPI) and the Hogan Development Survey (HDS). The former measures what Hogan terms the “bright-side” of personality (i.e., normal range personality) and the latter assesses the “dark-side” (negative changes to an individual’s normal personality when under stress of one type or another). Continuing the long-standing debate of how many factors adequately account for the entire domain of personality, it should be noted that the HPI contains seven main measurement scales: the Big-5 and a further two (Hogan & Holland, 2003). Subsequently, Costa and McCrae (1985) developed the NEO Personality Inventory and the Revised NEO Personality Inventory (NEO-PI-R) (1992). The NEO-PI-R assesses 30 specific traits, six for each of the five factors. It is the most widely used of a variety of available measures of the FFM (McCrae, 2002). This may be due to the prominence of its developers, its ease of acquisition – despite remaining relatively secure and not being posted throughout the Internet, the fact that it has been translated into many languages and the fact that it is a relatively short yet psychometrically robust measure of personality.

(text from and remains copyright PsyAsia’s research site at www.personality.cn; please visit that site for reference lists)

Validity means “Is the test fit for purpose?”

Some different types of validity:

Face Validity (low-level of importance overall)
• Asks: “Do the questions appear to measure what the test purports to measure?”
• Important for: Respondent buy-in
• How assessed: Simply by looking at the questions

Content Validity
• Asks: “Do there appear to be enough suitable questions to measure the complete construct we are trying to measure?”
• Important for: Ensuring a holistic assessment of the construct
• How assessed: Asking subject matter experts to review the questionnaire

Construct Validity (one of the two highest levels of importance overall)
• Asks: “Does the test actually assess the construct it purports to assess?”
• Important for: Ensuring that the construct is being measured by the test and for use in norm-referenced testing
• How assessed: Correlate respondents scores on the test under examination with a well established measure of the same construct. The expectation is that there will be a high correlation given that similar constructs should converge.

Criterion-related validity (one of the two highest levels of importance overall)
• Asks: “Can a respondent’s test score predict a real world outcome such as performance at work?”
• Important for: Knowing whether the test can predict anything meaningful and for use in criterion-referenced testing
• How assessed: Correlate test scores with criteria such as performance appraisal scores.

Concurrent criterion related validity – test scores predict a criterion (e.g., performance) now

Predictive criterion related validity – test scores predict a future criterion (e.g., whether or not respondent will pass training)

Vocab:
Construct = (e.g.) A personality attribute such as extraversion or an aspect of ability such as numerical reasoning
Correlation = Relationship between 2 or more variables/constructs/things

Reliability refers to consistency of measurement.

If you measure the length of your office wall on two occasions and get two different measurements, you know that something is not right! Maybe you changed your viewing angle of the tape measure between the measurements? Maybe you held the tape measure differently on each occasion?

Reliability is a crucial necessity in psychometric assessment. If there is a lack of reliability, there is a lack of consistency in the scores that test respondents receive and, as a result, the interpretation of their profile of behaviours and abilities changes.

No test can claim to be 100% reliable – indeed, neither can any method of assessment. Test publishers, distributors and users are all part of the process of enhancing the reliability of well designed psychometric tests. The following factors (and more) can all impact upon reliability:

Factors within the testing environment
Such as noise or temperature

Factors within the respondent
Such as mood or desire to undertake the assessment

Factors within the test itself
Such as ambiguous questions or language that a respondent does not understand

If a test lacks consistency in measurement, then it can never be a valid test.
Therefore, reliability is a necessary precursor to validity.

You need to be trained to a competent standard in psychometric testing first.  If you don’t have the time, find someone who has been trained or could be trained.  They can then advise you fully on each of the following issues.  Keep in mind that not every psychometric test is the same.  There are reasons why some can be made available online and others cost what appears to be a lot of money!  Make sure you consider each of the following points in some detail: 

Firstly, ensure that you understand the theoretical rationale upon which the test was developed.  It should be based upon sound scientific theory!  Keep this in mind, in addition to your own reason for using the test (staff selection, departmental development, counselling etc.) as you gather the following information:

1. Suitability of the questions: are the questions in the test suitable given your target and context? Are they easy enough for your group to understand? Do they appear to measure what the test purports to measure?  Do you think any of the questions might cause offence to the respondent?

2. Reliability: source reliability data from the test publisher.  You can usually find this in the test manual.  Ensure that internal consistency and test-retest data exists.  Coefficients listed in the manual or quoted by the publisher should reach at least .70 for personality tests and .80 for ability/aptitude tests.

3. Validity: look for evidence of validity in the test manual. The publisher should note at least one out of construct validity and criterion-related validity. Basically, the data should suggest that the construct the test purports to measure is validly measured by the test and/or that scores on the test can predict a real-world meaningful outcome such as performance or absenteeism.

4. Normative Data: find out if the publisher provides appropriate norm groups for score comparison purposes. Are the sample sizes of those norm groups adequate? Do they represent your target group? Are they representative in terms of ethnicity and gender?

5. Practical Considerations: How much does the test and all required items (such as software, scoring keys etc.) cost? What level of training is required and how much does that cost? What are the scoring procedures? Is it easy/complex to score? If complex, is it worth it in terms of information gained? Can the test be computer scored? How much does the software cost?

PsyAsia trains all of our course delegates to be able to do the above as second nature in our Psychometric Assessment at Work Course.