Posts Tagged ‘
reliability of psychometrics ’
Friday, July 30th, 2010
It’s perhaps quite natural to believe that the Chinese personality is so different to others that it requires a special psychometric test to assess it. What better way to sell your new Chinese personality test than to state that it is “high time a test for the Chinese” were developed. However, this throws doubt upon the utility of rigorously developed international psychometric tests of personality.
Given the above, we embarked on a research program to assess whether Chinese people differ significantly comparied to others in terms of personality structure and whether personality tests that purport to assess Chinese Personality are able to predict any more work performance than internationally developed tests have already been proven to do!
You can read our research findings it: personality.cn, our Chinese Personality at Work Research Site.
No time to read the whole site? Here’s a quick summary:
Locally developed psychometric tests which purport to assess “indigenous” aspects of Chinese Personality were found to be less reliable than reputable internationally developed tests of personality. Furthermore, there is a big question as to whether so-called “indigenous” traits are Chinese-specific. Issues such as traditionalism or face also exist in other cultures! Moreover, the research has demonstrated that whatever we choose to believe about Chinese Personality, locally developed (Hong Kong) tests of “indigenous” personality add nothing to the prediction of performance at work that is not already accounted for by reputable internationally developed personality tests.
We present this research in a free HRM webinar which you can watch here. We held a vote at the beginning and end of our webinar whereby we asked attendees if they believed that Chinese Personality is so different that Chinese people need their own personality test. At the beginning of the webinar, the majority of the attendees said yes! By the end of the webinar only one attendee still believed this to be the case! We recommend choosing well designed psychometric tests with high reliability and validity. Personality is a universal construct, thus locally developed tests may have little benefit to the hiring manager!
Wednesday, July 28th, 2010
In this session we will explore the following:
1. The relationship between reliability and validity in psychometric assessment
2. How psychometric test administrators can impact the reliability of tests
Psychometric Test Reliability
When choosing a reputable test, whether it be aptitude or personality, one of the properties of the test you will need to look for is reliability. We’ll consider reliability in appropriate detail in a later section of the course. For now, think of reliability as consistency. In order to have absolute confidence in our test scores we need them to be consistent. However, we can’t test and retest our candidates in the real world. Despite this, reputable test publishers would already have done this for you. This would have been carried out under optimal conditions. So, now you know that you are using a reliable test (one that produces consistent scores), it’s your task as the test administrator to ensure that the test remains a reliable test.
Why is reliability so important?
Whenever you assess something, you expect the score you get to be reliable. For example, if you assess your weight using bathroom scales, you expect the reading you get to be consistent across at least the short term. If you weigh yourself over 2 consecutive days and get significantly different readings you know something is wrong with the scales! The same is true of psychometric tests. The publisher first ensures that the test scores will be consistent over time and then you, as the administrator, need to ensure that your actions do not make the test less reliable.
Not only do we want and expect test results to remain reliable over time, but we also know that reliability is a precursor to validity. It sets an upper limit on the test’s validity. In other words, if your test is not reliable then it is not valid. Confusing? Let’s use the weighing scales example again…
Let’s suppose a medical doctor does some research which shows that those who weight more than 120kg are significantly more likely to suffer a heart attack. His research shows that weight is a valid indicator for predicting the heart attack. The scales are fit for the purpose of predicting a heart attack. Validity is all about being fit for purpose. Now if those scales are not reliable, they will provide inconsistent data over the time of the research program. In this case would you have confidence in the doctor’s findings? Of course not!
So, to apply this to psychometric tests let’s take an aptitude test. We’ve carried out research which confirms that a new numerical reasoning test can predict the performance of accountants. Those who score better on the test are rated as better accountants. This is validity. The test is fit for the purpose of predicting accountant performance. You will hopefully have full confidence in this finding if you know the test is reliable. If however you expect the test is coming up with inconsistent scores for your candidates, it is unreliable, and, as in the scales example above, you will not have confidence in the test’s prediction of accountant performance. This is why reliability is a precursor to validity.
And why is all of this so important for this course? It’s because you as the test administrator can enhance or reduce the reliability of the test by how you administer it in the first place. Let’s now take a look at what factors you can and can’t influence in terms of reliability.
How psychometric test administrators can impact the reliability of tests
Factors Affecting Psychometric Test Reliability (C)2010 PsyAsia International: No Copying
Take a look at the graphic on the left. It shows different factors which can impact the reliability of psychometric tests. This applies to both aptitude tests and personality assessments.
Factors within the test
Generally, a test administrator is not responsible for this. The test publisher must design tests that will be highly reliable. Factors within the test means that the questions chosen must be accessible to all groups for whom the test is intended. If a subsection finds some questions difficult based on their group membership (i.e. non-native-English speaking groups may not understand a colloquialism used in a test question), then the test will be less reliable for that group. Although the publisher needs to ensure a reliable test, not all test publishers are reputable or know what they are doing! This is why the person who purchases the test needs to know how to evaluate it. We’ll show you later how to evaluate the test in greater detail. Know for now that you do not evaluate a test or validate it by trialling it on yourself or your colleague as many untrained users think!
Factors within the respondent
Whilst the test administrator cannot control all the possible factors within a respondent, you can do your best to ensure you control for a much as possible. It’s a good idea to think here about how you would like to be treated if you were undergoing a psychometric assessment for the first time. You’d probably like a friendly invitation letter explaining what is going to happen and why. You’d like to know that your data and results will remain confidential and only shared with decision-makers and only for the purpose that you’re undertaking the test. You’d also like to know what you need to bring with you and if possible, a few example questions as approved by the test publisher might help to set your mind at rest. Finally it would be good to have a number to call should you have any special needs that you wish to convey to the administrators before the day. So, when you arrive at the test centre you already know what is going to happen and why, you won’t be overly concerned, you’ll have all the right things with you (e.g., reading glasses) and you’ll know how long the session is going to last. If it’s a personality test you’ll be more likely to be open and honest because you know your results won’t go further than the selection or development committee and won’t be used for reasons beyond the reason you’ve already been given.
Ultimately here you are attempting to control for mood and expectations. Ideally you don’t want these to vary between candidates in order to give everybody the same start line. On the actual day of the test you will go over all of these things again with the candidates in the room to ensure that they are all clear on what will happen and why. Again, this sets the scene and mood, demonstrates your organisation’s “humanness” in the assessment process and provides candidates with an opportunity to ask questions. Furthermore, on the day you will need to ensure that you administer the test instructions word for word and then administer the test exactly as intended by the test publisher. Doing all of this enhances consistency and thus increases reliability. This is essential as we saw before because reliability is the precursor to validity.
Factors within the environment
How well would you be able to complete an aptitude test in a noisy room? Or how about room that’s freezing from too much air conditioning or too hot due to broken air conditioning? Likewise, you need to ensure that the test environment is conducive to candidate performance each and every time. This applies to personality assessment too. Although there is no right or wrong, your candidate will certainly feel more able to make an effort and respond accurately if you provide them with the right environment! So, some time before the session you’ll need to check the room, make sure temperature controls work. On the day, switch them on in good time before the test so that by the time candidates arrive the room is just right. Place a sign on the door to ensure you are not disturbed during the testing session and be sure to silence all phones in the room. Candidates should of course have phones switched off too. Ensure that once the session is over, all candidates leave at the same time so that they do not disturb others. If a candidate really must make a restroom visit, they should be accompanied by an administrator and only one candidate at a time should go. Ensure that upon leaving and rejoining the room the candidate does not disturb others.
(Note: also a good idea to check there is no planned construction nearby and there are no fire drills scheduled on the day of testing. Do this before sending out your invitation to the candidate!)
By referring to these guidelines you’ll help to ensure that psychometric tests used by your organisation remain as reliable as the publisher intends them to be. By using short-cuts and not following the guidelines you’ll threaten the reliability and therefore the validity of the tests. If you threaten a test’s validity it becomes unfit for purpose which means your company is wasting its money buying psychometric tools!
Interested in learning more about psychometric testing for HRM? Keep reading – your next free session is not far away! To ensure you don’t miss a single instalment, we suggest you follow-us on twitter as each new post will be announced there. You may also like to join our face-to-face psychometric training courses in Singapore or Hong Kong – these range from simple introductory courses through to Certification Courses such as the BPS Level A and BPS Level B Certificates of Competence in Occupational Testing. Not in Singapore or Hong Kong? No problem – we also offer both recorded and live online training in psychometrics! For full details please see here or email us.
DO NOT COPY OR SAVE THIS ARTICLE TO YOUR COMPUTER.
THIS ARTICLE IS CLEARED FOR PUBLISHING ON PSYCHOLOGY1 GROUP SITES ONLY. IT REMAINS COPYRIGHT AND INTELLECTUAL PROPERTY OF PSYASIA INTERNATIONAL PTE. LTD. YOU ARE NOT AUTHORIZED TO PUBLISH IT ON ANY OTHER SITE. YOU ARE NOT PERMITTED TO COPY/PASTE THIS ARTICLE OR TO SAVE IT TO YOUR LOCAL DRIVE. YOU ARE ONLY PERMITTED TO READ IT ONLINE AT OUR WEBSITE. VIOLATION OF THESE TERMS WILL RESULT IN BANNING OF OFFENDING IPS AND LEGAL ACTION FOR THOSE WHO REPUBLISH THIS ARTICLE WHETHER IT BE WITH OR WITHOUT A REFERENCE TO THE ORIGINAL AUTHOR.
Tuesday, March 23rd, 2010
PsyAsia International offers Free Psychometric Testing Course in Hong Kong & Singapore
Introduction to Psychometric Testing Course: Hong Kong, 4 May 2010; Singapore, 11 May 2010
PsyAsia International is Asia’s independent Leader in Psychometric Test products and Training. We choose to distribute only the world’s best, most validated psychometric assessments and offer locally relevant, world-class training in psychometrics. The Introduction to Psychometrics Workshop expands on PsyAsia’s expertise in Psychometric Training in Asia by offering a course geared to those with very little experience or understanding in Psychometrics. Many first time clients don’t understand why they need to be careful in their choice or use of psychometrics and many do not understand why training is a necessity in competent test use.
This one-day course aims to provide experienced-based training in an accessible and economical way. The course is easy to understand and yet covers many of the important issues to be aware of when choosing and using psychometric tests. Given our passion for Asia and our passion for the competent use of psychometric tests in Asia, PsyAsia makes no profit on this course. We charge delegates a small fee that reflects the cost of the hotel venue (including buffet lunch and refreshments) where the training is held as well as materials that we provide to the delegates. What’s more, if you later decide to attend one of our accreditation courses in Psychometrics, we will issue you with a discount code that reduces your course fee by the amount you paid for this course!
| The history of psychometric testing
Comparison of psychometric tests with other modes of employee testing and assessment
The benefit of using psychometric tests in recruitment/selection, development and coaching
Reliability in psychometric testing
Validity in psychometric testing
Error in psychometric testing
Review of different aptitude, personality and values tests on the market
Questions to ask your test publisher or distributor
What next?Note: During the workshop, delegates will create quasi-psychometric tests in groups to enable a hands-on exploration of issues such as reliability, error and validity in psychometric tests.
To view full course details and to register, please click here.
Wednesday, March 17th, 2010
The Amazing Apollo Profile
This free webinar will be facilitated by Mr. Jim Bowden, the developer of the Apollo Profile. The session will be interactive (provided attendees kit themselves out with headphones and a mic!) and Jim will present numerous interesting case studies.
The webinar will cover the following:
• Introduction: The Amazing Apollo Profile- can transform Recruitment, Staff Development, and Organisation Performance –Client example
• Apollo Questionnaire – valid/reliable/comprehensive
• Why is Apollo amazing? Apollo Advantages
• Using and interpreting of Apollo reports with anecdotes
• Recruitment – Accurate, easy, low cost – Case Study using Apollo Best Match in China for filtering 12,000 applicants for 40 Graduate level jobs
• Training and Development – Unique Apollo report PLUS downloadable solutions. Convenient, low cost, motivating
• Organisation Development. Benchmarking: Can analyse and identify current corporate strengths and weaknesses – then create high performing models/culture, identify engagement issues – case studies
• Customising: Develop models that work specifically for your organisation. If your organisation is serious about leadership through people.
• Integrate everything together with flexible multi-purpose Internet Online solutions. Use your own competencies frameworks and vocabulary – examples
• Special Offer – have to listen to Webinar to find out!
Date: Monday, May 17, 2010
Time: 12:30 PM – 1:30 PM SGT
After registering you will receive a confirmation email containing information about joining the Webinar.
Required: Windows® 7, Vista, XP, 2003 Server or 2000
Required: Mac OS® X 10.4.11 (Tiger®) or newer
Space is limited.
Reserve your Webinar seat now at:
Friday, November 20th, 2009
The Market for Psychometrics in Singapore
There are so many Psychometric Tests on the market in Singapore now, the task of choosing the right one is not easy. Choice is always a good thing, however as humans we often look for easy or stereotypical ways of making those choices and they are not always the best ones to make. For example, a client of ours was preparing for an upcoming team-building session. He approached us asking if we had a certain test that he could use in that session. Our answer was that we don’t supply that test for various very good reasons. The client’s response was “but so many people use it”. This is a typical response. Another potential client had been looking around in Singapore for Psychometric Personality Tests to use in his training sessions as an added benefit. He categorically advised us that he was not interested in validity and was looking for something simple and cheap! The reality here is that at best he is wasting his time and the time of those who will complete his tests. At worst and most likely, his trainees will be led to believe things about themselves which frankly may not be true (reliable or valid!).
Science, Psychology, Psychometrics and the Real World of Business
As busy professionals we often assume that if lots of other people are using a test it must be a good one. This is a huge mistake. Our evolution has programmed us to be seduced by glossy advertising materials and confident, friendly salespeople. On the other hand, we have a tendency to be turned off by less glossy scientific figures, statistics and perhaps psychologists such as myself who speak about the science and real value behind a test, its validity! Ultimately then, both our clients and ourselves as psychologists have problems to overcome!!
Psychologists have to be able to explain in more “glossy” terms about the technical properties of a test and our clients, usually the HR and aligned professions, are invited to turn their ears our way for a little while, just long enough to get the notion that there is more to a psychometric test than meets the eye!
Technical Properties of Psychometric Tests
When we talk of the technical properties of a psychometric test, we are referring to things such as its reliability and validity as well as how it was constructed. If a test is constructed well, it will take time. Not months, often years. The test will also evolve over time such that more and validity data will be added to its manuals. This process is costly, hence good tests cost money.
If you come across cheap tests, that should start to ring alarm bells. It’s possible to write a few questions on a napkin in a restaurant and call it psychometric and even try to sell it. If it looks good and the questions look relevant perhaps it will sell and gain a huge following. But how reliable is that test?
In other words, can it provide consistent measurement of your candidate? If your bathroom scales provide different results each time you weight yourself you take them back and say these are not reliable. Likewise with a test, you need to ensure that it is consistently assessing the constructs that it purports to assess. We often come across new clients who are shocked when we tell them that good personality tests often contain around 200 questions. However, buyer beware! We know that the longer the test, the more reliable the results (as long as it is not so long that the candidate falls asleep!).
An unreliable test can not be a valid test, hence reliability is a precursor to validity. However, validity is arguably the most important aspect of a test. You choose to use tests because you want them to illustrate where a candidate stands in terms of their ability or personality or in order to predict how your candidate will perform or behave in a job. The test’s ability to meet this need is referred to as validity.
Some tests on the market are simply more valid that others. In fact, one test in the past year has proven to be more valid than all other tests it was compared with on the market! How come users stay with their current test then? Perhaps because of preference, habit, price, mass-following and so on. However, do ask yourself and your test supplier, how valid is your test – this is the single most important technical property in a psychometric test!
Sometimes tests which are more valid will be more expensive but this makes sense. If a test took a long time to develop, was developed well and by a reputable publisher and is based on well founded theories that have been researched internationally, then surely it is worth paying the extra as such a test will provide an excellent return on investment with its strong validity.
Training to use Psychometric Tests in Singapore
Properly developed psychometric tests require proper training to be used competently. If your test supplier requires that you undergo very limited or no training, this is a reflection of the test as well as their lack of understanding of psychometrics. You need to understand the concepts referred to above, as well as error in testing and how to make decisions based on test results, let alone how to feed back results properly to candidates and decision-makers. The type of questions (i.e., forced choice versus rating scales) will also dictate how you can use the results – you need to be trained to understand this! In some parts of the world (South Africa for example), only psychologists can use psychometric tests. Whilst this is a strict rule, it has its logical basis in how easy it is for untrained professionals to use tests wrongly.
Purchasing Psychometric Tests in Singapore
You may also wish to consider where you purchase your tests from, particularly in Singapore. In recent years we have seen an influx of profiteers in the industry who seek to make money but lack any depth of understanding in psychometrics or psychology at work. This will change in time as psychology in Singapore develops. For now however, be wary of this and we suggest that you only purchase psychometric tests from fully registered organisational psychologists who have a firm grounding in personality, psychometrics and psychology at work and who are answerable to professional competence and ethics boards. Many of those selling psychometric tests in Singapore are simply not answerable to anybody in terms of their conduct or competence. You can therefore not be certain that any advice they provide is relevant, up-to-date or will work in your organisation.
There are many more things to be aware of when choosing psychometric tests in Singapore. We cannot entertain them all here due to space constraints. You may wish to look out for training courses in Psychometric Assessment such as our our Psychometric Assessment at Work training which leads to the internationally recognised British Psychological Society Level A and B Certificates of Competence in Occupational Testing. Such courses will prepare you further for choosing the right test and therein avoid costly selection and development mistakes. Look for courses run by experts in psychometrics who are based in Singapore and hence have a strong understanding of test use aligned with local culture, laws and practice.
Note: some Singapore firms will ship in overseas trainers to run psychometric training. We suggest you avoid this training reseller model given that the facilitator is based overseas and is thus likely to lack knowledge of the Singapore business/legal and cultural environment for Psychometric Testing.
This article is Copyright PsyAsia International Pte Ltd.
It was originally written for Human Resources Magazine in Singapore
A shorter version of the article appears in the magazine’s November 2009 issue
Monday, August 24th, 2009
The first thing to remember is that if you are using a purely ipsative personality test then you should not be comparing test results between candidates. Ipsative tests are self-referencing – they are comprised of force-choice items. They are useful in coaching, team-building and career guidance, but should not be used alone in recruitment and selection scenarios.
Some tests on the market, such as the Saville Consulting Wave or the Apollo Profile are joint normative-ipsative tests and these would be fine to be used to compare between candidates. A normative test is one which allows the candidate to respond based on the strength of their agreement or disagreement with a statement. The end results are then compared with a group of similar others who have previously taken the test (the norm group).
Purely normative tests such as the Identity Self-Perception Questionnaire would also be good to use for comparing candidates. Aptitude tests are by their nature normative tests and hence can be used to compare between candidates.
So, let’s assume that we have administered a normative personality assessment to two candidates and we are particularly interested in finding a candidate with a high tendency towards creative thinking. We have decided to use a personality assessment alongside other means of assessment including an abstract reasoning test to assess this. We ask Lee and Jane to complete both of these tests. These are their scores on the test scale of interest (presented in sten scores):
Now, keeping in mind that we would never use test results on their own to make a decision, let’s look at how most decision-makers would approach the above scenario based on test results alone for simplicity.
It obviously appears that Lee is somewhat better suited to the position than Jane.
However, in psychometric testing just as in any assessment procedure undertaken for Human Resources, there is always a chance of error. In fact, it’s more than chance! We know that error is always present.
When interviewing somebody the error is present, when running an assessment center the error is also present. Likewise, error is also present in the use of psychometric tests. Given a desire to be scientific, reputable test publishers will actually assess their tests for error.
One way of doing this is to ask a group of respondents to complete the test today and to invite them back a month later to complete the same test. Ignoring practice effects (which are controlled for), the expectation is that there should be a strong relationship between how a candidate scored at time one and how they score at time two. The idea is that test results should remain consistent over time. Psychometricians refer to this as test-retest reliability.
We hope for high test-retest reliability and we really should be choosing tests which have proven high levels. If we don’t we will have little confidence in test results and be very limited in terms of how we use them.
The assessment for error that shows us how much confidence we can have in test scores is referred to as the standard error of measurement (SEM). It uses an equation to ascertain how confident we can be that a candidate’s test result is a reflection of their true score as opposed to their true score PLUS error.
The equation is very simple, it is just: Standard Deviation multiplied by the square root of 1 minus the test-retest reliability of the assessment. If you don’t like statistics, sorry – they really are necessary to use tests competently!
If you choose a reputable test, often the publisher will quote the SEM in the test manual. If not, you can use the equation above to calculate it. You would use the standard deviation for your scale of interest taken from the manual alongside the test-retest from the manual (note…if your publisher fails to provide these figures you should probably not be using their tests!!).
The point is that the lower the SEM (or the higher the test-retest reliability), the better. Why?
Going back to Lee and Jane above. If our test has an SEM of 1.5 STENS, this would mean that we are 68% confident that Lee’s true score for the creative thinking scale is between 6.5 and 9.5 (we add and subtract the SEM from the observed score). It would also mean that we are 68% certain that Jane’s true score lies between 4.5 and 7.5 on the same test.
Now we can see that some doubt begins to arise as to whether the differences observed between the two candidates is as a result of a real score difference or an error difference (i.e., the true score for both candidates could be 7!). We don’t want to make a mistake and choose the wrong candidate, so let’s now look at how we can compare the differences.
We can take this further and calculate something called the standard error of difference. This tells us how confident we can be that there is a true difference between the scores of the two candidates. Because both candidates completed the same test, we use the following equation: SEdiff= the square root of (1.414 * SEM squared of the test in question).
Let’s say that our test has an SEM of 1.5 STENS. Using the SEdiff equation, we get a figure of 3.18 for the SEdiff. This represents our “critical figure”. It means that the difference between the candidate’s scores must be at least 3.18 before we can conclude there is a true score difference.
In our example, the difference between the candidate’s scores is only 2. Hence we cannot conclude there is a true score difference. The implication for selection is that we should not (everything else being equal) select one candidate over the other because, although we observe differences, the differences may not be true differences, they may be simply error differences.
Note that if we choose a more reliable test it will reduce the SEM. So for example, if we have an SEM of 1 STEN, our SEdiff for the above example would be 1.19. In this case, since the difference between the candidate’s scores is 2 STENS, we could conclude that there is a true difference. We would be at least 68% certain and almost 96% certain. We won’t go into degrees of certainty in this article, but the point is made!
In summary, do not compare candidate’s test results without a knowledge of the test’s reliability and standard deviation or in other words, do not ignore the SEM. Every assessment technique has an error variable. Competent users of psychometric tests will be aware of this and ensure they do not make the wrong selection decision or give incorrect development/careers advice on the basis on error rather than true score differences.
This article is (C) 2009 PsyAsia International. Some websites have been given permission to post this article. The article must always contain our copyright, publisher details and a live link to our website. Please do not violate these terms.