In this session we will explore the following:

1. The relationship between reliability and validity in psychometric assessment
2. How psychometric test administrators can impact the reliability of tests

Psychometric Test Reliability

When choosing a reputable test, whether it be aptitude or personality, one of the properties of the test you will need to look for is reliability. We’ll consider reliability in appropriate detail in a later section of the course.  For now, think of reliability as consistency.  In order to have absolute confidence in our test scores we need them to be consistent.  However, we can’t test and retest our candidates in the real world. Despite this, reputable test publishers would already have done this for you. This would have been carried out under optimal conditions.  So, now you know that you are using a reliable test (one that produces consistent scores), it’s your task as the test administrator to ensure that the test remains a reliable test.

Why is reliability so important?

Whenever you assess something, you expect the score you get to be reliable. For example, if you assess your weight using bathroom scales, you expect the reading you get to be consistent across at least the short term. If you weigh yourself over 2 consecutive days and get significantly different readings you know something is wrong with the scales!  The same is true of psychometric tests. The publisher first ensures that the test scores will be consistent over time and then you, as the administrator, need to ensure that your actions do not make the test less reliable.

Not only do we want and expect test results to remain reliable over time, but we also know that reliability is a precursor to validity. It sets an upper limit on the test’s validity. In other words, if your test is not reliable then it is not valid. Confusing?  Let’s use the weighing scales example again…

Let’s suppose a medical doctor does some research which shows that those who weight more than 120kg are significantly more likely to suffer a heart attack.  His research shows that weight is a valid indicator for predicting the heart attack.  The scales are fit for the purpose of predicting a heart attack.  Validity is all about being fit for purpose.  Now if those scales are not reliable, they will provide inconsistent data over the time of the research program.  In this case would you have confidence in the doctor’s findings? Of course not!

So, to apply this to psychometric tests let’s take an aptitude test. We’ve carried out research which confirms that a new numerical reasoning test can predict the performance of accountants. Those who score better on the test are rated as better accountants.  This is validity. The test is fit for the purpose of predicting accountant performance.  You will hopefully have full confidence in this finding if you know the test is reliable.  If however you expect the test is coming up with inconsistent scores for your candidates, it is unreliable, and, as in the scales example above, you will not have confidence in the test’s prediction of accountant performance. This is why reliability is a precursor to validity.

And why is all of this so important for this course?  It’s because you as the test administrator can enhance or reduce the reliability of the test by how you administer it in the first place.  Let’s now take a look at what factors you can and can’t influence in terms of reliability.

How psychometric test administrators can impact the reliability of tests

Factors Affecting Psychometric Test Reliability (C)2010 PsyAsia International: No Copying

Take a look at the graphic on the left. It shows different factors which can impact the reliability of psychometric tests. This applies to both aptitude tests and personality assessments.

Factors within the test

Generally, a test administrator is not responsible for this. The test publisher must design tests that will be highly reliable. Factors within the test means that the questions chosen must be accessible to all groups for whom the test is intended. If a subsection finds some questions difficult based on their group membership (i.e. non-native-English speaking groups may not understand a colloquialism used in a test question), then the test will be less reliable for that group. Although the publisher needs to ensure a reliable test, not all test publishers are reputable or know what they are doing! This is why the person who purchases the test needs to know how to evaluate it. We’ll show you later how to evaluate the test in greater detail.  Know for now that you do not evaluate a test or validate it by trialling it on yourself or your colleague as many untrained users think!

Factors within the respondent

Whilst the test administrator cannot control all the possible factors within a respondent, you can do your best to ensure you control for a much as possible.  It’s a good idea to think here about how you would like to be treated if you were undergoing a psychometric assessment for the first time. You’d probably like a friendly invitation letter explaining what is going to happen and why. You’d like to know that your data and results will remain confidential and only shared with decision-makers and only for the purpose that you’re undertaking the test. You’d also like to know what you need to bring with you and if possible, a few example questions as approved by the test publisher might help to set your mind at rest.  Finally it would be good to have a number to call should you have any special needs that you wish to convey to the administrators before the day.  So, when you arrive at the test centre you already know what is going to happen and why, you won’t be overly concerned, you’ll have all the right things with you (e.g., reading glasses) and you’ll know how long the session is going to last. If it’s a personality test you’ll be more likely to be open and honest because you know your results won’t go further than the selection or development committee and won’t be used for reasons beyond the reason you’ve already been given.

Ultimately here you are attempting to control for mood and expectations. Ideally you don’t want these to vary between candidates in order to give everybody the same start line.  On the actual day of the test you will go over all of these things again with the candidates in the room to ensure that they are all clear on what will happen and why.  Again, this sets the scene and mood, demonstrates your organisation’s “humanness” in the assessment process and provides candidates with an opportunity to ask questions.  Furthermore, on the day you will need to ensure that you administer the test instructions word for word and then administer the test exactly as intended by the test publisher. Doing all of this enhances consistency and thus increases reliability.  This is essential as we saw before because reliability is the precursor to validity.

Factors within the environment

How well would you be able to complete an aptitude test in a noisy room?  Or how about  room that’s freezing from too much air conditioning or too hot due to broken air conditioning?  Likewise, you need to ensure that the test environment is conducive to candidate performance each and every time.  This applies to personality assessment too. Although there is no right or wrong, your candidate will certainly feel more able to make an effort and respond accurately if you provide them with the right environment!  So, some time before the session you’ll need to check the room, make sure temperature controls work. On the day, switch them on in good time before the test so that by the time candidates arrive the room is just right.  Place a sign on the door to ensure you are not disturbed during the testing session and be sure to silence all phones in the room.  Candidates should of course have phones switched off too.  Ensure that once the session is over, all candidates leave at the same time so that they do not disturb others.  If a candidate really must make a restroom visit, they should be accompanied by an administrator and only one candidate at a time should go. Ensure that upon leaving and rejoining the room the candidate does not disturb others.
(Note: also a good idea to check there is no planned construction nearby and there are no fire drills scheduled on the day of testing. Do this before sending out your invitation to the candidate!)

Summary

By referring to these guidelines you’ll help to ensure that psychometric tests used by your organisation remain as reliable as the publisher intends them to be. By using short-cuts and not following the guidelines you’ll threaten the reliability and therefore the validity of the tests.  If you threaten a test’s validity it becomes unfit for purpose which means your company is wasting its money buying psychometric tools!

Interested in learning more about psychometric testing for HRM? Keep reading – your next free session is not far away! To ensure you don’t miss a single instalment, we suggest you follow-us on twitter as each new post will be announced there. You may also like to join our face-to-face psychometric training courses in Singapore or Hong Kong – these range from simple introductory courses through to Certification Courses such as the BPS Level A and BPS Level B Certificates of Competence in Occupational Testing. Not in Singapore or Hong Kong? No problem – we also offer both recorded and live online training in psychometrics! For full details please see here or email us.

DO NOT COPY OR SAVE THIS ARTICLE TO YOUR COMPUTER.
THIS ARTICLE IS CLEARED FOR PUBLISHING ON PSYCHOLOGY1 GROUP SITES ONLY. IT REMAINS COPYRIGHT AND INTELLECTUAL PROPERTY OF PSYASIA INTERNATIONAL PTE. LTD. YOU ARE NOT AUTHORIZED TO PUBLISH IT ON ANY OTHER SITE. YOU ARE NOT PERMITTED TO COPY/PASTE THIS ARTICLE OR TO SAVE IT TO YOUR LOCAL DRIVE. YOU ARE ONLY PERMITTED TO READ IT ONLINE AT OUR WEBSITE. VIOLATION OF THESE TERMS WILL RESULT IN BANNING OF OFFENDING IPS AND LEGAL ACTION FOR THOSE WHO REPUBLISH THIS ARTICLE WHETHER IT BE WITH OR WITHOUT A REFERENCE TO THE ORIGINAL AUTHOR.