In this session we will explore the following:

1. The relationship between reliability and validity in psychometric assessment
2. How psychometric test administrators can impact the reliability of tests

Psychometric Test Reliability

When choosing a reputable test, whether it be aptitude or personality, one of the properties of the test you will need to look for is reliability. We’ll consider reliability in appropriate detail in a later section of the course.  For now, think of reliability as consistency.  In order to have absolute confidence in our test scores we need them to be consistent.  However, we can’t test and retest our candidates in the real world. Despite this, reputable test publishers would already have done this for you. This would have been carried out under optimal conditions.  So, now you know that you are using a reliable test (one that produces consistent scores), it’s your task as the test administrator to ensure that the test remains a reliable test.

Why is reliability so important?

Whenever you assess something, you expect the score you get to be reliable. For example, if you assess your weight using bathroom scales, you expect the reading you get to be consistent across at least the short term. If you weigh yourself over 2 consecutive days and get significantly different readings you know something is wrong with the scales!  The same is true of psychometric tests. The publisher first ensures that the test scores will be consistent over time and then you, as the administrator, need to ensure that your actions do not make the test less reliable.

Not only do we want and expect test results to remain reliable over time, but we also know that reliability is a precursor to validity. It sets an upper limit on the test’s validity. In other words, if your test is not reliable then it is not valid. Confusing?  Let’s use the weighing scales example again…

Let’s suppose a medical doctor does some research which shows that those who weight more than 120kg are significantly more likely to suffer a heart attack.  His research shows that weight is a valid indicator for predicting the heart attack.  The scales are fit for the purpose of predicting a heart attack.  Validity is all about being fit for purpose.  Now if those scales are not reliable, they will provide inconsistent data over the time of the research program.  In this case would you have confidence in the doctor’s findings? Of course not!

So, to apply this to psychometric tests let’s take an aptitude test. We’ve carried out research which confirms that a new numerical reasoning test can predict the performance of accountants. Those who score better on the test are rated as better accountants.  This is validity. The test is fit for the purpose of predicting accountant performance.  You will hopefully have full confidence in this finding if you know the test is reliable.  If however you expect the test is coming up with inconsistent scores for your candidates, it is unreliable, and, as in the scales example above, you will not have confidence in the test’s prediction of accountant performance. This is why reliability is a precursor to validity.

And why is all of this so important for this course?  It’s because you as the test administrator can enhance or reduce the reliability of the test by how you administer it in the first place.  Let’s now take a look at what factors you can and can’t influence in terms of reliability.

How psychometric test administrators can impact the reliability of tests

Factors Affecting Psychometric Test Reliability (C)2010 PsyAsia International: No Copying

Take a look at the graphic on the left. It shows different factors which can impact the reliability of psychometric tests. This applies to both aptitude tests and personality assessments.

Factors within the test

Generally, a test administrator is not responsible for this. The test publisher must design tests that will be highly reliable. Factors within the test means that the questions chosen must be accessible to all groups for whom the test is intended. If a subsection finds some questions difficult based on their group membership (i.e. non-native-English speaking groups may not understand a colloquialism used in a test question), then the test will be less reliable for that group. Although the publisher needs to ensure a reliable test, not all test publishers are reputable or know what they are doing! This is why the person who purchases the test needs to know how to evaluate it. We’ll show you later how to evaluate the test in greater detail.  Know for now that you do not evaluate a test or validate it by trialling it on yourself or your colleague as many untrained users think!

Factors within the respondent

Whilst the test administrator cannot control all the possible factors within a respondent, you can do your best to ensure you control for a much as possible.  It’s a good idea to think here about how you would like to be treated if you were undergoing a psychometric assessment for the first time. You’d probably like a friendly invitation letter explaining what is going to happen and why. You’d like to know that your data and results will remain confidential and only shared with decision-makers and only for the purpose that you’re undertaking the test. You’d also like to know what you need to bring with you and if possible, a few example questions as approved by the test publisher might help to set your mind at rest.  Finally it would be good to have a number to call should you have any special needs that you wish to convey to the administrators before the day.  So, when you arrive at the test centre you already know what is going to happen and why, you won’t be overly concerned, you’ll have all the right things with you (e.g., reading glasses) and you’ll know how long the session is going to last. If it’s a personality test you’ll be more likely to be open and honest because you know your results won’t go further than the selection or development committee and won’t be used for reasons beyond the reason you’ve already been given.

Ultimately here you are attempting to control for mood and expectations. Ideally you don’t want these to vary between candidates in order to give everybody the same start line.  On the actual day of the test you will go over all of these things again with the candidates in the room to ensure that they are all clear on what will happen and why.  Again, this sets the scene and mood, demonstrates your organisation’s “humanness” in the assessment process and provides candidates with an opportunity to ask questions.  Furthermore, on the day you will need to ensure that you administer the test instructions word for word and then administer the test exactly as intended by the test publisher. Doing all of this enhances consistency and thus increases reliability.  This is essential as we saw before because reliability is the precursor to validity.

Factors within the environment

How well would you be able to complete an aptitude test in a noisy room?  Or how about  room that’s freezing from too much air conditioning or too hot due to broken air conditioning?  Likewise, you need to ensure that the test environment is conducive to candidate performance each and every time.  This applies to personality assessment too. Although there is no right or wrong, your candidate will certainly feel more able to make an effort and respond accurately if you provide them with the right environment!  So, some time before the session you’ll need to check the room, make sure temperature controls work. On the day, switch them on in good time before the test so that by the time candidates arrive the room is just right.  Place a sign on the door to ensure you are not disturbed during the testing session and be sure to silence all phones in the room.  Candidates should of course have phones switched off too.  Ensure that once the session is over, all candidates leave at the same time so that they do not disturb others.  If a candidate really must make a restroom visit, they should be accompanied by an administrator and only one candidate at a time should go. Ensure that upon leaving and rejoining the room the candidate does not disturb others.
(Note: also a good idea to check there is no planned construction nearby and there are no fire drills scheduled on the day of testing. Do this before sending out your invitation to the candidate!)

Summary

By referring to these guidelines you’ll help to ensure that psychometric tests used by your organisation remain as reliable as the publisher intends them to be. By using short-cuts and not following the guidelines you’ll threaten the reliability and therefore the validity of the tests.  If you threaten a test’s validity it becomes unfit for purpose which means your company is wasting its money buying psychometric tools!

Interested in learning more about psychometric testing for HRM? Keep reading – your next free session is not far away! To ensure you don’t miss a single instalment, we suggest you follow-us on twitter as each new post will be announced there. You may also like to join our face-to-face psychometric training courses in Singapore or Hong Kong – these range from simple introductory courses through to Certification Courses such as the BPS Level A and BPS Level B Certificates of Competence in Occupational Testing. Not in Singapore or Hong Kong? No problem – we also offer both recorded and live online training in psychometrics! For full details please see here or email us.

DO NOT COPY OR SAVE THIS ARTICLE TO YOUR COMPUTER.
THIS ARTICLE IS CLEARED FOR PUBLISHING ON PSYCHOLOGY1 GROUP SITES ONLY. IT REMAINS COPYRIGHT AND INTELLECTUAL PROPERTY OF PSYASIA INTERNATIONAL PTE. LTD. YOU ARE NOT AUTHORIZED TO PUBLISH IT ON ANY OTHER SITE. YOU ARE NOT PERMITTED TO COPY/PASTE THIS ARTICLE OR TO SAVE IT TO YOUR LOCAL DRIVE. YOU ARE ONLY PERMITTED TO READ IT ONLINE AT OUR WEBSITE. VIOLATION OF THESE TERMS WILL RESULT IN BANNING OF OFFENDING IPS AND LEGAL ACTION FOR THOSE WHO REPUBLISH THIS ARTICLE WHETHER IT BE WITH OR WITHOUT A REFERENCE TO THE ORIGINAL AUTHOR.

PsyAsia’s range of early-bird and group booking discounts for our October Psychometric Training Courses in Singapore will expire in just 7 days!  If you intend coming on our October training for BPS Level A, BPS Level B (including Identity and Apollo Personality Questionnaire accreditation), Saville Consulting Wave or BPS Test Administration, please do ensure that you register by 30 July. All discounts will expire in our systems and cannot be reinstated!  Please view all course dates at http://dates.psyasia.com.

Remember, PsyAsia International is Asia’s Independent Leader in Psychometric Tests and Training. We only sell the world’s best psychometric tests, our courses are run by resident  fully registered psychologists and we offer blended e-learning with our face-to-face training for all BPS certification courses.

If you have any queries about our psychometric courses, do please get in touch – an easy way to get an instant response is to use the LIVE CHAT facility on our website. Alternatively, please email us.

Finally, don’t forget that we’ve arranged a session for anyone interested in Singapore Level A and B training to meet with the facilitator online. He will show you course slides and workbooks and answer any questions you may have. This session runs on Tuesday 27th July at 5pm Singapore/Hong Kong time. Click here to register.

FREE HRM WEBINAR

PsyAsia International is pleased to announce the next webinar in our series of professional HR webinars. This time we will be discussing the topic of Chinese Personality and performance at work.

Some HR people in Asia believe that culture plays such a significant role in personality that indigenous personality attributes need to be assessed at recruitment/selection. To this end, personality tests have been developed “in Chinese for the Chinese by the Chinese”. A significant question to ask is: Do these tests add any prediction over and above that afforded by mainstream personality tests developed by world renowned experts in the field?

The above questions will be answered through discussion of the trait model of personality and its biological basis. Peer-reviewed and published research conducted by PsyAsia International’s award-winning Psychologist, Dr. Graham Tyler; award-winning Dr. Peter Newcombe of the University of Queensland; and world-renowned Professor Paul Barrett, formerly of the University of Auckland will be presented in an easy to understand format.

Click to register…

Saville Consulting Wave® – Highest Validity per 15 minutes of test-time!

The Saville Consulting Wave was developed by the originator of the OPQ and co-founder of SHL, Professor Peter Saville as an alternative to the static assessments available in the market. Wave assesses candidate’s motives and talents within a validation-centric framework. It was validated within 100 separate businesses.

The tool uses both normative and dynamic ipsative questions and has been shown to have greater validity in predicting performance and leadership than any tool it has been compared to in research. Furthermore, the shorter version of Wave, the Focus Questionnaire has more validity per 15 minutes of test-taking time than any comparison questionnaire.

Profile jobs, run 360 performance appraisal, assess personality type for team-building, assess entrepreneurial potential and provide extensive development advice all within one framework

The performance culture framework which underlies the Saville Consulting Wave allows the tool to be used extensively for different HR applications from recruitment/selection through to performance appraisal and development.
Follow the links below for further details:

Wave Professional Styles	Performance Card Set
Wave Focus Styles	Wave Performance 360
Wave Types	Wave Development
Wave Entrepreneurial	Wave Culture
Wave Job Profiler

View some sample reports from Wave here

For the early-bird offer (25% discount), please register at http://www.psyasia.com/register
and quote WAVESEB for Singapore courses and WAVEHEB
for Hong Kong Courses. Deadline is 15 May for Singapore and 31 May for Hong
Kong.

Note – delegates on our BPS Level B course in
Singapore always get 50% discount off the Wave Conversion
course which follows their Level B training. More details at http://www.psyasia.com/bpscourses

Course Reviews from Previous Attendees

“Informative, relevant to work, knowledgeable facilitator”

Danny
Hay Group

“Good introduction to the tool and practical session was useful”

Geraldyn Fong
Managing Consultant (Psychologist)

Hudson Global Resources, Singapore

“Insightful and informative. The methodology behind the Saville
Consulting Wave Report is light years ahead of other psychometric tests yet
it is a breeze to use! The interface between motives, competencies and culture
is exactly the missing link that recruiters are looking for.”

Diana Yeo
Ministry of Defence, Singapore

“A highly practical and enjoyable approach to the application
of an extremely useful tool for selection and development – well worth the investment,
highly recommended.”

Neville Pearsall
Director HR
RMIT International University, Vietnam

Not ready for training and accreditation
but still want to use the Wave?

Then please consider PsyAsia’s Psychologist-on-Call™
service instead. Our registered psychologists will take care of the complete process for you, including a call and behavioural interview for your candidates and a feedback session with the decision-maker. More details at http://www.psyasia.com/psychologist_candidate_screening.php.

PsyAsia International offers Free Psychometric Testing Course in Hong Kong & Singapore

Introduction to Psychometric Testing Course: Hong Kong, 4 May 2010; Singapore, 11 May 2010

PsyAsia International is Asia’s independent Leader in Psychometric Test products and Training. We choose to distribute only the world’s best, most validated psychometric assessments and offer locally relevant, world-class training in psychometrics. The Introduction to Psychometrics Workshop expands on PsyAsia’s expertise in Psychometric Training in Asia by offering a course geared to those with very little experience or understanding in Psychometrics. Many first time clients don’t understand why they need to be careful in their choice or use of psychometrics and many do not understand why training is a necessity in competent test use.

This one-day course aims to provide experienced-based training in an accessible and economical way. The course is easy to understand and yet covers many of the important issues to be aware of when choosing and using psychometric tests. Given our passion for Asia and our passion for the competent use of psychometric tests in Asia, PsyAsia makes no profit on this course. We charge delegates a small fee that reflects the cost of the hotel venue (including buffet lunch and refreshments) where the training is held as well as materials that we provide to the delegates. What’s more, if you later decide to attend one of our accreditation courses in Psychometrics, we will issue you with a discount code that reduces your course fee by the amount you paid for this course!

The history of psychometric testing Comparison of psychometric tests with other modes of employee testing and assessment The benefit of using psychometric tests in recruitment/selection, development and coaching Reliability in psychometric testing Validity in psychometric testing Error in psychometric testing Review of different aptitude, personality and values tests on the market Questions to ask your test publisher or distributor What next?Note: During the workshop, delegates will create quasi-psychometric tests in groups to enable a hands-on exploration of issues such as reliability, error and validity in psychometric tests.

To view full course details and to register, please click here.

PsyAsia International is pleased to announce that until the end of March we will be offering free daily webinars to showcase our product range. Their will be no set agenda. The agenda will be set by attendees. Please note however that product knowledge may differ depending on which of our consultants is running the webinar. Come along and chat with our consultants, see the Saville Consulting Wave, Identity Personality Assessment and the Apollo Profile in action. Ask questions about training and consulting options and so forth!

For times and to register, please click here…

The Amazing Apollo Profile

This free webinar will be facilitated by Mr. Jim Bowden, the developer of the Apollo Profile. The session will be interactive (provided attendees kit themselves out with headphones and a mic!) and Jim will present numerous interesting case studies.

The webinar will cover the following:

• Introduction: The Amazing Apollo Profile- can transform Recruitment, Staff Development, and Organisation Performance –Client example
• Apollo Questionnaire – valid/reliable/comprehensive
• Why is Apollo amazing? Apollo Advantages
• Using and interpreting of Apollo reports with anecdotes
• Recruitment – Accurate, easy, low cost – Case Study using Apollo Best Match in China for filtering 12,000 applicants for 40 Graduate level jobs
• Training and Development – Unique Apollo report PLUS downloadable solutions. Convenient, low cost, motivating
• Organisation Development. Benchmarking: Can analyse and identify current corporate strengths and weaknesses – then create high performing models/culture, identify engagement issues – case studies
• Customising: Develop models that work specifically for your organisation. If your organisation is serious about leadership through people.
• Integrate everything together with flexible multi-purpose Internet Online solutions. Use your own competencies frameworks and vocabulary – examples
• Special Offer – have to listen to Webinar to find out!

Date: Monday, May 17, 2010

 Time: 12:30 PM – 1:30 PM SGT

 After registering you will receive a confirmation email containing information about joining the Webinar.

System Requirements

PC-based attendees
Required: Windows® 7, Vista, XP, 2003 Server or 2000

Macintosh®-based attendees
Required: Mac OS® X 10.4.11 (Tiger®) or newer

Space is limited.
Reserve your Webinar seat now at:
https://www1.gotomeeting.com/register/522465752

PsyAsia International is pleased to once again be supporting Singapore’s Human Resource professionals as a sponsor of the Singapore Human Resources Institute’s Annual Human Resource Congress.

The Singapore HR Congress and Business-Connect Exposition 2010 will address the newly derived term of HR TransmutationTM and explore the topic in deeper context.  The current economic churning has made it explicitly clear that industry is not just facing another downturn but it is accompanied by impactful structural, demographic and mindset changes across industry and top management cannot afford to respond with anything less than a complete overhaul of the system to survive and sustain. Renowned speakers and leaders from the HR fraternity will share their experiences and provide useful insights on the know-how of managing paradoxes in a turbulent world.

PsyAsia’s clients are entitled to a 35% discount on the price of conference tickets.  Please contact us in the first instance to avail of this special offer.
“A strong and capable HR community can be the catalyst and change agents to initiate and implement people development efforts in organisations, and help build stronger capabilities amongst our business leaders and managers.”
 
PM Lee Hsien Loong
11th World HR Congress 2006 organised by SHRI

PsyAsia International is Asia’s leading independent distributor of Psychometric Tests of Personality and Aptitude. From offices across Asia, including Singapore and Hong Kong, our psychologists assist the world’s top organisations and local governments to recruit, select, assess and retain the best employees. Our services are only offered by fully registered organisational psychologists with years of experience in their field. PsyAsia also offers world-class training in Psychometric Testing in Singapore, Malaysia, Hong Kong and Online.

Types of Bias in Psychometric Test Translation

With the demand and need for psychological tests increasing in various different cultures and countries, there has been much greater awareness regarding some of the issues that are associated with the development or adaptation of tests to be used in contexts and situations that may be different from which the test was developed for. This article focuses on one of the key aspects of translating tests, the types of bias that can occur.

When utilizing the test in a new cultural group, it is not quite as simple as directly translating the test, administering it and then comparing the results for its validity. There are a number of issues that need to be considered such as whether the area assessed with the test applies to the new culture or whether is may be biased towards that group and whether what is assessed by the test also has similar behavioral indicators? These are just some of the potential areas where bias can be found in the translation of tests and affect the validity of the test being utilized in the new context.

Van der Vijer & Hambleton (1996) differentiates between three distinct types of bias that may affect the validity of tests that have been adapted for different cultural contexts and these are construct bias, method bias and item bias.

Construct bias occurs when the construct (e.g. personality) that is measured by the test displays significant differences between the original culture for which it was developed and the new culture where it is going to be utilized. These differences can occur in the way that the construct was formulated and developed as well as in the relevant behaviors that are associated with the construct. It is critical to examine whether the underlying theory of the test is subject to construct bias and this can be examined through the studies examining the construct and its associated behaviors in the context that it will be utilized in. If there are significant differences found in these studies, it may indicative that there is construct bias. Major revisions may be required to overcome this bias. If not, the validity of the test will be affected.

Method bias refers to factors or issues related to the administration of the test that may affect the validity of the test. Examples of areas that method bias can occur include social desirability, acquiescence response styles, the conditions in which the test was conducted and the motivation of the respondents. Across cultures, there potentially can be differences that can occur in these areas and these can affect the way that the respondents answer the items in the test. This potentially may lead to differences between found that can be erroneously attributed to cultural differences when in fact, these differences are the result of differences in the administration procedures. As a result, it is threat to the validity of tests that have been adapted for use in new cultures. Test developers also not only need to focus on the adaptation of the test itself but also need to be aware of issues regarding the implementation of the test in a new context.

Item bias is another source of bias that can occur in the translation of tests and these refer to biases that occur with the items in the test. This is usually the result of either poor translation choices for items or due to culturally inappropriate translations. For example, the phrase “kick the bucket” is essentially a phrase that referring to passing away in the Western context and is commonly known by most people in that culture; unfortunately, this phrase would have no meaning for people from cultures without any prior experience with that phrase. In this manner, a literal translation of that phrase would be a poor translation as it does not convey the correct meaning of the item. The items in the test need to be culturally equivalent, where the meaning of the items needs to be correctly translated so as to maintain the validity of the test in the new cultural context.

These are some of the biases that may occur during the translation of tests. Test developers will need to be aware of the sources of bias and take the appropriate measures to avoid these biases.

References:

Van der Vijer, F. and Hambleton, R. K. (1996). Translating tests: some practical guidelines. European Psychologist, 1, 89-99.

Psychometric Training in Singapore, Hong Kong, Malaysia, and China
If you are serious about using psychometric tests properly then we recommend joining PsyAsia International’s Psychometric Assessment at Work Course which leads to a certificate of competence in Occupational Testing Level A and Level B from the British Psychological Society. The Course is run publically in Singapore and Hong Kong or in-house anywhere.
More details about BPS Level A and B in Singapore and Hong Kong

Online Psychometric Training – Worldwide
Alternatively, you might be interested in introductory Online Psychometric Test Training presented live by a registered psychologist. PsyAsia is offering a special fee of just US$12 for anybody who registers for the February online psychometric training course!
More details about online psychometric test training

Introduction
PsyAsia International is pleased to announce an exceptional offer for those interested in learning psychometric test administration from the comfort of their office or home.

Why only US$12?
Some months ago PsyAsia International was chosen to become part of the panel of beta-testers for a new online training software. The beta program will end next month as the software is almost ready to go live. In view of this PsyAsia is now ready to use the software as our final stage in the beta program with our valued clients. For this reason, rather than charging the regular fee for our online psychometric test administration training course, we are offering similar training for just a $12 registration fee.

Your facilitator
PsyAsia International is well known for professional and competent training in Psychometrics. This course will be no different. It will be facilitated live by award-winning psychologist Dr. Graham Tyler.

Course Outline
This is a first-level introductory course in psychometric test administration. The course will prepare attendees to become more competent in administering psychometric tests.

-What is a psychometric test
-Types of psychometric tests
-Requirements of good tests
-Psychometric Principles
-Reliability
-Validity
-Running a psychometric test session
-Candidate questions and deviance
-Scoring and norming of tests
-Ethical test use

More offers
Those who register for this online course, pay the US$12 fee and attend all 3 one hour sessions may also register for our 1-day BPS Test Administration Certificate Course in February in Singapore or March in Hong Kong at a 15% discount. Alternatively, you may choose to attend only the afternoon of the above training (which is mainly practical assessments) and pay only 50% of the published course fee. You will need to pay the BPS Assessment fee at the current rate if you wish to be assessed for the BPS certificate. A PsyAsia certificate will be presented to all attendees at the real-life course. Discount codes will be sent to eligible persons after the final training session.

Registration Process
Registration is simple! Click on the link below. Complete the short registration form and ensure that you provide your work email address (free accounts such as yahoo/gmail/hotmail etc will be rejected). Within one working day of completing the form you will receive an email from Paypal with a request for your US$12 payment. Click on the link in the email and pay using your credit card at the paypal site. Within one working day of your payment we will approve your registration and send you a link to download some files that we will use during the training and complete a pre-course quiz. The email will also contain a link for you to log-into each training session. Please pay the registration fee as soon as you can to avoid losing your place. Places are available on a first-paid/first-confirmed basis!!

Click here to register

How the training works
You will need a computer with a reasonable internet connection along with a microphone/headphone set. The training will be transmitted to your computer in slide form and live demo of various systems and websites related to psychometrics. The facilitator will use a mic to teach you. You will also have the opportunity to share with other attendees and ask questions either by way of your microphone or text chat.

Terms and Conditions
The training is open to anybody worldwide who is able to attend on all 3 days between 5pm and 6pm Singapore/Hong Kong time and who can satisfy ALL of the following:

1. Register with accurate data and a work email account
2. Pay the US$12 registration fee
3. Have a computer with an internet connection/headset and mic. However, you may choose to telephone in rather than use a mic/headphone. In this case you must be happy calling a number in USA, New Zealand or Australia. The number will be provided with your registration.
4. Must not be already trained in test administration or BPS Level A – this is because we want to give this opportunity to those who want to learn something new rather than those who may be more interested in finding out how our system works!
5. Must not be a competitor

Click here to register