Psyching out tests

Occupational testing continues to cause controversy on the selection scene, but some employers are putting psychometric tools to much wider use.


Learning points

  • Psychometric testing can help recruiters to build up an accurate profile of a candidate's ability and personality if used with a combination of different selection techniques.

  • Psychometrics are commonly used for training and development activities, particularly for management development plans.

  • The main testing trends are in the speed and delivery of data rather than in test content, influenced greatly by the rise of the internet.

    • Psychometric testing is now a firmly established selection tool for recruiters. Indeed, recent IRS research has revealed that very nearly nine out of 10 employers (88.8%) use some form of occupational testing1. Yet their surge in popularity has done little to allay the reluctance of many practitioners to trust the validity of psychometric tests. The most recent recruitment data from the Chartered Institute of Personnel and Development (CIPD) shows that testing trails far behind the more traditional selection techniques in its perceived effectiveness in finding quality recruits2.

    Interviews top the league table, with one in three (32.5%) recruiters rating them as their most effective selection method, while general ability tests were chosen by a mere 2.9% of employers. Personality questionnaires fare slightly better, at 3.5%, while tests of specific skills are considered the most useful type of selection method, by 7.1% of respondents.

    This contradictory evidence needs explaining. There has always been - and remains - controversy surrounding the use of psychometric testing, particularly personality assessments. The fear that tests can somehow be faked, and a wariness on the part of some recruiters to lend too much weight to a psychological evaluation, partly accounts for their lack of influence in selection decisions.

    Stories of their misuse, such as dismissing an individual on the basis of test results alone or using the data as part of the selection criteria for redundancy, have done little to ease employers' concerns. There are also considerable risks - as is the case for any selection tool - attached to the administration and interpretation of psychological tests that most practitioners are well aware of. At the same time, there is mounting pressure on organisations to develop more sophisticated methods to recruit staff, and so testing is now firmly positioned as another string to the selector's bow.

    Moreover, a job interview represents an indispensable part of the selection process, and its high score attests to this fact. Test results may be contributory factors in shaping selectors' decisions, but both interviewees and interviewers tend to place more store by the face-to-face encounter of the formal interview.

    Testing, testing

    Psychometric testing has been around for at least a century, although not always in a work setting. It was in the 1970s that occupational testing became more mainstream, chiefly in the USA. Since then, its use as a management tool has steadily increased, and the UK market of tests and suppliers has burgeoned in parallel to meet the demand.

    So what exactly is psychological testing? The CIPD provides the following definition in the guide in its Quick Facts series: "Psychological tests are tests which can be systematically scored and administered, which are used to measure individual differences (for example, in personality, aptitude, ability, attainment or intelligence). They are supported by a body of evidence and statistical data which demonstrate their validity and are used in an occupational setting."3

    Testing can be broadly divided into two camps. One type involves an assessment of an individual's ability or aptitude, while the other measures personality traits. There is a subtle difference between ability and aptitude tests.

    The former assesses an individual's current performance, while the latter gauges that person's future potential to perform with the benefit of adequate training. There are also different categories of ability tests.

    General ability tests measure the overall level of intelligence as a general trait, usually by using critical, non-verbal or abstract reasoning. There are also specific ability tests, such as verbal, numerical and spatial reasoning. And there are those used for particular roles, such as administrative, clerical or call centre staff. A further option involves bespoke testing, which tailors the content of the assessment instruments to an organisation's needs.

    Typically, a battery of ability and aptitude tests is used in selection. Here, the data collected from a range of exercises build up a picture of the candidate's ability - and suitability - for the post.

    Personality tests

    It is personality testing that is at the core of much of the controversy surrounding psychological assessment. Often, it is used in conjunction with ability testing to give a more rounded profile of the candidate. There are many questionnaires on the market that measure either personality type or traits that are intended to match individuals to jobs that suit their character, such as the Myers Briggs Type Indicator (MBTI), used for development purposes by Dixons (see box 1).

    Certain qualities have become known as the "big five" common traits, which are said to be at the heart of all personalities. They are:

  • adjustment/neuroticism - single-facet traits that determine behaviour and include anxiety, hostility, depression, self-consciousness, impulsive behaviour and vulnerability;

  • agreeableness - trust, frankness, altruism, compliance, modesty and compassion;

  • conscientiousness - competence, order, sense of duty, striving for achievement, self-discipline and deliberation;

  • extroversion/introversion - warmth, gregariousness, assertiveness, excitement-seeking, activity, and positive emotions; and

  • openness to experience - fantasy, aesthetics, feelings, actions, ideas and values.

    There are two main approaches to designing personality questionnaires: the normative and the ipsative method.

    The normative approach compares an individual's results against the relevant norm groups - these comprise other individuals whose scores have been obtained by the test developers and are used as a yardstick when results are analysed. The relevance of the norm group is crucially important for potential _test-users, given the potential for differences based on gender, ethnicity, nationality and other factors.

    Ipsative testing measures individual characteristics relative to one another, without making external comparisons.

    Questionnaires typically contain a series of multiple-choice questions, to which there are no "wrong" or "right" answers. The use of personality questionnaires is more prevalent in managerial selection and development activities. Testing is time-consuming and expensive, and it is in such senior-level roles that these requirements can be more easily justified.

    Selection

    There is no doubt that psychometric testing is now accepted as a valid and effective selection tool by most employers (see box 2 for its use at Government Communications Headquarters (GCHQ)). This year's CIPD recruitment survey showed that well over 40% of recruiters use each of the four main psychometric categories of testing as part of their selection processes - that is, tests of specific skills, literacy and/or numeracy tests, general ability tests and personality questionnaires.

    The survey found, too, that in the majority of cases, respondents' use of testing has increased. For example, the percentage of employers using general ability tests increased from 31.9% in 2002 to 44.9% this year. However, some of this change may reflect differences in the survey composition.

    Ability and aptitude testing is not designed so that organisations can recruit those candidates with the highest scores - rather, that they should be used to ensure that a minimum standard is attained. Selecting the best person for the job involves a much wider consideration of an individual's suitability. This is why psychometric testing should never be used in isolation and, as all the main test-providers will attest, it is much more effective when it represents one piece of the selection jigsaw.

    Choosing the right test

    There is a plethora of different types of tests and products on the market, and a considerable number of potential suppliers. Selecting an appropriate test can be a daunting prospect, but this is the first stepping-stone in developing a robust assessment process.

    Ian Florance is an independent consultant for psychometric test provider and developer ASE. In his view, a key piece of advice for selectors before choosing a specific test is to be clear about why they are using it.

    "It sounds obvious, but it is vital that managers define exactly what qualities they are looking for in a person before they select the test," he advises. "A job analysis therefore needs to be undertaken or the test results won't mean very much. Tests measure specific areas very accurately, but they should not be considered in isolation, as is the case with any selection tool. To improve the validity of the appointment decision, the test should be supplemented with other evidence gathered through the selection process, for instance from a face-to-face meeting or CV."

    Another tip is to make the link between selection and development at the recruitment stage. As Florance comments: "An employer knows the most about an employee at their point of entry to the organisation. As it is generally in the first few months of employment that an individual will exert the greatest effort, a development plan should be put in place immediately."

    Robert McHenry, chair of psychometric test publisher OPP, says that, of the hundreds of assessment products on the market, only a handful stand out in terms of their reliability and validity. If in doubt, the best advice for HR practitioners is to approach a reputable supplier.

    "Some specific questionnaires have been around for 40 or 50 years, and have been carefully and skilfully developed at a cost of millions of pounds to reach their present standard," he says. "The Myers-Briggs Type Indicator that we supply, for example, took over 30 years to produce. It is regularly updated and is adapted for global use to be relevant to local languages and cultures."

    OPP promotes good practice in the use of its psychometric instruments, and all organisations using the MBTI, for example, are asked to sign a code of ethics every three years. As McHenry comments: "It is greatly in the public interest that all users are properly qualified, technically competent and promote good practice in how the tools are administered."

    Best practice

    Both the CIPD and the British Psychological Society (BPS) offer best-practice guidelines for employers to help them maintain a fair and ethical approach when conducting selection testing. The main test suppliers also offer training and best-practice advice as part of the package when organisations purchase their evaluation tools. Indeed, many providers insist that all test users are qualified to administer the tests as a condition of their sale.

    The BPS has also this year launched a new Psychological Testing Centre website (www.psychtesting.org.uk ) that aims to be an authoritative source of information on occupational testing for test-users, test-takers and for those developing psychometric tools. In addition, several of the big providers, including PSL and OTL, have now formed a new trade body - the Business Test Publishers Association (BTPA) - which also aims to promote best practice in the use of testing.

    The CIPD notes in its Quick Facts guide that there are a number of issues for employers to consider if best value for money is to be achieved from the use of testing, including:

  • "Feedback - feedback [on an individual's test results] should always be given unless there are good reasons why this should not be the case. Before a test is carried out, candidates should always be informed about the arrangements for giving feedback.

  • Openness - candidates should always be aware that tests will be conducted and the procedure should be explained in advance.

  • Confidentiality - access to test results should be restricted to those with a genuine need to know. Candidates should be informed as to who will have access to their test results.

  • Screening - a single test should never be used for screening purposes although a battery of tests may be appropriate in, for example, shortlisting of applicants for employment. Personality tests should never be used for screening purposes.

  • Norms - only tests for which appropriate norm tables are available should be used."

    That those individuals administrating tests and interpreting the results are properly qualified to do so is of fundamental importance. Aside from helping to ensure the veracity of the data, trained users should have the knowledge and experience to avoid discrimination and unfairness creeping into the testing process.

    The BPS offers both Level A and Level B qualifications, which are well-recognised standards within the testing industry. More generally, the BPS has a code of good practice4 that reiterates the guideline that tests should only be used in conjunction with other assessment methods. It also advises test-users to administer, score and interpret tests in accordance with the instructions provided by the test distributor and to the standards defined by the BPS.

    Development

    The use of psychometric tools for development activities is often overlooked, as most of the debate about testing is usually restricted to its application during the selection process. However, occupational tests have for some time commonly been used in this wider organisational context. Although it is less straightforward to generalise about their use in development activities as employers can apply the tools in a range of different ways, they could be used for personal development plans, succession planning, team development and careers guidance.

    Ian Florance believes that testing for developmental purposes has assumed even greater importance in recent years. "Testing has been used in the development field for many years, but many companies have realised since the 1980s that they discarded valuable knowledge and expertise when they failed to develop and retain key employees," he says.

    "A psychometric test, which can provide an objective assessment of an individual's strengths and weaknesses, can really help managers to manage people better. Individuals are usually very open to undertaking tests for development and career purposes, as they can find out some very interesting things about themselves."

    Florance points to recent research indicating low levels of motivation among UK employees, around 60% of whom would not recommend their firm as a place to work. Again, psychometric testing can play a useful role in identifying what motivates people. "Motivation is not just about providing financial incentives; in fact, these can demotivate certain individuals," he explains. "Testing can objectively get under the skin of individuals and help individuals understand each other better at work."

    Roy Davis, head of communications at test developer and provider SHL, also points out that psychometric tests are now increasingly used in training and development and performance management activities:

    "As the UK climbs out of recession and recruitment tightens, it is even more important that organisations develop their employees. We now do a significant amount of work with _blue-chip companies who have recognised the need to change the behaviour of people within their own organisation. Tools such as 360-degree instruments and personality tests can help organisations to identify potential, and coach and develop their own talent pool," he says.

    Internet delivery

    One of the most talked-about developments on the psychometric scene is the current and future potential of online testing.

    According to the CIPD's 2003 recruitment survey, organisations' use of the internet to administer tests is still relatively modest, with just 6.2% of respondents opting for this method of delivery. According to Robert McHenry of OPP, the online use of psychometric tools has not increased at quite the speed that was envisaged.

    "At the peak of the dotcom boom four years ago, it was forecast that almost all use of psychometric instruments would take place on the web," he comments. "In reality, it is less than 20%, although the percentage is increasing every year. The internet has opened up the potential for international recruitment and introduced very flexible options for remote use." OPP provides one system in more than 10 languages for this purpose. The product enables a test to be administered electronically in one language and generate a report in another language if necessary.

    "Without doubt, the future of psychometric assessment is online," says SHL's Roy Davis. "SHL undertakes around one million tests a year this way. The internet offers huge potential in cost savings and efficiencies to the client organisation. Testing is a labour-intensive process, traditionally undertaken at the end of the recruitment process. The internet enables recruiters to introduce a rigorous psychometric questionnaire much earlier on in selection, thereby producing a more suitable pool of candidates at the first stage. The final selection phase is therefore a hiring rather than a screening event."

    The growth in online testing has also intensified the debate about some more general, but important, issues concerned with using tests, such as quality and the development of a kitemark for the industry. "The International Testing Committee is looking at this matter at the moment, and it will only be a matter of time before a kitemark that guarantees the standard of a test is available," comments Roy Davis.

    Another interesting trend taking place in online usage is the development of simulation exercises. "Questionnaire design has probably reached the limit in forming clever questions, but, with simulations an individual feels like they are actually performing the job," explains Robert McHenry. "Some tools have already been developed, such as one developed for a private client by Stephen Blinkhorn that generates telephone calls by computer, in a call centre environment. These are more realistic and can be effective when used as part of an assessment centre."

    McHenry is cautious about the degree to which online use of psychometric tools will take off in the development field. "Particularly in today's economic climate, the majority of tools are used for development and leadership purposes, and 90% of these are paper-based," he explains. "The element of human interaction involved in administering an evaluation for personal development is important, and could not easily be reproduced by using a computer-based package."

    In Ian Florance's view, there are many potential savings and opportunities arising from online testing, but there are also risks involved. "There is nothing wrong with computer-based systems for testing in themselves, but there is a huge amount of unregulated material on the web," he says.

    "There are also some fundamental issues about how the testing is carried out. Psychometric testing is perfectly acceptable as a sifting mechanism, as tests are at their most effective when a large volume of candidates needs to be reduced to a much smaller pool. However, there are real concerns about the accuracy of very brief personality 'quizzes'. There is a direct correlation - up to a point - between the length of the test and its reliability."

    Evaluation

    As is the case with every recruitment and selection technique, evaluation is the key to assessing whether testing is a successful measure of future job performance.

    Validation does not need to be an onerous activity: even gathering informal data from line managers on the perceived effectiveness of the instruments being used is a step in the right direction. It would also not be a difficult exercise to track the progress of new recruits and their respective test scores and compare them with the performance and retention of other employees.

    In some cases, there is also the wider organisational role that the assessment instrument plays to consider. For example, in our Dixons case study the use of the MBTI is constantly reviewed and refreshed in order to maximise its benefit to the company.

    McHenry believes that the evaluation of psychometric tools is on the increase in many organisations, and is a very necessary activity to justify their use in both selection and development. "Users should think ahead to determine whether the instrument used has been successful," he says. "The most straightforward approach is to canvass opinion from the individuals who were involved, supervisors of the process and customers. In the case of assessment for selection, it is possible to track the retention and performance of employees to evaluate the success of the technique."

    The future

    Apart from the significant changes in delivery heralded by the internet, the other main development in testing has been in the area of outputs. As Roy Davis comments: "We have reached the stage with the content of tests where we are measuring what we need to measure. The question now is how can we get cleverer at what the results are telling us about people? For example, some of the outputs about personality can help organisations understand an individual's preferred learning style."

    Another major development on the psychometric scene, in Robert McHenry's view, is speed. "Advances in technology mean that companies expect results almost immediately, but too much speed could endanger the validity of the testing," he says. "There is a temptation to shorten the questionnaires or use only a part of them to hasten the process. This should generally be avoided."

    Setting aside discussion about the most avant garde technological developments, one of the main messages for practitioners to take away from the continuing debate on psychometrics is this: given the right circumstances and when not used in isolation, occupational testing can be a valid and effective selection tool. And its use need not be confined to selection.

    Many organisations are broadening the application of psychometric tools into other occupational fields, such as employee development, managerial decision-making and internal communication. Viewed from the perspective of a growing interest in the potential value of psychology in a workplace setting, organisations could benefit from seriously considering a less traditional approach to testing.

    1. "Testing times for selectors ", IRS Employment Review 769, 7 February 2003.

    2. Recruitment and retention 2003, Chartered Institute of Personnel and Development, www.cipd.co.uk .

    3. "Psychological testing", Quick Facts, Chartered Institute of Personnel and Development, www.cipd.co.uk .

    4. "Code of good practice for psychological testing", British Psychological Society, 2002, free from www.psychtesting.org.uk .

    This article was written by Rachel Suff, a freelance employment researcher and writer, rmsuff@dsl.pipex.com.


    1. Case Study: Dixons Fast-Track Training

    The Dixons Group employs around 31,000 people at its head office and in its chain of electrical stores. Management development is a strategic priority for the company, and it has put in place a number of programmes to nurture in-house talent, one of which is the fast-track executive development programme (EDP).

    In the most recent intake to the programme, around 50 middle managers took part in the selection process and 20 were accepted onto the EDP. "The EDP is designed to identify managers with high potential from the middle-manager population, who may one day sit on the board," explains Nicole Denham, training and development manager. "Psychometric testing forms an important part of the assessment process, and the tools have been very useful in other development areas, for instance exploring the concept of a Dixons 'type', teambuilding and understanding each other better."

    The first stage of selection for the EDP involves a self-selection questionnaire, where would-be participants are able to answer questions online via the company's intranet to assess their own readiness and suitability for the programme. The next stages are an application form that draws on the company's leadership competency framework and an online case study. An assessment centre follows, which includes alpha-numeric exercises, abstract reasoning and the Myers Briggs Type Indicator (MBTI), as well as other activities, such as individual presentations and group events.

    "The psychometric testing takes place in advance of the interview, and so provides us with valuable data on areas to probe further with delegates," explains Denham. "At this stage, it is important that we brief and coach the other members of the assessment panel who have not been trained in the use of these instruments." The company always gives feedback to individuals on the test results at the end of the process, when they have an opportunity to ask questions about the data and its interpretation. "And if an individual is not accepted onto the EDP, the test results are still used in a positive way, for that person's future development plan," she adds.

    "We have found the MBTI very powerful as a development instrument," she says. "It is simpler than some tests; there are no right or wrong answers or percentile scores; and it therefore has high face-validity with employees. Because we are branching out and realising the potential of the test in a wider organisational context, such as communication and change management, line managers are seeing the links and becoming even more comfortable with its application."

    The MBTI is provided by its sole supplier, OPP, whose staff provide training to levels A and B for the HR team to administer the tests and interpret the results. Dixons therefore has an ongoing relationship with the test-provider that, in Denham's view, is very valuable. "OPP is willing to provide additional feedback and coaching on using the test," she explains. "It is important, from our perspective, that we keep up to date with the tools and pick up fresh ideas on how to apply them within the organisation."


    2. Case Study: GCHQ Tests to Select

    Government Communications Headquarters (GCHQ) employs around 4,000 staff who study international communications for intelligence purposes and protect UK communications from hostile attack. The agency recruits between 200 and 300 people annually to its civil service operation. This involves running up to 30 different recruitment campaigns that could take on between one and 60 employees in any one round.

    GCHQ uses a number of different psychometric tests, in a number of different ways, as part of its selection process. Assessment centres are used for around 40% of new employees and are used at two levels: for graduate-level recruitment and for internal promotion to some management grades. For this type of recruitment, psychometric tools are used in the final stages of selection and are based on the organisation's competency framework, which includes seven core competencies, such as analytical decision-making, leadership and teamwork.

    "For graduate-level entry, the assessment event starts with a matrix of various different tests including verbal and numerical reasoning that evaluates candidates' analytical skills," explains Alan Graham, head of recruitment and marketing. "After another series of exercises, candidates complete a personality questionnaire and have an informal one-to-one discussion with an occupational psychologist. Considering the line of work individuals are applying for, it is very important that any potential areas for concern are brought into the open."

    Testing is an integral part of GCHQ's assessment process for graduate recruitment, with the results of each selection technique helping selectors to form a rounded opinion of each candidate. "We build up a picture with reference to all the indicators," explains Graham. "We then use the information to make a final decision and would never use any one set of data to fail a candidate." The agency retains its own pool of specially trained assessors to administer the tools and has an ongoing relationship with the same occupational psychologist, who always analyses the test results and oversees the process. The psychometric instruments are developed against norm groups appropriate to the organisation's profile.

    Occupational testing is also used as an initial screening exercise in GCHQ's recruitment tonon-graduate administrative roles. "If an applicant meets the set qualifying criteria, having undertaken the evaluation, they will be interviewed," says Graham. "We advertise locally for these positions but still receive a high number of applications, so the test enables us to whittle the pool down to a manageable number of candidates." GCHQ may extend this screening exercise to other recruitment campaigns where the volume of applications is increasing. As Graham comments: "Sifting through CVs is a very labour-intensive process. Using an appropriate psychometric tool could be a much faster, more effective approach."

    Graham believes that psychological testing has an important role to play in selecting staff to its intelligence and security operation, but points out that the tools are not infallible. "In the case of our administrative recruitment, it is usually possible to see the correlation between the testing and the quality of the candidates," he advises. "However, there was one example that showed an anomaly and it is therefore crucial that a combination of methodologies are used for selection."


    3. The law and testing

    Practitioners need to take great care when administering tests that they do not discriminate against certain groups, and that they operate within the confines of the Data Protection Act 1998 and its Code of Practice.

    Discrimination

    The Sex Discrimination Act 1975 and Race Relations Act 1976 outlaw discrimination at every stage of the employment process. Testing is a potentially vulnerable area of selection, particularly in the case of gender differences. Indirect sex discrimination can slip into the process because men and women may have different responses to some of the psychometric questions.

    The Equal Opportunities Commission's Code of Practice on Sex Discrimination, Equal Opportunity Policies, Procedures and Practices in Employment1, recommends that tests should be relevant to job and/or career requirements and should assess a person's actual or inherent ability to perform the role. It advises that psychometric instruments should be reviewed and updated on a regular basis to ensure that they remain appropriate and free from unjustifiable bias, either in content or scoring methodology.

    The Commission for Racial Equality also has valuable advice for test users in its Code of Practice for the Elimination of Racial Discrimination and the Promotion of Equality of Opportunity in Employment2. As well as general advice on developing and applying an equal opportunities policy to avoid discrimination on racial grounds throughout the recruitment and selection process, it recommends that selection criteria and the use of tests should be firmly related and restricted to job requirements. For instance, "selection tests which contain irrelevant questions or exercises on matters which may be unfamiliar to racial minority applicants should not be used (for example, general knowledge questions on matters more likely to be familiar to indigenous applicants)."

    Under the Disability Discrimination Act 1995, employers have a responsibility to avoid less favourable treatment of people with disabilities, and to make "reasonable adjustments" to accommodate them during the recruitment and selection process. A job applicant with disabilities could very easily be disadvantaged and therefore discriminated against in an occupational test. For example, this could occur where a partially-sighted person is expected to complete an abstract reasoning exercise, or where someone with a hearing impairment is required to undertake a verbal reasoning test.

    Data protection

    All eight data protection principles set out in the Data Protection Act 1998 need to be adhered to when employers engage in psychometric testing. For example, personal data should be processed fairly and lawfully, obtained and processed for limited purposes and should not be kept for longer than necessary. It is unlikely that analysis carried out by unqualified users would be regarded as fair processing of personal data. The testing also needs to be highly relevant to the recruitment process in order to be viewed as fair. Employers should ensure that they ask candidates for their informed consent prior to the testing process taking place.

    In addition, part one of the Employment Practices Data Protection Code 20023 contains specific recommendations about the use of testing:

    "1. Be consistent in the way personal data are used in the process of shortlisting candidates for a particular position.

    2. Inform applicants if an automated shortlisting system will be used as the sole basis for making a decision. Make provisions to consider representations from applicants about this to take these into account before making the final decision.

    3. Ensure that tests based on the interpretation of scientific evidence, such as psychological tests and handwriting analysis, are only used and interpreted by those who have received appropriate training."

    1 Code of Practice on Sex Discrimination: Equal Opportunity Policies, Procedures and Practices in Employment, Equal Opportunities Commission, 2000, www.eoc.org.uk .

    2 Code of Practice for the Elimination of Racial Discrimination and the Promotion of Equality of Opportunity in Employment, Commission for Racial Equality, 1984, www.cre.gov.uk .

    3 Employment Practices Data Protection Code 2002, part one, www.dataprotection.gov.uk. Part one of the Code is being revised, and the Code will not be implemented until all four parts have been issued.