Putting online psychometrics to the test

Online psychometric testing is gaining ground as a scientific and cost-effective method for measuring a job applicant's suitability - but it also raises a number of concerns. How accurate is it? What are the safeguards against fraud? Will it fall foul of data, race and sex discrimination rules? Keith Rogers looks for some answers.

Rick Woodward, European learning and development director at Kimberly-Clark, is an enthusiastic advocate of online psychometric testing. Having introduced the process in both its graduate and executive recruitment programmes, the company has slashed its cost base, streamlined its administrative processes and improved both internal communications and its feedback to job applicants. Best of all, Woodward argues, the candidates themselves say they like it.

Kimberly-Clark's experiences, based on services provided by SHL, fly in the face of much recent research into the online testing market. To date, HR managers have demonstrated a reluctance to go down the internet route at this stage of the selection process, citing numerous fears ranging from the difficulties of administering tests remotely, to concerns about their equal opportunities responsibilities. Although the internet recruitment market is expanding fast, psychometric testing seems to be running up against both psychological and practical hurdles.

But given how widely the procedure is being deployed, particularly among Times 1,000 companies, acceptance of online testing is becoming a major issue. In a depressed economy where the volume of applications for each job grows, the ability to automate parts of the recruitment process is becoming ever more pressing - particularly as organisations expand the reach of their job advertisements through internet advertising, generating a corresponding increase in candidates. Although the employment market might favour the hirer today, the conditions also create a growing administrative burden that becomes costly and inefficient.

As a pre-interview screening process, psychometric testing is a powerful tool, whether used online or carried out with pen and paper. Used to measure both a candidate's job-related abilities and to assess their personality, it provides a relatively scientific context to the biographical data that's typically used as the basis for the interview process. Organisations such as GlaxoSmithKline have already moved from paper-based tests to running the process on PCs within their assessment centres, so speeding up data analysis and report generation, removing the potential for human error in the administrative process and allowing them to store candidate-related data centrally. In theory, online testing merely takes that process one stage further, retaining all the administrative benefits while allowing candidates to carry out the tests from their own chosen location.

While the theory is good, in practice it raises a number of concerns. To begin with, one advantage of bringing candidates to an assessment centre rather than allowing them to take tests at home is that the employer can control the environment in which they operate. As Mark Evans at GlaxoSmithKline points out: "They are being supervised - it's the same temperature in the room, there's no noise, no distractions - the kids aren't running around, the guy next door isn't using a pneumatic drill. And there's something good about the fact that you can give similar verbal briefings. I think there's something to be said for having the human element there to offer reassurance."

But there's a counterpoint to this argument, according to Dr Colin Selby of Penna Consulting, which provided Glaxo-SmithKline with its PC-based testing. Selby, who is also chairman of a British Psychological Society committee, argues that while assessment centres can provide reassurance, they can also be prejudicial - if the administrator is unfriendly, for example, that may inhibit a candidate's performance. The online testing experience of Woodward at Kimberly-Clark supports that view: "It was a win from the candidates' point of view - they liked it, and said it was better than sitting in an assessment centre with other candidates, and us breathing down their necks."

Managing the test environment is only one aspect of a wider control issue that persistently crops up when HR managers express their reservations about running the process online. John Hackston, managing consultant at test provider OPP, draws a distinction between ability testing, which requires strict controls, and personality tests - but even the latter can present problems online. For one thing, some candidates will try to find out what the "right" answer is - and given the speed with which organisations can provide feedback to applicants, there's little to stop the same person logging in under a variety of pseudonyms to try different approaches. Tests that are supposedly subject to strict time controls can also be hard to police.

There are some technical solutions to these issues. SHL's testing program, for example, drops a Java applet onto the applicant's system that times them out when the test period is complete. The company reinforces that approach by distributing an "honesty contract", which points out that some attributes will be tested again if the candidate progresses to the next stage of assessment and any disparities will be investigated. Others take a more pragmatic approach - Kimberly-Clark, for example, puts no time limit on the tests it carries out online, partly in recognition of the fact that some candidates (like those for whom English is a second language) will require more time to complete the process than others.

Catering for these kind of cultural variances is a further area of concern for potential adopters, both in ethical and legal terms. Will all candidates have access to a PC - and will those that are unfamiliar with the PC environment spend more time working out how to use a mouse than filling in the online form? Is the online process liable to breach any key employment legislation, particularly the Sex Discrimination, Race Relations and Disability Discrimination Acts? What are the implications under the Data Protection Act?

In some areas, the issues are no different whether the tests are carried out online or on paper. As the IRS Employment Review pointed out in January, indirect discrimination can occur purely because of differences in the way that men and women respond to psychometric questionnaires. Guidelines issued by the Commission for Racial Equality advise companies on how to head off these kinds of problems, ensuring, for example, that tests don't contain irrelevant questions in areas that may be unfamiliar to racial minority applicants. As OPP's Hackston points out, organisations should use the same criteria for online testing service providers as they would for any other medium, looking for reliability and validation in the testing procedures. Selby also suggests that companies validate their PC or online procedures with existing employees before they use it on candidates, measuring their 10 highest and lowest performers and ensuring that the psychometric process successfully distinguishes between the two.

Data protection issues are a little more specific to PC-based and online testing, and processing of personal data is strictly regulated. As the IRS report points out, employers must comply with all the data protection principles enshrined in the 1998 Act, including ensuring that personal data is relevant to the purposes for which it is processed, isn't kept longer than necessary and is protected against unauthorised or unlawful processing. In technical terms, that means stored data must be secured against unauthorised outside access, and controlled effectively internally. Some proponents of PC-based or online testing argue that the automatic data collation is a benefit in addressing all of these legal issues, since it allows for rapid collation of data for analysis.

But the issue of candidates' access to and familiarity with PCs is less clear cut. GlaxoSmithKline's Evans, who is keeping an open mind about moving down the online route, says his only concern right now "is the fact that you might be excluding sections of the population. There's a danger of excluding very good candidates by restricting testing to the internet." Ultimately, the decision to go down the online route will be heavily influenced by local environmental factors.

"Access to the web is still an issue for some people," says Selby. "If you want to recruit a nurse in the Philippines, access issues could prevent some candidates applying. You've got to accept that there are inequalities in this respect. If somebody can't be bothered to find access to the web in the US, they're not thought to be serious - that point has been raised in the UK as well. While it's okay for IT engineers - it's not fair for people who want to work in a children's home." Woodward emphasises these cultural differences apply across Europe - while psychometric testing is increasingly common for graduates in Germany, for example, it raises eyebrows when applied to more senior positions.

While these types of objections are all valid, few dispute the fact that online testing offers three core benefits - better speed, accuracy and cost. As well as quicker collation and analysis of electronic data, both PC and online psychometric tests allow for faster dissemination of data within organisations, speeding up the internal administration that accompanies selection and bringing uniformity to the processes. In particular, candidate feedback - which Selby describes as an applicant's "right" - can be dramatically improved.

Administrative accuracy is also enhanced as processes are automated and the re-keying of data is removed. That does not, however, necessarily improve the accuracy of the data itself. While Selby points to BPS research indicating that responses to personal questions tend to be more reliable online than on paper or face-to-face, Woodward's experience at Kimberly-Clark is that the consistency of online responses is lower compared to previous testing exercises. "Because people have got more time, they're thinking 'what's the politically correct answer?'" he says. "One engineer's consistency score was one (against a norm of five to seven) - when I met him I did not believe a word he said."

From a cost perspective, however, the statistics are compelling. While there is a cost associated with training testers to manage a new online environment, Kimberly-Clark has seen its selection overheads drop significantly. The company used to accommodate candidates for two nights in hotels while they went through its assessment process - by running the psychometric tests offsite, the company saved one night's hotel accommodation, amounting to an overhead reduction of some £7,000 a year. The tests themselves are also cheaper. "Every time we did a test with paper and pencil and calculators it was costing us £25: this is costing us £13.50," says Woodward.

But before organisations jump into online testing, they need to take into account several key factors. To begin with, all the major software and service providers insist that psychometric testing should be viewed as just one element of the overall selection process. Steve Newhall, head of business development at DDI, says: "We would not recommend that you use any level of psychometric test as a tool on its own. It needs to be balanced with behavioural information - what they can do and what they've done. We tend to use psychometrics as a way to supplement the other information we get."

He adds: "It's about the robustness of the assessment tools. You take a decision in any other process about the point at which you switch to a human interface. Are we bringing people through to that point, and have we weeded out unsuitable people; gathered data that's going to add significant value to that human piece; and have I managed to get that data more cheaply and efficiently?"

Also, organisations need to ensure that their existing recruitment processes are suitable for the online environment. Roy Davis, head of communications at SHL, points out that the internet is merely the transport mechanism for the testing process - success depends on how effective the underlying HR processes are. "If you've got a bad test to start with," he says, "putting it on the internet won't improve it." Because internet recruitment causes potential overload problems as the organisation's reach expands, Davis suggests that companies should also give candidates an opportunity to opt out of the process, perhaps by being more specific in describing what the advertised job really entails. In that vein, Selby says companies need to be far clearer about the nature of both the job and the personality specifications associated with it before they go down the internet route. It's no longer enough to ask for "leadership potential" - rather, firms should be stressing the nature of the leadership skills required and situations in which they apply.

Right now, the market for online testing remains relatively immature, and uptake is only likely to increase when the number of reference sites expands. The emergence of more sophisticated techniques - such as adaptive testing, where systems change the questions posed to candidates in relation to their response - will also begin to swing the balance. Ultimately, however, it will be the experiences of organisations like Kimberly-Clark that determine whether other users follow suit.

"I've been surprised by the results of some of the research [into online testing]," says Woodward. "We've got a cheap, sophisticated test that we should be using for all executive appointments - it's giving you a quality second interview."

To see how Personnel Today's guinea pigs fared when they took some online psychometric tests, go to Testing the testers.