Measuring up

The announcement of national targets for improvements in health and safety has given measurement of performance a new emphasis. Chris Dyer examines some of the advice available.

In 2000, the government and the HSC launched the Revitalising health and safety strategy (Employers face major health and safety at work shake-up) with the aim of improving health and safety performance in Britain. At the heart of Revitalising are national targets for reductions in work-related injury, illness and absence that are intended to provide an aspirational standard against which progress can be measured.

In December 2002, the HSC issued a progress report on the targets, covering the first two years of the strategy (see box 1). The indicator of the rate of fatal and major injury shows no change, and baseline figures for absence and ill health are only just being established.

Commenting on the report, HSC chair Bill Callaghan said that it was early days in terms of establishing benchmarks and measuring progress against the 10-year targets. But he added: "The fact remains that the overall fatal and major injury rate has been static for a number of years. We need to secure lasting improvements across the board to break through this plateau and achieve a real step change." Improving the incidence of ill health presents an even greater challenge.

Although the government set the targets, achieving the targets is dependent on all stakeholders in the health and safety system. The HSC's Strategic plan contains supporting sectoral and risk-specific targets. The HSC stresses that none of these targets can be met without the commitment of stakeholders, who aspire to the targets and contribute to their delivery, for example by devising and publishing their own supporting targets.

Holistic approach

Health and safety performance is measured to gain information on the effectiveness of control of risks to health and safety. But occupational health and safety performance is composed of many factors, and no single measure, such as lost-time injury, can provide an overriding indication of an organisation's success or failure in managing work-related risk. The headline Revitalising targets represent a worthwhile reduction in the level of failure in Britain's occupational safety and health performance, but are not directly relevant to most businesses.

There are 3.7 million enterprises in the UK, of which 99% have fewer than 50 employees. Among this group, it will be rare for a business to experience an event reportable under the Reporting of Injuries, Diseases and Dangerous Occurrences Regulations 1995. Furthermore, the average interval between injuries in these companies will be long. Targets based on lost-time injury and ill-health trends have little meaning within small firms and even within individual business units that are part of large organisations. Any organisation that relies solely on these measures of performance can easily be misled as to its true situation (see box 2).

Both the HSE1 and the Royal Society for the Prevention of Accidents (RoSPA)2 have recently published guidance on the measurement of health and safety management in organisations, including the use of targets. The RoSPA guidance explores good practice in setting corporate health and safety improvement targets. It is intended to help organisations think through their approach to setting health and safety improvement targets for their organisation as a whole and at each subsidiary level, focusing on the need to take a holistic view of performance, establish an evidenced based approach and to develop and agree targets with key internal and external stakeholders, including safety representatives.

The HSE guidance covers similar ground but with greater emphasis on the measurement process. It also stresses a holistic approach: "There is no single reliable measure of health and safety performance. What is required is a 'basket' of measures or a 'balanced scorecard', providing information on a range of health and safety activities."

The HSE's experience is that organisations find health and safety performance measurement difficult; the guidance contains references to sources of further information including tools and techniques.

Plan-do-check-act

The basis of the HSE's guidance on safety management systems, HSG 65, is the continuous "plan-do-check-act" management cycle:

  • plan: establish standards for health and safety management based on risk assessment and legal requirements;

  • do: implement plans to achieve objectives and standards;

  • check: measure progress with plans and compliance with standards; and

  • act: review against objectives and standards and take appropriate corrective action.

    Measurement is an integral part of the cycle and should:

  • provide information on how the system operates in practice;

  • identify areas where remedial action is required;

  • provide a basis for continual improvement; and

  • provide feedback and motivation.

    A holistic approach to measurement uses a portfolio of evidence comprising indicators of input, output and outcome. If the target is to achieve an outcome of no injuries or work-related ill health, risks need to be controlled. The HSE model of risk control envisages uncontrolled hazards as the input, which are processed through the health and safety management system, supported by a positive health and safety culture, from which the output is controlled hazards and risks, and the outcome is no incidents, injuries or ill health (see figure 1 below).

    XpertHR article image

    To answer the question "What is our health and safety performance?", the HSE recommends that performance measurement should cover all the elements of the system:

  • Input is measured by monitoring the scale, nature and distribution of hazards created by the organisation's activities, ie the "hazard burden". Measuring the hazard burden gives an important input into the planning and review processes, enabling proportionate effort, priority and emphasis to be given to different hazards. Measuring the hazard burden will be a largely qualitative process answering questions such as "What are the hazards associated with our activities?" and "How does the nature and significance of the hazards vary over time?"

  • Process is measured by the active monitoring of the adequacy, development, implementation and deployment of the health and safety management system and the activities that promote a positive health and safety culture - these are measures of success. The health and safety management system is the process that turns uncontrolled hazards into controlled risks. The HSE model health and safety management system has five stages, described in detail in HSG 65: policy; organising; planning and implementation; measuring performance; and audit and review. The health and safety culture, which supports the health and safety management system, can be measured using tools that canvass the views of employees on important aspects of health and safety within their organisation. The HSE has produced and sells its own tool to do this (HSB 315).

  • Outcomes are measured by reactive monitoring of adverse outcomes resulting in injuries, ill health, loss and accidents with the potential to cause injuries, ill health or loss-these are measures of failures.

    MANAGEMENT SYSTEM

    The HSE model health and safety management system envisages three levels of control:

  • Level 1: the key elements of the health and safety management system; the management arrangements (including plans and objectives) necessary to organise, plan, control and monitor the design and implementation of risk control systems (RCSs).

  • Level 2: RCSs; the basis for ensuring that adequate workplace precautions are provided and maintained.

  • Level 3: effective workplace precautions provided and maintained to prevent harm to people at the point of risk.

    A positive health and safety culture supports each level of the system. Measuring arrangements for levels one and two need to cover three aspects:

  • capability;

  • compliance (implementation); and

  • deployment.

    Capability

    Measuring capability is about checking that particular management arrangements, such as the accident investigation system, or RCSs, such as the system to control contractors, are capable of delivering the required outcome and are fit for purpose. There are two aspects to consider: whether there is a system in place and whether the system is technically adequate for the required application.

    Establishing that a system is in place means checking that there is a plan-do-check-act process, so that:

  • it is clear what the system is intended to deliver;

  • individuals within the system have clear responsibilities and are held accountable for them;

  • the competencies of people operating the system are defined;

  • the people who operate the system have had input into its design;

  • there are procedures that define how the system is implemented and the performance standards expected;

  • there are arrangements for reviewing the design and operation of the system and taking appropriate action to correct deficiencies and for continual improvement; and

  • adequate resources are provided to operate the system effectively.

    Having the plan-do-check-act elements will not ensure that a system is technically adequate. For example, an accident investigation system will be of limited value if it does not identify the root causes of the accident. The yardsticks for checking technical adequacy are legal requirements and best practice, including the consideration of human factors.

    Compliance

    Management systems and RCSs will only deliver the desired outcome if they are complied with, so performance measurement must provide information on levels of compliance. Employees must understand how the particular management arrangement or RCS is expected to operate, and this can be illustrated using a process flowchart.

    This can also be used to decide what aspects of the process need to be measured to check that the process is being implemented as intended: the basis of these checks are performance standards that define who does what, when and with what result.

    In larger organisations, management will need a measure to check that the elements of the health and safety management system are deployed and operating across the whole organisation and not just in isolated parts. This should include information on levels of compliance with the particular management arrangement or risk control system across various parts of the organisation.

    Workplace precautions

    The output of RCSs are suitable workplace precautions (level 3 in the health and safety management system) in place at the point of risk. Risk assessment is central to measuring workplace precautions; it defines the precautions needed to control particular risks that must be included in the RCS. Measurement here means comparing "as is" against a defined standard or yardstick (compliance) to show whether workplace precautions are in place, operating and effective. Monitoring compliance with defined workplace precautions is the major element in day-to-day measurement activity.

    TAKING MEASURES

    "Targets", often used interchangeably with words such as "aims", "goals" and "objectives" should be "smart". That is to say:

  • specific;

  • measurable;

  • agreed with those who must deliver them;

  • realistic/relevant; and

  • time-bound by a suitable time-scale.

    The first check in the measurement process is whether plans and targets meet this test. Measurement of performance needs to be continuous, but if it is to be efficient and effective the frequency of measurement needs to be planned (see box 3). Each target needs a specific timescale for achievement, but it is also important that progress is subject to continuous monitoring and review. It may be necessary to modify targets in the light of feedback from monitoring and review, particularly if they are not achieving their objectives or are having a distorting effect on behaviour.

    Checks need to take place at levels within the organisation that reflect its hierarchical structure. Individuals within the organisation will need information appropriate to their position and responsibilities within the health and safety management system, and individual measurements and targets need to operate within an overall framework so that there is a hierarchical set of linked measures and targets that reflect the organisation's structure. Performance needs to be measured at each management level, starting with the most senior. Managers must not assume that everything is working as it should just because no problems or deficiencies have been bought to their attention (see box 4).

    Effectively measuring inputs, outputs and outcomes poses practical challenges. Some can be measured by continuous whole population monitoring, such as accidents and routine health surveillance; some only by using sampling techniques, such as health and safety management system audits and safety climate surveys. Some can be measured through routine monitoring, such as planned, periodic inspections or monitoring levels of workplace contaminants.

    Measurement should be based on specifications and performance standards against which measures can be taken. Measurement may mean:

  • direct observation of conditions and peoples' behaviour, including inspections and monitoring of the work environment; and/or

  • talking to people to elicit facts and their experiences and to gauge their views and opinions; and/or

  • examining written reports, documents and records.

    Direct observation may be made easier by using checklists, which should be designed so that they require objective rather than subjective judgements of conditions. They should facilitate:

  • the planning and initiation of remedial action, by requiring those doing the inspection to rank deficiencies in order of importance;

  • taking remedial actions, with names and timescales to track progress to implement improvements;

  • periodic analysis to identify common features or trends that might reveal underlying weaknesses in the system; and

  • information to aid judgements about changes in the frequency or nature of the monitoring arrangements.

    Checklists can be structured using the "four Ps": premises, plant and substances, procedures and people (see box 5), which are the areas concerned with work activities and risk creation.

    Developing a system

    The HSE has identified what it considers the nine most important steps in developing a health and safety measurement system.

    1. Identify the key processes

    These will be the management arrangements, RCSs and workplace precautions.

    2. Analyse the key management arrangements and RCSs to produce a process map or flow chart

    It is vitally important to understand how the process actually operates on the ground, so this step must involve those responsible for implementing it.

    3. Identify the critical measures for each management arrangement and RCS

    This should consider:

  • What outcome do we want?

  • When do we want it?

  • How would we know if we achieved the desired outcome?

  • What are people expected to do?

  • What do they need to be able to do it?

  • When should they do it?

  • What result should it produce?

  • How would we know that people are doing what they should be doing?

    Again, it is important to include the people involved in implementing the arrangements and systems when deciding what the critical measures might be.

    The measures derived should be:

  • accepted by, and meaningful to, those involved in the activities being measured and those who need to use the measures;

  • simple/understandable/repeatable/objective;

  • capable of showing trends;

  • unambiguously defined;

  • cost-effective in terms of data collection;

  • timely;

  • sensitive; and

  • should drive appropriate behaviour.

    This last point is important because the choice of measures can promote behaviours that are in conflict with the desired outcomes. For example, rewarding low accident rates (in the absence of other measures of performance) can lead to under-reporting.

    4. Establish baselines for each measure

    Once the individual measures have been established, baseline data need to be established. Problems with baselines can arise as awareness of health and safety issues increase. For example, reporting rates for "near misses" may improve so that the figures give the impression that the situation is getting worse when, in reality, things are getting better. Whatever indicators are chosen as a basis for target setting, it is important to ensure that, as far as possible, current baselines are an accurate reflection of existing status.

    5. Establish goals or targets for each measure

    Again, this should be done by involving the people who are expected to operate the particular activity. Target setting has to be based on good data, robust analysis and a sound understanding of the processes through which improved risk management can be achieved. If targets appear to be arbitrary, not only will they lack transparency, meaning and credibility, but they will not secure workforce and management commitment.

    Targets that merely assert unrealistic goals -but without indicating why they have been chosen or the means by which they were set or can be achieved - serve no useful purpose and need to be challenged.

    Rather than focus solely on outcomes, organisations need to look at output indicators, such as:

  • reductions in exposure to harmful agents in the work environment, eg airborne contaminants, noise, radiation; and

  • reductions in exposure to harmful burdens - both physical and psychological.

    They also need to be able to identify meaningful process targets, such as:

  • higher management audit scores or rankings; or

  • higher health and safety climate survey scores.

    In smaller firms, or within individual business units in large organisations, it may make more sense to construct qualitative targets within a basic action plan. Elements might include:

  • upgrading the health and safety policy;

  • reviewing all risk assessments;

  • upgrading standards of control for principal risks (and levels of compliance);

  • setting training targets;

  • carrying out regular monitoring; and

  • improving consultation.

    6. Assign responsibility for collecting and analysing the data

    It is important to assign responsibility for collecting and analysing the data, and to hold people accountable for this activity.

    7. Compare actual performance against target

    The emphasis should be on achievements rather than failures, but it is important to analyse the reasons for substandard performance if improvements are to be made.

    8. Decide on corrective action

    The measurement data should provide information to enable decisions to be made about what corrective action is required and where and when it is necessary.

    9. Review the measures

    The measures derived need to be reviewed regularly to ensure that they remain appropriate, useful and cost-effective. There should not be frequent changes of measures because this can lead to confusion.

    Means to an end

    Effective measurement provides information on both the level of performance and why the performance level is as it is; if measurement cannot be used to understand performance it is of little use. It should seek to answer questions, such as "where are we now relative to our overall health and safety aims and objectives?" and aid decision making, for example, "what progress is necessary and reasonable in the circumstances?"

    Performance targets should only ever be seen as a means to an end, and must never be allowed to become an end in themselves. The process of target setting is as important as the targets that are set, and examining and developing that process goes to the heart of the way in which organisations are managed and led.

    To be meaningful, target setting must be underpinned by a robust understanding of current performance, including continuing problems and their causes and possible solutions.

    Besides helping to achieve improvement, much of the value of target setting derives from the extent to which it helps those involved to develop their understanding of strategic management issues. From this perspective, understanding, in the light of experience, why a target was not reached or why it was exceeded is what is of fundamental importance.


    Box 1: Progress towards Revitalising targets

    Injury

    Target: to reduce the incidence rate of fatal and major injury accidents by 10% by 2009/10 and by 5% by 2004/05.

    Baseline: 1999/2000: injury indicator: 263.2. The Revitalising injury indicator is the sum of two parts: the worker (employees and self-employed) rate of fatal injury and the employee rate of major injury uprated by the estimated reporting level of employee injuries.

    Progress 2001/02: injury indicator: 268.9. The indicator increased because the marginal decrease in the rate of reported major injury (-0.6%) is outweighed by the decrease in the reporting level in 2001/02 and consequent increase in the uprating factor. Since 1999/2000, the indicator has fluctuated by small amounts that are not statistically significant. There is no discernible improvement since the base year.

    The target is to reduce the indicator by 10% in the 10-year period 1999/2000 to 2009/10. The indicative rates of fatal and major injury would be 250 in 2004/05 and 236.9 in 2009/10.

    Absence

    Target: to reduce the number of working days lost per 100,000 workers from work-related injuries and ill health by 30% by 2009/10 and by 15% by 2004/05.

    Baseline: only absolute numbers (rather than rates) are currently available. An estimated 40.2 million total days per year were lost in 2001/02. Based on this, illustrative targets would be 34 million in 2004/05 and 28 million in 2009/10.

    Progress 2001/02: progress can only be measured once further statistics become available; an initial judgement will be made in 2004. The latest information suggests that the scale of the problem for work-related ill health and the resulting days lost is greater than was previously estimated.

    Ill health

    Target: to reduce the annual incidence rate of new cases of work-related ill health by 20% by 2009/10 and by 10% by 2004/05.

    Baseline: in 2001/02, an estimated 2,200 out of every 100,000 people employed in the previous 12 months suffered from a new work-related illness. Illustrative targets would be 2,000 per 100,000 in 2004/05 and 1,800 per 100,000 in 2009/10.

    Progress 2001/02: progress can only be measured once further statistics become available. The latest information suggests that the scale of the problem to be addressed by the strategies in terms of work-related ill health is now greater than was previously estimated.


    Box 2: Problems with injury and ill-health statistics

    The use of injury and ill-health statistics as the only measure of health and safety performance has several potential drawbacks.

  • Under-reporting - an emphasis on injury and ill-health rates as a measure, particularly when related to reward systems, can lead to events not being reported in order to "maintain" performance.

  • Whether a particular event results in an injury is often a matter of chance, so it will not necessarily reflect whether or not a hazard is under control. An organisation can have a low injury rate because of luck or because fewer people are exposed, rather than good health and safety management.

  • Injury rates often do not reflect the potential severity of an event, merely the consequence. For example, the same failing adequately to guard a machine could result in a cut finger or an amputation.

  • People can stay off work for reasons that do not reflect the severity of the event.

  • There is evidence to show there is not necessarily a relationship between "occupational" injury statistics (eg slips, trips and falls) and control of major accident hazards (eg loss of containment of flammable or toxic material).

  • A low injury rate can lead to complacency.

  • A low injury rate results in few data points being available.

  • There must have been a failure, ie injury or ill health, in order to get a data point.

  • Injury statistics reflect outcomes not causes.

    		•


    Box 3: When to measure

  • Smart targets and objectives will have milestones set; monitoring progress towards targets should be aligned with the timescales for achievement.

  • Management arrangements and RCSs do not change from day-to-day, checks are appropriate when there is the potential for change from one state to another. For example, whenever changes are made that could have an impact on the operation of the systems, or when information is obtained that indicates that the system as designed has failed in some way, eg there has been an injury. Similarly, checks for workplace precautions should be appropriate; suppliers of plant and equipment will often prescribe inspection and maintenance intervals to ensure optimum performance.

  • The relative importance of an activity or precaution in the overall control of risk will affect how often measurements need to be made. Some precautions may need continuous monitoring, eg flow of cooling water.

  • Some legislation, such as that covering lifting equipment, requires monitoring at prescribed intervals.

  • When there is evidence of non-compliance, more frequent monitoring may be appropriate to check remedial action has been effective.

  • Where there is evidence of regular compliance, it may be appropriate to reduce monitoring so that resources can be focused elsewhere.

  • Some activities may only take place outside of normal working hours or at particular times of the year: monitoring will need to take account of this.

    		•


    Box 4: Who should measure?

    Senior managers must satisfy themselves that arrangements to control risks are:

  • in place;

  • complied with; and

  • effective.

    Organisations must decide how responsibility for active and reactive monitoring is allocated in the management chain. Managers should be given responsibility for monitoring the achievement of plans and targets and compliance with standards for which they and their subordinates are responsible. Managers and supervisors responsible for direct implementation of standards should monitor compliance in detail and be competent to do so.

    Above this level of control, monitoring needs to be more selective, but it must provide assurance that first-line monitoring of appropriate quality and quantity is taking place. Managers need performance standards - who does what, when and to what effect -to indicate how they are to monitor performance.

    		•


    Box 5: The "four Ps"

    The headings "premises", "plant and substances", "procedures" and "people" can be used to ensure that risk issues are covered. For example, the workplace precautions associated with workplace transport could include:

  • premises - defined roadways/one way system; need for reversing eliminated/minimised; roadways in good condition; speed bumps;

  • plant - vehicle selection, eg good driver access/visibility; vehicles maintained in good condition (tyres/brakes); seat restraints fitted; reversing aids provided;

  • procedures - speed limits set for vehicles; vehicles chocked appropriately; reversing controlled; authorised drivers; drivers in safe position during loading; and

  • people - competent drivers; drivers following the speed limit; pedestrians using designated walkways.

    In compiling a checklist to be used in monitoring a department, it may be decided that the important issues are:

  • premises - access/escape; housekeeping; working environment;

  • plant and substances - machinery guarding; local exhaust ventilation; use/storage/ separation of materials/chemicals;

  • procedures - permits to work; use of personal protective equipment; procedures followed; and

  • people - health surveillance; people's behaviour; appropriate authorised person.

    		•

    1"A guide to measuring health and safety performance", www.hse.gov.uk/opsunit/perfmeas.htm, free.

    2"Targets for change: guidance on setting corporate health and safety improvement targets", www.rospa.co.uk (go to "Occupational safety", then "Performance targets"), free.

    3"Health and safety targets: how are we doing?", MISC489, HSE Books or at www.hse.gov.uk/aboutus/reports, free.