Observation on Site

Last modified by Ivan Kharitonov on 2022/01/30 17:23
Contents

Description

The candidate is observed in the authentic professional context of a real-life situation.

This method is used for skills that can be shown on the workspace.

It proofs the ability to carry out certain tasks. In general, competencies such as "job knowledge, quality and quantity of output, initiative, leadership abilities, supervision, dependability, cooperation, judgment, versatility, health" (M. Faseeh. 2013: 66) can be assessed.

Quality Concepts

Validity

The advantage is that this is close to reality. The disadvantage is that the testing set-up is not fully under control. For example, you can’t foresee the content and technical needs of a play. This unpredictability increases the outcome validity. Uncontrollable factors like content and technical needs of the play only add a further dimension, which is most suitable for participants to demonstrate their working experience. The external factor means that the complexity of a competence can vary.

There is always some risk of confounded results due to “observer effect;” i.e., subjects may behave atypically if they know they’re being observed. The practical situation in a real live environment is never the same and will not always contain all the competences that need to be assessed. The validity of the method increases with clearly defined criteria relevant to the event technology industry, according to which the candidate is observed and evaluated.  Here the principle applies: The high external validity of the results  (= transferability to reality) goes hand in hand with a low internal validity (many uncontrollable variables).

The method is less good for testing underpinning knowledge, due to lack of control of the test situation.

Reliability

The procedure to carry out an examination in a real-life environment is extremely resource-consuming and does not provide the possibility of a standardized testing environment. In the complex machinery of the build of a production, it is hard to distinguish the results of one individual form the collective result.

The test situation is hard to control, neighbouring skills, that are not part of the assessment or behaviour of colleagues can influence performance. A standardization and the corresponding comparability of the observation is difficult to carry out, since every work situation is different. When a candidate is assessed in his/her own environment, there is a risk for routine with specific equipment, which does not reflect the ability to work with different equipment. When a candidate is tested in a (for him/her) unknown situation, the candidate must have time to acclimatise and familiarise himself with the organisation and the situation. In order to increase reliability, criteria should be specified in advance, which should be observed when observing.

A hallmark of high reliability is that different observers of the same candidate show a high degree of agreement in their results.

Limitations

Since not all companies have easily observed job duties or complete job cycles, the observation on site method is limited. The method is less suitable for querying specific, defined competencies. The typical "deadline" situation makes that the assessment is not the priority and the assessment can influence the production result (for non stage activities, this is less a problem e.g. workshop work). There can be privacy issues, because of other workers on the floor.

Considerations

Tips

Choose an environment that is representative for the competences tested.
It helps to make a list of the key performance criteria that can be observed (e.g. attitude, timeliness, effort, consistency, ethics, safety). 
A measure should be developed that is distributed to the field of event technology and the interference forces.
When observing, it helps to take notes.
Not only the use of several observers, but also consultation with co-workers, for example, can also increase the accuracy of the performance evaluation, reduce bias and increase candidate's perceptions of fairness. Though the consultation of coworkers can be a privacy issue.

Traps

The candidate should not feel monitored during the observation. Framework conditions should be created in which the candidate behaves as much as possible in everyday work.
The candidate should not feel monitored during the observation. Framework conditions should be created in which the candidate behaves as possible in everyday work.
The activity on the day of assessment can be different as expected and miss some skills.
When working in teams, collegues can support or counteract the candidate, also unconsciously.

Scoring Tools

A list with the criteria represents the basis for the assessment, after which it is decided whether the expected behavior has been observed and whether the candidate has the competence to be assessed. 
The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation.
As the situation can differ, the scoring tool will be more general and leave more room for interpretation than other methods.

Implementation

Information for Standard

The standard needs to describe the concrete situations and the expected complexity of the skills to be assessed.

Development

Developing an assessment situation needs the development of expected situations and a corresponding list of observable criteria, derived from the competences.

Needs/Set-Up

There is no need for a specific setup, but:

  • The organisation and the workers need to be briefed in advance of the assessment. 
  • There needs to be place for the assessors to observe without distracting the normal work.
  • This type of assessment needs multiple assessors, as it is physically not possible to see everything.

Depending on the target behavior, there may be socially or professionally sensitive issues to be dealt with (e.g., invasion of privacy on student political activities or living arrangements) or even legal considerations (e.g., substance abuse).
Competences will occur at different moments of the process, which means a long time span for the assessment.

Requirements for Assessors

Assessors need competences for valid observations, such as those that can be acquired in observer training courses. They should have a basic knowledge of diagnostics, be able to deal with perceptual effects (e.g. errors of observation and assessment) and be able to recognize their own subjectivity. A professional competence is essential for the evaluation of the candidate's performance against the background of the assessment standard.
Assessors must also be able to identify interpersonal processes with other co-workers and their influence on the candidate's actions.

Examples

For the skills "Working on heights" a candidate should perform several activities on ladders, scaffolding, … Based on on a checklist, his/her behaviour is observed.

In Combination with

To test the underpinning knowledge of the skills, a second method like interview or theoretical test can be used.

References/Notes

Tags: