Wiki source code of Observation on Site

Last modified by Ivan Kharitonov on 2022/01/30 17:23
Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 = Description =
6
7 The candidate is observed in the authentic professional context of a real-life situation.
8
9 This method is used for skills that can be shown on the workspace.
10
11 It proofs the ability to carry out certain tasks. In general, competencies such as "job knowledge, quality and quantity of output, initiative, leadership abilities, supervision, dependability, cooperation, judgment, versatility, health" (M. Faseeh. 2013: 66) can be assessed.
12
13 ----
14
15 = Quality Concepts =
16
17 === Validity ===
18
19 The advantage is that this is close to reality. The disadvantage is that the testing set-up is not fully under control. For example, you can’t foresee the content and technical needs of a play. This unpredictability increases the outcome validity. Uncontrollable factors like content and technical needs of the play only add a further dimension, which is most suitable for participants to demonstrate their working experience. The external factor means that the complexity of a competence can vary.
20
21 There is always some risk of confounded results due to “observer effect;” i.e., subjects may behave atypically if they know they’re being observed. The practical situation in a real live environment is never the same and will not always contain all the competences that need to be assessed. The validity of the method increases with clearly defined criteria relevant to the event technology industry, according to which the candidate is observed and evaluated.  Here the principle applies: The high external validity of the results  (= transferability to reality) goes hand in hand with a low internal validity (many uncontrollable variables).
22
23 The method is less good for testing underpinning knowledge, due to lack of control of the test situation.
24
25 === Reliability ===
26
27 The procedure to carry out an examination in a real-life environment is extremely resource-consuming and does not provide the possibility of a standardized testing environment. In the complex machinery of the build of a production, it is hard to distinguish the results of one individual form the collective result.
28
29 The test situation is hard to control, neighbouring skills, that are not part of the assessment or behaviour of colleagues can influence performance. A standardization and the corresponding comparability of the observation is difficult to carry out, since every work situation is different. When a candidate is assessed in his/her own environment, there is a risk for routine with specific equipment, which does not reflect the ability to work with different equipment. When a candidate is tested in a (for him/her) unknown situation, the candidate must have time to acclimatise and familiarise himself with the organisation and the situation. In order to increase reliability, criteria should be specified in advance, which should be observed when observing.
30
31 A hallmark of high reliability is that different observers of the same candidate show a high degree of agreement in their results.
32
33 == Limitations ==
34
35 Since not all companies have easily observed job duties or complete job cycles, the observation on site method is limited. The method is less suitable for querying specific, defined competencies. The typical "deadline" situation makes that the assessment is not the priority and the assessment can influence the production result (for non stage activities, this is less a problem e.g. workshop work). There can be privacy issues, because of other workers on the floor.
36
37 ----
38
39 = Considerations =
40
41 == Tips ==
42
43 Choose an environment that is representative for the competences tested.
44 It helps to make a list of the key performance criteria that can be observed (e.g. attitude, timeliness, effort, consistency, ethics, safety). 
45 A measure should be developed that is distributed to the field of event technology and the interference forces.
46 When observing, it helps to take notes.
47 Not only the use of several observers, but also consultation with co-workers, for example, can also increase the accuracy of the performance evaluation, reduce bias and increase candidate's perceptions of fairness. Though the consultation of coworkers can be a privacy issue.
48
49 == Traps ==
50
51 The candidate should not feel monitored during the observation. Framework conditions should be created in which the candidate behaves as much as possible in everyday work.
52 The candidate should not feel monitored during the observation. Framework conditions should be created in which the candidate behaves as possible in everyday work.
53 The activity on the day of assessment can be different as expected and miss some skills.
54 When working in teams, collegues can support or counteract the candidate, also unconsciously.
55
56 == Scoring Tools ==
57
58 A list with the criteria represents the basis for the assessment, after which it is decided whether the expected behavior has been observed and whether the candidate has the competence to be assessed. 
59 The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation.
60 As the situation can differ, the scoring tool will be more general and leave more room for interpretation than other methods.
61
62 ----
63
64 = Implementation =
65
66 == Information for Standard ==
67
68 The standard needs to describe the concrete situations and the expected complexity of the skills to be assessed.
69
70 == Development ==
71
72 Developing an assessment situation needs the development of expected situations and a corresponding list of observable criteria, derived from the competences.
73
74 == Needs/Set-Up ==
75
76 There is no need for a specific setup, but:
77
78 * The organisation and the workers need to be briefed in advance of the assessment.
79 * There needs to be place for the assessors to observe without distracting the normal work.
80 * This type of assessment needs multiple assessors, as it is physically not possible to see everything.
81
82 Depending on the target behavior, there may be socially or professionally sensitive issues to be dealt with (e.g., invasion of privacy on student political activities or living arrangements) or even legal considerations (e.g., substance abuse).
83 Competences will occur at different moments of the process, which means a long time span for the assessment.
84
85 == Requirements for Assessors ==
86
87 Assessors need competences for valid observations, such as those that can be acquired in observer training courses. They should have a basic knowledge of diagnostics, be able to deal with perceptual effects (e.g. errors of observation and assessment) and be able to recognize their own subjectivity. A professional competence is essential for the evaluation of the candidate's performance against the background of the assessment standard.
88 Assessors must also be able to identify interpersonal processes with other co-workers and their influence on the candidate's actions.
89
90 == Examples ==
91
92 For the skills "Working on heights" a candidate should perform several activities on ladders, scaffolding, … Based on on a checklist, his/her behaviour is observed.
93
94 == In Combination with ==
95
96 To test the underpinning knowledge of the skills, a second method like interview or theoretical test can be used.
97
98 = References/Notes =
99
100 * Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]  (last 17.08.2020)
101 * [[https:~~/~~/www.itb-consulting.de/de/produktkatalog/beobachterschulung/>>https://www.itb-consulting.de/de/produktkatalog/beobachterschulung/]] (last 25.08.2020)
102 * [[http:~~/~~/www.powerflexweb.com/1491/docs/EncompassSupervisor/obs01/01obs01.htm>>http://www.powerflexweb.com/1491/docs/EncompassSupervisor/obs01/01obs01.htm]] (last 29.07.2020)
103 * Khan, Faseeh. (2013). Role of Performance Appraisal System on Employees Motivation. IOSR Journal of Business and Management. 8. 66-83. 10.9790/487X-0846683.
104 * Ryan, T. (2020): How to Observe Employee Performance. Online: [[https:~~/~~/smallbusiness.chron.com/observe-employee-performance-19393.html >>https://smallbusiness.chron.com/observe-employee-performance-19393.html]](last 29.07.2020)
105 * Spear-Swerling L. (2013) Observational Assessments. In: Volkmar F.R. (eds) Encyclopedia of Autism Spectrum Disorders. Springer, New York, NY. [[https:~~/~~/doi.org/10.1007/978-1-4419-1698-3_1772>>https://doi.org/10.1007/978-1-4419-1698-3_1772]]