Summary

• Page properties (3 modified, 0 added, 0 removed)

Details

Page properties

Title

...	...	@@ -1,1 +1,1 @@
1		-Role Play
	1	+Post Box Exercise

Author

...	...	@@ -1,1 +1,1 @@
1		-XWiki.RandellGreenlee
	1	+XWiki.EricDeRuijter

Content

...	...	@@ -4,10 +4,8 @@
4	4
5	5	= Description =
6	6
7		-The candidate is placed in a situation with an actor as counterpart. The actor steers the situation, based on a predefined scenario, passing specific realistic situations. Observation is done based on a checklist.
	7	+This method is used for skills that result in a written or drawn result. The candidate gets an assignment on paper and has time to prepare the written result. The result is checked with a prepared checklist of sample solutions. Examples or results could be personnel planning, a light plot, an Email. The advantage is a high certainty of competence, compared with assessing prior work. For more artistic skills, this can be combined with a role play or an interview. This method is highly effective for assessing action orientation and entrepreneurial thinking. The candidate can act independently in a wide frame. Complex information has to be quickly analysed, processed and combined to recommend rational actions.
8	8
9		-The role play is effective for testing communicative and social competences like interviewing, rhetoric, argumentation, empathy, assertiveness, persuasiveness, sensitivity (behavioral observation). It's also useful for assessing operational readiness, goal orientation, frustration tolerance, persistence, problem solving skills, analytic skills, decision making skills. and. It can test the skill of adapting communication and language to different target groups. The setting of the role play can be a difficult collaborator interview, a conflict with a leading person, a critical incident on the workplace, a discussion with a designer or a discussion with a technical director.
10		-
11	11	----
12	12
13	13	= Quality Concepts =
...	...	@@ -14,38 +14,23 @@
14	14
15	15	=== Validity ===
16	16
17		-|(((
18		-The validity of the method is increased by the tasks being appropriate for the competences to be measured. Above all, interpersonal and artistic skills can be assessed with the role play. The following communicative and social skills can be assessed: interviewing, rhetoric, reasoning, empathy, assertiveness, persuasiveness and sensitivity. In order to increase the validity, the selected role-playing situations should be representative of the event technology. It is important to make the situation challenging and relevant. The situations need to occur regularly in real life. As the scenario only reflects one concrete situation there is a risk that it doesn't reflect all situations in professional practice. However, the method is close to real life and is well controllable.
	15	+Due to the high visual validity, the post box exercise is a popular method. It can be used to evaluate action orientation and entrepreneurial thinking well. The method is used, among other things, to evaluate work organization, decision-making ability, organizational skills, conceptual work, planning and decision-making behavior, recognizing relations, forming priorities and ability to delegate. The factors of behavior include preparation and planning for action, work performance (quantity) and the search for precise instructions.
	16	+The validity of Post Box Exercise is limited by the fact that processes can be very interdependent. To solve one problem, you have to solve a prior one.
	17	+Previous studies have shown different correlation coefficients (from -.45 to .76) for the relation between post basket result and job success (see Schippmann et al. 1990). Obermann (2018) uses a value of .18, which corresponds to a very low correlation. However, it should be noted that in the assessment context of validation, job success is only indirectly affected.
19	19
20		-A particular challenge for the validity of the role play is the requirement on the candidate to be active as an actor. Care should be taken here to make it clear that the acting performance is not assessed. The candidate should be able to develop his role freely and without pressure in order to be able to express his inherent competencies.
	19	+(% style="color:inherit; font-family:inherit; font-size:23px" %)Reliability
21	21
22		-According to Obermann (2018), the relationship between job success and the results of role play is .16. This corresponds to a very low correlation. It should be noted, however, that in the assessment context, job success is only indirectly relevant in the context of validation.
23		-)))
24	24
25		-=== Reliability ===
	22	+Reliability and evaluation objectivity are special challenges in post box exercises. A meta-analysis based on 16 individual studies showed an interrater reliability of .76 (Whetzel et al. 2014). The assessment of different raters are therefore "good" or "substantial".
	23	+Since chance hits can distort the assessment results, a sufficient number of tasks should be set. Standardization can further increase reliability. In the scenario, an order and a time window for processing should be defined.
26	26
27		-|(((
28		-The role play should be constructed in such a way that the competences to be measured should not be reflected in a single situation. Otherwise there is a risk that the candidate will accidentally behave correctly. If several situations are to reveal the same competencies, the reliability is increased.
	25	+Reliabilität und Auswertungsobjektivität sind besondere Herausforderungen bei Post Box Exercise. Eine Metaanalyse auf Basis von 16 Einzelstudien ergab eine Interrater-Reliabilität von .76 (Whetzel et al. 2014). Die Urteile unterschiedlicher Bewerter stimmen also "gut" bzw. "substanziell" miteinander überein.
	26	+Da chance hits die Assessment Ergebnisse verzerren können, sollten ausreichend viele Aufgaben gestellt werden. Durch eine Standardisierung kann die Reliabilität zusätzlich erhöht werden. So sollte im Szenario eine Reihenfolge und ein Zeitfenster zur Bearbeitung festgelegt werden.
29	29
30		-The situations and the start of the conversation should be clearly worded. Unclear formulations can lead to the candidate misunderstanding the task and situation, which disrupts reliability. Language barriers should also be taken into account accordingly. The starting position is identical for each candidate if the scenario is well developed. But the path can be different and contain unpredicted situations. The assessor needs to find a balance to steer on one hand, but leave the initiative to the candidate on the other.
31		-
32		-Various factors can limit the candidate's concentration and performance:
33		-
34		-poor room lighting or ventilation, disturbing sources of noise, lack of equipment, poor health or psychological condition of the candidate (e.g. fear, stress).
35		-
36		-The reliability is increased by not changing the order of the methods. In this way, the assessment processes remain comparable.
37		-
38		-The assessors / observers should be extensively trained to ensure evaluation and interpretation, to ensure comparability and objectivity. It should be clear which behaviors can be attributed to certain skills. So, reliability is enforced by a good scoring structure.
39		-
40		-The behavior of the interlocutors is standardized by means of a written instruction. This is a prerequisite for objectivity, which in turn ensures that different results are due to the performance of the participants and not to variations in the interlocutors.
41		-)))
42		-
43	43	== Limitations ==
44	44
45		-Role plays are less suitable for assessing knowledge-based skills. Likewise, no competencies are expressed that are expressed in written and drawn results. The methodology is less usefull for observing physical skills, exept if combined with an observation in a simulated environment.
	30	+Due to the low validity, post box exercises should be used in combination with other methods. This method is less suitable for querying factual knowledge. Complex practical skills are more likely to be assessed with an observation in a simulated environment or on site.
46	46
47		-To check how the candidate behaves in a non-simulated environment, real-life observations should be preferred. A good scenario should be able to bring forward knowledge based skills.
48		-
49	49	----
50	50
51	51	= Considerations =
...	...	@@ -53,24 +53,23 @@
53	53	== Tips ==
54	54
55	55	|(((
56		-Educate the assessee that their acting performance is not being tested. He should behave as he would in everyday work. The atmosphere should be as realistic as possible. Assessee and interlocutors (= role players) need prior written information. This gives the other party information on how to react to certain situations and questions. The person watching should not have direct eye contact with the assessee. In advance, it is recommended for observer training to practice role-playing with video analysis. Assessors need to be trained for the specific scenario's, idealy in interaction with their collegues.
57		-
58		-It can help to get the candidate on the right track to use some properties (assessories) to visualise the role of the actor (for example a typical piece of clothing, documents, models, ...). Make clear the candidate is playing his/her own role in the given situation.
	39	+The more realistic the presentation of the documents and the settings, the higher the tasks will be accepted.
	40	+After the written processing, an oral discussion should follow. Otherwise, excellent ideas for task processing that deviate from the proposed solutions may not be recognized accordingly. In the debriefing, the candidate can resolve ambiguities and answer questions.
59	59	)))
60	60
61	61	== Traps ==
62	62
63		-The situation characteristics from the requirements analysis cannot be implemented carefully enough. This can have the consequence that the behavior is not relevant for the competence to be tested. Sentences like "In reality I would do it completely differently." are signs of this. There is a risk for socialy expected answers and behavior.
	45	+One of the traps is that the assignments could be too easy or too difficult.
	46	+The solvability of the processes should not depend too much on one another. On the other hand, the documents should be linked to each other in a complex manner, otherwise the analysis level will remain at a low level.
	47	+If the processing time is too short, the competence is overemphasized how quickly texts are recorded.
64	64
65		-If the assessor has no experience with role-playing games, an unpredictable momentum can develop which affects the comparability. Assessors need to be aware of the reasons for all scenario steps and have to balance natural behaviour and following the storyline of the scenario that leads to the visibility of the skills. Errors can be a high level of willingness to compromise or hardness. Principle of local independence: The interview partner must act neutrally in the individual situations, even if the assessee had a poor start in a previous situation. Otherwise only the overall performance can be assessed, but not individual competencies. Avoid that the same assessor is playing different roles for the same candidate.
66		-
67		-A short instruction in the role play overemphasizes the situational flexibility. Communicative observations can take place less.
68		-
69		-Uncertainties about the setting (e.g. Has the exercise already started? Who ends the conversation?) should be avoided. Other participants should not act as conversation partners as this does not standardize the exercise. An assessor can't be the role player at the same time.
70		-
71	71	== Scoring Tools ==
72	72
73		-There could be used a check List and notes about the behaviour of the candidate with reference to the expected behaviour. The criteria should be derived from the sectoral layer skills, in other words, they are a concretization of the visible, observable result of the skill in a specific situation. As the situation can differ, the scoring tool will be more general and leave more room for interpretation than other methods.
	51	+There are three options for evaluation:
	52	+~1. The candidate processes the tasks with free formulations. The observers then evaluate the documents. This procedure is less recommendable, since the comparability and evaluation objectivity are endangered.
	53	+2. Open questions are pre-formulated for each process. The candidates answer them in interviews or a presentation.
	54	+3. Pre-formulated questions with given answers are handed out.
	55	+It should be noted that post box exercise solutions are usually not right or wrong, but more or less successful.
74	74
75	75	----
76	76
...	...	@@ -78,33 +78,34 @@
78	78
79	79	== Information for Standard ==
80	80
81		-The method should include a short description and the expected behaviour.
	63	+The tasks and scheduled processing time should not vary compared to other candidates. Possible language barriers should be considered. The answers should be evaluated according to a marking guide.
82	82
83	83	== Development ==
84	84
85		-The development of an assessment scenario starts from the analysis of the skills that need to be assessed. The scenario reflects a real life situation in which the tested skills are prominently present. Based on the scenario, a list of observable criteria, derived from the competences is created.
	67	+An appropriate task difficulty must be chosen for the development. The planned time for the tasks must not be too short. The main focus during development is on an extensive and diversified creation of tasks that are as close as possible to the daily work environment.
86	86
87	87	== Needs/Set-Up ==
88	88
89		-In most cases a quiet space with a table and some chairs is all that is needed. Depending on the test, documentation can be needed (light plots, sound documentation, …).
	71	+In addition to an examination room with chair and work table, the correspondingly prepared equipment is required. This must be sorted according to the intended order of processing.
90	90
91	91	== Requirements for Assessors ==
92	92
93		-Assessors must be able to instruct the actors according to the situation. As observers, they need skills in observing, diagnosing and evaluating the candidate's behavior. The assessment of competencies requires professional expertise.
	75	+Assessors need appropriate pedagogical and technical knowledge to prepare the tasks and evaluate the approaches and results.
94	94
95	95	== Examples ==
96	96
97		-The candidate has a discussion with a designer about a design concept that needs to be translated into a practical plan. In this discussion e.g. conceptual and artistical understanding, technical problem solving and predicting problems and language use can be assessed.
	79	+A possible Post Box Exercise to test management skills is a combination of scheduling, consultation with technicians and email correspondence.
98	98
99	99	== In Combination with ==
100	100
101		-Role Plays can be used in combination with a post box exercise, where the information needed is not (or limited) put on paper, but the counterpart is presenting the information and the candidate can ask questions about the required results. It can be complemented with a criterium focused interview for skills that didn't become visible.
	83	+To check the theoretical knowledge, written tests or oral tests should also be conducted.
102	102
103	103	= References/Notes =
104	104
105		-* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evctoolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evctoolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]. (last 17.08.2020)
106	106	* Eck, C. et al. (2016): Assessment-Center. Entwicklung und Anwendung – mit 57 AC-Aufgaben und Checklisten zum Downloaden und Bearbeiten im Internet. 3. Aufl. Berlin / Heidelberg: Springer.
107	107	* Obermann, C. (2018): Assessment Center. Entwicklung, Durchführung, Trends. Mit neuen originalen AC-Übungen. 6., vollständig überarb. u. erw. Aufl. Wiesbaden: Springer Fachmedien.
	89	+* Whetzel DL, Rotenberry PF, McDaniel MA (2014): In-basket Validity: a systematic review. Int J Sel Assess 22(1):62–79.
	90	+* Schippmann, J.S., Prien, E.P. and Katz, J.A. (1990): RELIABILITY AND VALIDITY OF IN‐BASKET PERFORMANCE MEASURES. Personnel Psychology, 43:837-859.
108	108
109	109	== AT ==
110	110