Summary

• Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Title

...	...	@@ -1,1 +1,1 @@
1		-Observation in a Simulated Environment
	1	+Oral Examination

Content

...	...	@@ -4,10 +4,8 @@
4	4
5	5	= Description =
6	6
7		-The simulated environment reflects a real live situation, but is standardised. This makes it possible to build in incentives for behaviour or choices. The situation can be a "copy" of a real live situation, but also a roleplay (for more behaviour skills). The candidate is observed in this simulated situation.
	7	+The oral test is a method of assessing the candidate's spoken word. Here assessor and candidate enter into a spoken interaction. An oral test can take place as a presentation (see AMD Presentation), interrogation or application. The oral test generally does not assess linguistic competence as such, but rather knowledge, understanding and problem solving ability.
8	8
9		-This method is used for skills that can be shown in the workspace. The assessment method allows to test very specific competences, as the environment can be controlled. Mainly for practical, observable skills.
10		-
11	11	----
12	12
13	13	= Quality Concepts =
...	...	@@ -14,18 +14,20 @@
14	14
15	15	=== Validity ===
16	16
17		-Since all factors are under control, the internal validity of this method is high. The method excludes unpredictability of situation and environment. So, it is easier to ensure safety. Very specific competencies can be tested. Since the behavior of people can change as a result of the observation situation (Hawthorne effect), internal validity is also threatened. This effect can be partially reduced if the work situation is only filmed (indirect observation). Since it is less a real-life situation, the external validity (transferability) of the observed behavior is lower. A good test will reflect real life situations in a controlled environment as much as possible.
	15	+Oral tests could be valid for assessing e.g. understanding, applied problem solving, interpersonal competence, intrapersonal qualities (confidence, self-awareness, professionalism, ethics) and intergrated practice.
	16	+Oral tests are valid if the candidate can demonstrate the competencies that are relevant to the validation process. It must therefore be checked whether the test queries what it should query (face validity). To check the content validity, it must be asked whether the discussed content is part of the required competencies, whether it goes beyond it or whether it is too barely discussed. In order to increase the construct validity of the oral test, the tasks must be in line with the theoretical understanding of the skills of event technology.
18	18
19	19	=== Reliability ===
20	20
21		-The quality of simulated environment observation depends on the accuracy and repeatability of the test setup.
22		-Simulated environments guarantee equal treatment of candidates, the result should be identical, wherever and by whatever assessors they are conducted. Therefore every candidate is assessed in an identical situation.
23		-One of the elements in this are well trained assessors and a levelling system, this avoids that assessment would be biased by assessors influenced by previous tests or looking outside the competences to for example not occupation related behaviour.
24		-The reliability is increased by the possibility to easily develop exact observable criteria.
	20	+Reliability can be affected by candidate anxiety, candidate hearing or speech difficulties or assessor bias (e.g. due to gender, ethnicity, language).
	21	+In order to standardize the test and increase comparability, a criteria list and an answer key should be created. The fact that the assessor can ask questions can affect reliability. An answer that was only given on request should possibly get fewer points.
	22	+In order to increase the reliability, a competence should be tested in different ways.
	23	+The use of multiple assessors also increases reliability.
	24	+Due to the limited number of questions and the many disruptive factors in the interaction, the oral test is less reliable.
25	25
26	26	== Limitations ==
27	27
28		-Development of a assessment set-up is time consuming.
	28	+Since the time and question options are limited, only a limited amount of knowledge can be queried. Due to disturbing factors (excitement, stress, unusual situation) the method should be used in combination with other methods.
29	29
30	30	----
31	31
...	...	@@ -33,25 +33,18 @@
33	33
34	34	== Tips ==
35	35
36		-Organise the test in a way that the candidate feels at ease. If it is a tradition to have a cup of coffee at the start of a working day, include this in the startup of the test.
37		-Give the candidate time to discover the situation.
38		-Do not built in traps or tricky situations that hardly ever occur in real life.
39		-Be clear and open about the role and activities of the observers. Attention points can be:
	36	+The candidates should be informed in advance about the framework conditions of the oral test: e.g. time period, location, requirements and items that are not included.
	37	+A criteria list helps the assessors to objectively assess the results of the oral test. The assessors should be aware that inquiries can give unwanted tips.
	38	+Candidates should be given sufficient time to answer.
40	40
41		-* Observers write also about positive points.
42		-* Observers are silent, because they keep a distance.
43		-* Observers will only stop the test in case of danger or overtime.
44		-
45	45	== Traps ==
46	46
47		-If the candidate needs support, the assistant must be trained to limit the intervention to what the candidate requires and not (as we would do in reality) to take over the decision-making process or be proactive.
48		-There is a risk that the assessor is biased. That is why assessors should be professionals from the field of competences being assessed. Assessors have to be aware that there are different methods to perform a specific task and should take distance from one prefered method, for as far as the goal is reached.
	42	+It should not be the presentation and speaking skills that are assessed, but rather the competencies that are queried as part of the assessment process.
	43	+If the oral test is not recorded, the results can be objected afterwards.
49	49
50	50	== Scoring Tools ==
51	51
52		-Observing can be done through a list of observable criteria. The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation.
53		-As the situation is always identical, the scoring tool can be very specific and leave little room for interpretation.
54		-The final decision is made based on the link of the criteria with the competence and by comparing the observations of the different assessors.
	47	+The results of the oral test can usually be evaluated immediately afterwards. A marking guide can be used here, which also serves as a framework for feedback to the candidate. Since it can be difficult to take notes and interview the candidate at the same time, it makes sense to use several assessors. In this case, one can record the conversation while the other is conducting the conversation. Audio or video recording of the oral test is also highly recommended.
55	55
56	56	----
57	57
...	...	@@ -59,40 +59,34 @@
59	59
60	60	== Information for Standard ==
61	61
62		-The standard must describe the specific situations, incentives and expected complexity of the skills to be assessed.
	55	+The created marking guide should be aligned with the competence descriptions of the UNITs. The tasks and questions of the oral test should be precisely formulated in advance and each candidate should be asked the same questions. The candidate's answers are compared with the expected answers from the marking guide. In addition, cues and support that the assessors can provide are defined in advance.
63	63
64	64	== Development ==
65	65
66		-The development of an observation in a simulated environment starts with the analysis of the skills that need to be evaluated. Since not every skill can be tested in all variations, representative situations are chosen to reflect the mastery of the general skill. The skills are built into a well-chosen scenario that reflects a real-life experience, but also integrates behavioural incentives and choices. The candidate is asked to perform a task, but the environment limits or alters the way the task is performed. In this way, the candidate must make his/her own decisions.
67		-The activities should reflect different contexts. Often a skill or behavior is built in twice to improve reliability and avoid "false positives".
68		-Assessment facilities must be tested and updated before they are used with "real" candidates.
	59	+The development process involves the creation of tasks and questions that are designed to assess the relevant competencies. Expected answers and possible cues are set out in a marking guide.
69	69
70	70	== Needs/Set-Up ==
71	71
72		-This is an observation in a “real life” professional setting. It must be organized as a normal day in the life of the candidate (= working day). One assessor could be acting as a “colleague” the other would assess from a distance. There could also be trained “colleagues” (must not have an assessor qualification), who “work with” the candidate in the observation environment. This is only necessary when a colleague is “physically” necessary to assess the competence at hand. One assessor can't oversee all activities, idealy there are at least two assessors, one who is observing from a distance and a second one observing close.
73		-Technical competence is relatively easy to assess. Knowledge behind the action can be assessed in most cases, if the test is prepared in the proper way. Competences are tested in the “group” working environment, as it is in reality. Several competences can almost always be assessed at one time. The proper atmosphere is very important.
74		-The assessments could be done at educational institutions with the necessary equipment.
	63	+Depending on the task, a preparation room with writing materials for the candidate, an examination room and technical equipment that is part of the task is required. In addition to the questioning assessor, a minute taker is advisable.
75	75
76	76	== Requirements for Assessors ==
77	77
78		-Assessors need competences for valid observations, such as those that can be acquired in observer training courses. They should have a basic knowledge of diagnostics, be able to deal with perceptual effects (e.g. errors of observation and assessment) and be able to recognize their own subjectivity. A professional competence is essential for the evaluation of the candidate's performance against the background of the assessment standard. It is also needed to construct a work situation appropriate to the competences to be assessed.
	67	+The assessor needs basic skills in conducting unbiased dialogue. He needs professional skills in order to be able to deduce appropriate competencies from the candidate's report according to the marking guide. Professional and pedagogical competences are necessary to create the tasks.
79	79
80	80	== Examples ==
81	81
82		-For the skill "Working on heights" a candidate should perform several activities on ladders, scaffolding, … Based on a checklist, his/her behaviour is observed.
	71	+An oral test can be used to test management skills. For example, the candidate should explain in a subtask how to check which materials and equipment are required for a given job.
83	83
84	84	== In Combination with ==
85	85
86		-This Method can be combined with a criterion focused interviews to fill the gaps or skills that have not been observed (not negative or positive). It can be combined with a multiple choice or open answer test for knowledge that is not made visible in practice.
	75	+Since the oral test does not test practical skills, a combination with corresponding methods like observation or role play is appropriate.
87	87
88	88	= References/Notes =
89	89
90		-* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]  (last 17.08.2020)
91		-* Jhpiego (2011): Simulation Training for Educators of Health Care Workers. Online: [[http:~~/~~/reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf>>http://reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf]]  (last 05.08.2020)
92		-* Multiprofessional Faculty Development (2012): Teaching and Learning in Simulated Environments. Online: [[https:~~/~~/faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments>>https://faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments]]  (last 05.08.2020)
93		-* Scottish Qualifications Authority (2019): Guide to Assessment. Online: [[https:~~/~~/www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf>>https://www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf]]  (05.08.2020)
94		-* Vincent-Lambert, C. / Bogossian, F. (2006): A guide for the assessment of
95		-* clinical competence using simulation. Online: [[https:~~/~~/pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf>>https://pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf]]  (last 05.08.2020)
	79	+* Joughin, G. (2010): A short guide to oral assessment. Online: [[https:~~/~~/www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/FilestoreDONOTDELETE/Filetoupload,213702,en.pdf>>https://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/FilestoreDONOTDELETE/Filetoupload,213702,en.pdf]] (last 21.07.2020).
	80	+* Assessment Resource Centre (2014): Types of Assessment Methods. Online: [[https:~~/~~/ar.cetl.hku.hk/am_orals.htm>>https://ar.cetl.hku.hk/am_orals.htm]] (last 21.07.2020).
	81	+* Iqbal, I. et al. (2010): The value of oral assessments: A review. Online: [[https:~~/~~/publishing.rcseng.ac.uk/doi/pdf/10.1308/147363510X511030>>https://publishing.rcseng.ac.uk/doi/pdf/10.1308/147363510X511030]] (last 21.07.2020).
	82	+* Centre of Expertise for Higher Education / University of Antwerp (n.d.): Oral tests and objective assessment. Online: [[https:~~/~~/www.uantwerpen.be/en/centres/centre-expertise-higher-education/didactic-information/teaching-tips-english/assessing-students/oral-tests-objectivity/>>https://www.uantwerpen.be/en/centres/centre-expertise-higher-education/didactic-information/teaching-tips-english/assessing-students/oral-tests-objectivity/]] (last 21.07.2020).
96	96
97	97	== AT ==
98	98