Changes for page Written Test - Open Answers
Last modified by Ivan Kharitonov on 2022/01/30 17:25
Change comment:
There is no comment for this version
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 -O bservationinSimulated Environment1 +Oral Examination - Content
-
... ... @@ -4,10 +4,8 @@ 4 4 5 5 = Description = 6 6 7 -The simulatedenvironmentreflectsareal livesituation,but is standardised.Thismakesit possibletobuildinncentives forbehaviourorchoices.The situation canbe a"copy"ofa real live situation,butlsoa roleplay (formorebehaviour skills).Thecandidate is observed inthis simulated situation.7 +The oral test is a method of assessing the candidate's spoken word. Here assessor and candidate enter into a spoken interaction. An oral test can take place as a presentation (see AMD Presentation), interrogation or application. The oral test generally does not assess linguistic competence as such, but rather knowledge, understanding and problem solving ability. 8 8 9 -This method is used for skills that can be shown in the workspace. The assessment method allows to test very specific competences, as the environment can be controlled. Mainly for practical, observable skills. 10 - 11 11 ---- 12 12 13 13 = Quality Concepts = ... ... @@ -14,18 +14,20 @@ 14 14 15 15 === Validity === 16 16 17 -Since all factors are under control, the internal validity of this method is high. The method excludes unpredictability of situation and environment. So, it is easier to ensure safety. Very specific competencies can be tested. Since the behavior of people can change as a result of the observation situation (Hawthorne effect), internal validity is also threatened. This effect can be partially reduced if the work situation is only filmed (indirect observation). Since it is less a real-life situation, the external validity (transferability) of the observed behavior is lower. A good test will reflect real life situations in a controlled environment as much as possible. 15 +Oral tests could be valid for assessing e.g. understanding, applied problem solving, interpersonal competence, intrapersonal qualities (confidence, self-awareness, professionalism, ethics) and intergrated practice. 16 +Oral tests are valid if the candidate can demonstrate the competencies that are relevant to the validation process. It must therefore be checked whether the test queries what it should query (face validity). To check the content validity, it must be asked whether the discussed content is part of the required competencies, whether it goes beyond it or whether it is too barely discussed. In order to increase the construct validity of the oral test, the tasks must be in line with the theoretical understanding of the skills of event technology. 18 18 19 19 === Reliability === 20 20 21 -The quality of simulated environment observation depends on the accuracy and repeatability of the test setup. 22 -Simulated environments guarantee equal treatment of candidates, the result should be identical, wherever and by whatever assessors they are conducted. Therefore every candidate is assessed in an identical situation. 23 -One of the elements in this are well trained assessors and a levelling system, this avoids that assessment would be biased by assessors influenced by previous tests or looking outside the competences to for example not occupation related behaviour. 24 -The reliability is increased by the possibility to easily develop exact observable criteria. 20 +Reliability can be affected by candidate anxiety, candidate hearing or speech difficulties or assessor bias (e.g. due to gender, ethnicity, language). 21 +In order to standardize the test and increase comparability, a criteria list and an answer key should be created. The fact that the assessor can ask questions can affect reliability. An answer that was only given on request should possibly get fewer points. 22 +In order to increase the reliability, a competence should be tested in different ways. 23 +The use of multiple assessors also increases reliability. 24 +Due to the limited number of questions and the many disruptive factors in the interaction, the oral test is less reliable. 25 25 26 26 == Limitations == 27 27 28 - Development ofaassessment set-upis timeconsuming.28 +Since the time and question options are limited, only a limited amount of knowledge can be queried. Due to disturbing factors (excitement, stress, unusual situation) the method should be used in combination with other methods. 29 29 30 30 ---- 31 31 ... ... @@ -33,25 +33,18 @@ 33 33 34 34 == Tips == 35 35 36 -Organise the test in a way that the candidate feels at ease. If it is a tradition to have a cup of coffee at the start of a working day, include this in the startup of the test. 37 -Give the candidate time to discover the situation. 38 -Do not built in traps or tricky situations that hardly ever occur in real life. 39 -Be clear and open about the role and activities of the observers. Attention points can be: 36 +The candidates should be informed in advance about the framework conditions of the oral test: e.g. time period, location, requirements and items that are not included. 37 +A criteria list helps the assessors to objectively assess the results of the oral test. The assessors should be aware that inquiries can give unwanted tips. 38 +Candidates should be given sufficient time to answer. 40 40 41 -* Observers write also about positive points. 42 -* Observers are silent, because they keep a distance. 43 -* Observers will only stop the test in case of danger or overtime. 44 - 45 45 == Traps == 46 46 47 -I fthecandidateneedssupport, thesistantmust betrained tolimit theinterventionto whatthe candidaterequires andnot (aswe would do in reality)to take over thedecision-makingprocessor be proactive.48 - Thereis a riskthat theassessoris biased.Thatis why assessors should be professionals fromthefield of competencesbeing assessed. Assessors havetobe awarethat there aredifferent methodsto performa specific task and shouldtake distancefrom onepreferedmethod, forasfar asthegoal isreached.42 +It should not be the presentation and speaking skills that are assessed, but rather the competencies that are queried as part of the assessment process. 43 +If the oral test is not recorded, the results can be objected afterwards. 49 49 50 50 == Scoring Tools == 51 51 52 -Observing can be done through a list of observable criteria. The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation. 53 -As the situation is always identical, the scoring tool can be very specific and leave little room for interpretation. 54 -The final decision is made based on the link of the criteria with the competence and by comparing the observations of the different assessors. 47 +The results of the oral test can usually be evaluated immediately afterwards. A marking guide can be used here, which also serves as a framework for feedback to the candidate. Since it can be difficult to take notes and interview the candidate at the same time, it makes sense to use several assessors. In this case, one can record the conversation while the other is conducting the conversation. Audio or video recording of the oral test is also highly recommended. 55 55 56 56 ---- 57 57 ... ... @@ -59,40 +59,34 @@ 59 59 60 60 == Information for Standard == 61 61 62 -The standard must describe the specificsituations,incentives and expectedcomplexityof theskillstobe assessed.55 +The created marking guide should be aligned with the competence descriptions of the UNITs. The tasks and questions of the oral test should be precisely formulated in advance and each candidate should be asked the same questions. The candidate's answers are compared with the expected answers from the marking guide. In addition, cues and support that the assessors can provide are defined in advance. 63 63 64 64 == Development == 65 65 66 -The development of an observation in a simulated environment starts with the analysis of the skills that need to be evaluated. Since not every skill can be tested in all variations, representative situations are chosen to reflect the mastery of the general skill. The skills are built into a well-chosen scenario that reflects a real-life experience, but also integrates behavioural incentives and choices. The candidate is asked to perform a task, but the environment limits or alters the way the task is performed. In this way, the candidate must make his/her own decisions. 67 -The activities should reflect different contexts. Often a skill or behavior is built in twice to improve reliability and avoid "false positives". 68 -Assessment facilities must be tested and updated before they are used with "real" candidates. 59 +The development process involves the creation of tasks and questions that are designed to assess the relevant competencies. Expected answers and possible cues are set out in a marking guide. 69 69 70 70 == Needs/Set-Up == 71 71 72 -This is an observation in a “real life” professional setting. It must be organized as a normal day in the life of the candidate (= working day). One assessor could be acting as a “colleague” the other would assess from a distance. There could also be trained “colleagues” (must not have an assessor qualification), who “work with” the candidate in the observation environment. This is only necessary when a colleague is “physically” necessary to assess the competence at hand. One assessor can't oversee all activities, idealy there are at least two assessors, one who is observing from a distance and a second one observing close. 73 -Technical competence is relatively easy to assess. Knowledge behind the action can be assessed in most cases, if the test is prepared in the proper way. Competences are tested in the “group” working environment, as it is in reality. Several competences can almost always be assessed at one time. The proper atmosphere is very important. 74 -The assessments could be done at educational institutions with the necessary equipment. 63 +Depending on the task, a preparation room with writing materials for the candidate, an examination room and technical equipment that is part of the task is required. In addition to the questioning assessor, a minute taker is advisable. 75 75 76 76 == Requirements for Assessors == 77 77 78 - Assessorsneedcompetencesfor valid observations, suchasthosethat can be acquired in observertrainingcourses. They should have abasic knowledgeofdiagnostics, be able todealwith perceptualeffects(e.g. errors ofobservationndassessment)and be able torecognizetheirwn subjectivity. Aprofessionalcompetenceisessentialforthe evaluationofthe candidate'sperformanceagainst thebackground oftheassessmentstandard.Itisalsoneeded toconstructaworksituation appropriatetothecompetencesto be assessed.67 +The assessor needs basic skills in conducting unbiased dialogue. He needs professional skills in order to be able to deduce appropriate competencies from the candidate's report according to the marking guide. Professional and pedagogical competences are necessary to create the tasks. 79 79 80 80 == Examples == 81 81 82 - For the skill"Workingonheights"acandidate shouldperform severalctivitiesonladders,scaffolding, … Basedonachecklist,his/herbehaviour isobserved.71 +An oral test can be used to test management skills. For example, the candidate should explain in a subtask how to check which materials and equipment are required for a given job. 83 83 84 84 == In Combination with == 85 85 86 - This Method canbecombinedwith acriterionfocusedinterviewsofillthe gapsor skillsthathave notbeenobserved (not negativeorpositive).Itcan becombinedwithamultiplechoice or open answerestforknowledgethatisnot made visible inpractice.75 +Since the oral test does not test practical skills, a combination with corresponding methods like observation or role play is appropriate. 87 87 88 88 = References/Notes = 89 89 90 -* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]] (last 17.08.2020) 91 -* Jhpiego (2011): Simulation Training for Educators of Health Care Workers. Online: [[http:~~/~~/reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf>>http://reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf]] (last 05.08.2020) 92 -* Multiprofessional Faculty Development (2012): Teaching and Learning in Simulated Environments. Online: [[https:~~/~~/faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments>>https://faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments]] (last 05.08.2020) 93 -* Scottish Qualifications Authority (2019): Guide to Assessment. Online: [[https:~~/~~/www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf>>https://www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf]] (05.08.2020) 94 -* Vincent-Lambert, C. / Bogossian, F. (2006): A guide for the assessment of 95 -* clinical competence using simulation. Online: [[https:~~/~~/pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf>>https://pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf]] (last 05.08.2020) 79 +* Joughin, G. (2010): A short guide to oral assessment. Online: [[https:~~/~~/www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/FilestoreDONOTDELETE/Filetoupload,213702,en.pdf>>https://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/FilestoreDONOTDELETE/Filetoupload,213702,en.pdf]] (last 21.07.2020). 80 +* Assessment Resource Centre (2014): Types of Assessment Methods. Online: [[https:~~/~~/ar.cetl.hku.hk/am_orals.htm>>https://ar.cetl.hku.hk/am_orals.htm]] (last 21.07.2020). 81 +* Iqbal, I. et al. (2010): The value of oral assessments: A review. Online: [[https:~~/~~/publishing.rcseng.ac.uk/doi/pdf/10.1308/147363510X511030>>https://publishing.rcseng.ac.uk/doi/pdf/10.1308/147363510X511030]] (last 21.07.2020). 82 +* Centre of Expertise for Higher Education / University of Antwerp (n.d.): Oral tests and objective assessment. Online: [[https:~~/~~/www.uantwerpen.be/en/centres/centre-expertise-higher-education/didactic-information/teaching-tips-english/assessing-students/oral-tests-objectivity/>>https://www.uantwerpen.be/en/centres/centre-expertise-higher-education/didactic-information/teaching-tips-english/assessing-students/oral-tests-objectivity/]] (last 21.07.2020). 96 96 97 97 == AT == 98 98