Summary

• Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Title

...	...	@@ -1,1 +1,1 @@
1		-Oral Examination
	1	+Written Test - Multiple Choice

Content

...	...	@@ -4,9 +4,13 @@
4	4
5	5	= Description =
6	6
7		-The simulated environment reflects a real live situation, but is standardised. This makes it possible to build in incentives for behaviour or choices. The situation can be a "copy" of a real live situation, but also a roleplay (for more behaviour skills). The candidate is observed in this simulated situation.
	7	+The multiple-choice test consists of questions where only one answer can be correct. In contrast, the multiple-response tests consists of questions where several answers can be correct. Neither craft skills nor creativity can be measured with either type of test. Nevertheless, it is not only possible to query factual knowledge. Unless otherwise stated, both test types are summarized below under the term "multiple-choice test". Following cognitive performance levels can be tested via multiple-choice tests:
8	8
9		-This method is used for skills that can be shown in the workspace. The assessment method allows to test very specific competences, as the environment can be controlled. Mainly for practical, observable skills.
	9	+* Reproduction of stored knowledge
	10	+* Reorganization: Learned knowledge is processed and arranged independently.
	11	+* Transfer: Basic principles are transferred to new, similar tasks.
	12	+* Problem-solving thinking: Tasks with new questions and aspects are solved.
	13	+
10	10
11	11	----
12	12
...	...	@@ -14,18 +14,15 @@
14	14
15	15	=== Validity ===
16	16
17		-Since all factors are under control, the internal validity of this method is high. The method excludes unpredictability of situation and environment. So, it is easier to ensure safety. Very specific competencies can be tested. Since the behavior of people can change as a result of the observation situation (Hawthorne effect), internal validity is also threatened. This effect can be partially reduced if the work situation is only filmed (indirect observation). Since it is less a real-life situation, the external validity (transferability) of the observed behavior is lower. A good test will reflect real life situations in a controlled environment as much as possible.
	21	+The validity is not necessarily present. The main problem is the discrepancy between complex cognitive processes and simply ticking an answer. Multiple-Choice Tests can evaluate a competence to act if they require the quick assessment of different facets of a complex subject.
18	18
19	19	=== Reliability ===
20	20
21		-The quality of simulated environment observation depends on the accuracy and repeatability of the test setup.
22		-Simulated environments guarantee equal treatment of candidates, the result should be identical, wherever and by whatever assessors they are conducted. Therefore every candidate is assessed in an identical situation.
23		-One of the elements in this are well trained assessors and a levelling system, this avoids that assessment would be biased by assessors influenced by previous tests or looking outside the competences to for example not occupation related behaviour.
24		-The reliability is increased by the possibility to easily develop exact observable criteria.
	25	+The objectivity of Multiple-Choice Tests is significantly higher than for classic written exams. The reason for this is that the correction is independent of the correcting person and is often automated. In addition, well-formulated questions go hand in hand with high reliability. If the tests are carried out regularly and with a large number of test persons, they are particularly economical.
25	25
26	26	== Limitations ==
27	27
28		-Development of a assessment set-up is time consuming.
	29	+Creativity or craft skills cannot be tested with multiple choice tests.
29	29
30	30	----
31	31
...	...	@@ -33,25 +33,15 @@
33	33
34	34	== Tips ==
35	35
36		-Organise the test in a way that the candidate feels at ease. If it is a tradition to have a cup of coffee at the start of a working day, include this in the startup of the test.
37		-Give the candidate time to discover the situation.
38		-Do not built in traps or tricky situations that hardly ever occur in real life.
39		-Be clear and open about the role and activities of the observers. Attention points can be:
	37	+To reduce the possibility of scoring by gambling, the alternative answers should be of high quality. According to studies, three answer options are sufficient to make successful gambling unlikely.
40	40
41		-* Observers write also about positive points.
42		-* Observers are silent, because they keep a distance.
43		-* Observers will only stop the test in case of danger or overtime.
44		-
45	45	== Traps ==
46	46
47		-If the candidate needs support, the assistant must be trained to limit the intervention to what the candidate requires and not (as we would do in reality) to take over the decision-making process or be proactive.
48		-There is a risk that the assessor is biased. That is why assessors should be professionals from the field of competences being assessed. Assessors have to be aware that there are different methods to perform a specific task and should take distance from one prefered method, for as far as the goal is reached.
	41	+Disadvantages of too many alternative answers are a greater effort in the designing, risk of not plausible distractors (distractor = false answer option) and revealing of context hints for other tasks. Depending on the task, five or more answer options can also be useful.
49	49
50	50	== Scoring Tools ==
51	51
52		-Observing can be done through a list of observable criteria. The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation.
53		-As the situation is always identical, the scoring tool can be very specific and leave little room for interpretation.
54		-The final decision is made based on the link of the criteria with the competence and by comparing the observations of the different assessors.
	45	+A major advantage of Multiple-Choice / response tests (short: MC-tests) is that they can be evaluated easily. In addition, the quality of particular questions can be determined statistically. Bad formulated questions can be recognized as such. As already mentioned, only one answer can be correct for Multiple-Choice questions. If the right answer is marked, one point will be awarded. If a wrong answer or no answer is marked, no point will be awarded. The awarding of points can be different for Multiple-Response tests. Every correct marked answer scores one point (right answers marked, or false answers not marked). Every incorrect marked answer brings one minus point (right answers not marked or false answers marked). The difference determines the total score of the question. If the difference is negative, the score is 0 Points.
55	55
56	56	----
57	57
...	...	@@ -59,40 +59,37 @@
59	59
60	60	== Information for Standard ==
61	61
62		-The standard must describe the specific situations, incentives and expected complexity of the skills to be assessed.
	53	+For a high degree of standardization, the test sheets should be formulated in a clear and understandable way and used without modification. Language barriers should be taken into account.
63	63
64	64	== Development ==
65	65
66		-The development of an observation in a simulated environment starts with the analysis of the skills that need to be evaluated. Since not every skill can be tested in all variations, representative situations are chosen to reflect the mastery of the general skill. The skills are built into a well-chosen scenario that reflects a real-life experience, but also integrates behavioural incentives and choices. The candidate is asked to perform a task, but the environment limits or alters the way the task is performed. In this way, the candidate must make his/her own decisions.
67		-The activities should reflect different contexts. Often a skill or behavior is built in twice to improve reliability and avoid "false positives".
68		-Assessment facilities must be tested and updated before they are used with "real" candidates.
	57	+When preparing the test, high quality distractors should be chosen that are plausible without fooling the candidate. The test should be focused on the competencies to be measured and not just on smartness. A solution sheet gives the necessary information about the correct answers.
69	69
70	70	== Needs/Set-Up ==
71	71
72		-This is an observation in a “real life” professional setting. It must be organized as a normal day in the life of the candidate (= working day). One assessor could be acting as a “colleague” the other would assess from a distance. There could also be trained “colleagues” (must not have an assessor qualification), who “work with” the candidate in the observation environment. This is only necessary when a colleague is “physically” necessary to assess the competence at hand. One assessor can't oversee all activities, idealy there are at least two assessors, one who is observing from a distance and a second one observing close.
73		-Technical competence is relatively easy to assess. Knowledge behind the action can be assessed in most cases, if the test is prepared in the proper way. Competences are tested in the “group” working environment, as it is in reality. Several competences can almost always be assessed at one time. The proper atmosphere is very important.
74		-The assessments could be done at educational institutions with the necessary equipment.
	61	+The test can be performed either with a pen and paper or electronically on a computer.
75	75
76	76	== Requirements for Assessors ==
77	77
78		-Assessors need competences for valid observations, such as those that can be acquired in observer training courses. They should have a basic knowledge of diagnostics, be able to deal with perceptual effects (e.g. errors of observation and assessment) and be able to recognize their own subjectivity. A professional competence is essential for the evaluation of the candidate's performance against the background of the assessment standard. It is also needed to construct a work situation appropriate to the competences to be assessed.
	65	+The design of the test requires a high level of technical expertise in order to develop useful distractors. For the evaluation of the test no special skills are required.
79	79
80	80	== Examples ==
81	81
82		-For the skill "Working on heights" a candidate should perform several activities on ladders, scaffolding, … Based on a checklist, his/her behaviour is observed.
	69	+For example, a multiple choice test can be used to test knowledge about the use of special effects.
83	83
84	84	== In Combination with ==
85	85
86		-This Method can be combined with a criterion focused interviews to fill the gaps or skills that have not been observed (not negative or positive). It can be combined with a multiple choice or open answer test for knowledge that is not made visible in practice.
	73	+Written Test - Open Answers
87	87
88	88	= References/Notes =
89	89
90		-* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]  (last 17.08.2020)
91		-* Jhpiego (2011): Simulation Training for Educators of Health Care Workers. Online: [[http:~~/~~/reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf>>http://reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf]]  (last 05.08.2020)
92		-* Multiprofessional Faculty Development (2012): Teaching and Learning in Simulated Environments. Online: [[https:~~/~~/faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments>>https://faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments]]  (last 05.08.2020)
93		-* Scottish Qualifications Authority (2019): Guide to Assessment. Online: [[https:~~/~~/www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf>>https://www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf]]  (05.08.2020)
94		-* Vincent-Lambert, C. / Bogossian, F. (2006): A guide for the assessment of
95		-* clinical competence using simulation. Online: [[https:~~/~~/pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf>>https://pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf]]  (last 05.08.2020)
	77	+* Baghaei, P. / Amrahi, N. (2011): The effects of the number of options on the psychometric characteristics of Multiple-Choice items. In: Psychological Test and Assessment Modeling. 53 (2), p. 192-211.
	78	+* CTL Center for Teaching and Learning / Universität Wien (2015): Erstellen von schriftlichen, mündlichen und Multiple-Choice-Prüfungen. Online: [[https:~~/~~/ctl.univie.ac.at/fileadmin/user_upload/z_ctl/Qualitaet_von_Studien/Qualitaet_von_Pruefungen/170314_Erstellen_von_schriftlichen_muendlichen_MC_Pruefungen.pdf >>https://ctl.univie.ac.at/fileadmin/user_upload/z_ctl/Qualitaet_von_Studien/Qualitaet_von_Pruefungen/170314_Erstellen_von_schriftlichen_muendlichen_MC_Pruefungen.pdf]](last 25.05.2020).
	79	+* Elsa / Leibniz Universität Hannover (n.d.): Erstellen und Bewerten von Multiple-Choice-Aufgaben. Online: [[https:~~/~~/www.zqs.uni-hannover.de/fileadmin/zqs/PDF/E-Learning/elsa_handreichung_zum_erstellen_und_bewerten_von_mc-fragen_2013.pdf >>https://www.zqs.uni-hannover.de/fileadmin/zqs/PDF/E-Learning/elsa_handreichung_zum_erstellen_und_bewerten_von_mc-fragen_2013.pdf]](last: 25.05.2020).
	80	+* Kubinger, K. (2014): Gutachten zur Erstellung “gerichtsfester” Multiple-Choice-Prüfungsaufgaben. In: Psychologische Rundschau. 65 (3), p. 169-178.
	81	+* Rodriguez, M. (2005): Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. In: Educational Measurement. Issues and Practice. 24 (2), p. 3-13.
	82	+* Universität Kassel (n.d.): Handreichung für Klausuren mit Aufgaben nach dem Antwort-Wahl-Verfahren (Single-Choice/Multiple-Choice). Online: [[http:~~/~~/www.uni-kassel.de/einrichtungen/fileadmin/datas/einrichtungen/scl/E-Klausuren/Handreichung_Antwort_Wahl_Aufgaben_final.pdf>>http://www.uni-kassel.de/einrichtungen/fileadmin/datas/einrichtungen/scl/E-Klausuren/Handreichung_Antwort_Wahl_Aufgaben_final.pdf]] (last: 25.05.2020).
	83	+* Universität Zürich (n.d.): Hochschuldidaktik A – Z. Multiple-Choice-Prüfungen. Online: [[http:~~/~~/www.hochschuldidaktik.uzh.ch/dam/jcr:ffffffff-9a08-8cca-0000-00002cfe461f/A_Z_Multiple-Choice.pdf >>http://www.hochschuldidaktik.uzh.ch/dam/jcr:ffffffff-9a08-8cca-0000-00002cfe461f/A_Z_Multiple-Choice.pdf]](last: 02.06.2020).
96	96
97	97	== AT ==
98	98