Changes for page Criterion Based Interview

Last modified by Ivan Kharitonov on 2022/01/30 17:24

From version < 8.1 >

edited by Chris Van Goethem
on 2020/12/25 13:53

To version < 13.1 >

edited by Randell Greenlee
on 2021/11/09 10:12

< >

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (4 modified, 0 added, 0 removed)

Details

Page properties

Title

@@ -1,1
          +1,1 @@
--Observation in simulated environment
++Criterion Based Interview

Parent

@@ -1,1
          +1,1 @@
--Main.Sectoral Layer.Sectoral Layer Glossary.WebHome
++01 Sectoral Layer.Sectoral Layer Glossary.WebHome

Author

@@ -1,1
          +1,1 @@
--XWiki.ChrisVanGoethem
++XWiki.RandellGreenlee

Content

@@ -4,29
          +4,44 @@
  = Description =
--The simulated environment reflects a real live situation, but is standardised. This makes it possible to build in incentives for behaviour or choices. The situation can be a "copy" of a real live situation, but also a roleplay (for more behaviour skills). The candidate is observed in this simulated situation.
++|(((
++The Criterion Based Interview (also called STAR method) is comparable in many ways to the competency-based interview. It gives the candidate the opportunity, guided by directional questions, to demonstrate his/her skills, based on a concrete situation that happened in the own professional life.
--This method is used for skills that can be shown in the workspace. The assessment method allows to test very specific competences, as the environment can be controlled. Mainly for practical, observable skills.
++By focussing on the measurable aspects of a task, and narrowing, but deepening the focus, this is a good method to get a second opinion where competences did not become visible in testing or portfolio (not good or bad). The method needs highly skilled assessors. It is useful as extra tool to assess skills that are not observed (in positive nor negative way) It can also be used for situations that can’t be simulated, like an accident, audience panic or fire. This method can be also be used for problem solving, reflective or organising skills and for underpinning knowledge that doesn't become visible in the skill itself.
++)))
  ----
  = Quality Concepts =
--== Validity ==
++=== Validity ===
--Since all factors are under control, the internal validity of this method is high. The method excludes unpredictability of situation and environment. So, it is easier to ensure safety. Very specific competencies can be tested. Since the behavior of people can change as a result of the observation situation (Hawthorne effect), internal validity is also threatened. This effect can be partially reduced if the work situation is only filmed (indirect observation). Since it is less a real-life situation, the external validity (transferability) of the observed behavior is lower. A good test will reflect real life situations in a controlled environment as much as possible.
++|(((
++In the Criterion Based Interview, the basic assumption is that past behavior allows a prediction for future behavior. The validity depends on this basic assumption. The following value demonstrates the prediction rate for job success through the assessment method. It should be noted that in the assessment context, job success is only indirectly relevant in the context of validation.
++Standardized interviews have a validity of .71
++
++(Obermann 2018: 114). According to Schmidt & Hunter (2000), the prognostic validity of structured interviews is .51.
++
++The candidate wants to sell himself in the best way, which limits the validity.
++)))
++
  === Reliability ===
--The quality of simulated environment observation depends on the accuracy and repeatability of the test setup.
--Simulated environments guarantee equal treatment of candidates, the result should be identical, wherever and by whatever assessors they are conducted. Therefore every candidate is assessed in an identical situation.
--One of the elements in this are well trained assessors and a levelling system, this avoids that assessment would be biased by assessors influenced by previous tests or looking outside the competences to for example not occupation related behaviour.
--The reliability is increased by the possibility to easily develop exact observable criteria.
++|(((
++For hiring interviews, the interrater reliability is .68, which corresponds to a good or substantial reliability. Even if the assessment context in the validation process differs from the recruitment process, it can certainly be said that there is good reliability when the criteria-based interviews are carried out correctly. The reliability is increased by the questions being pre-formulated in advance and the wording remaining the same. This increases comparability with other interviews. To ensure objectivity in the evaluation, the interviews should be recorded and evaluated by several assessors.
++The assessors / observers should be extensively trained to ensure evaluation and interpretation, to ensure comparability and objectivity. It should be clear which behaviors can be attributed to certain skills. So, reliability is enforced by a good scoring structure.
++
++The behavior of the interlocutors is standardized by means of a written instruction. This is a prerequisite for objectivity, which in turn ensures that different results are due to the performance of the participants and not to variations in the interlocutors.
++)))
++
  == Limitations ==
--Development of a assessment set-up is time consuming.
++The interview does not allow to observe/confirm the capacity of the participant to actually carry out a task. The method is very intensive and time consuming for assessors.
++It can only be used for a limited set of competences.
++
  ----
  = Considerations =
@@ -33,26
          +33,35 @@
  == Tips ==
--Organise the test in a way that the candidate feels at ease. If it is a tradition to have a cup of coffee at the start of a working day, include this in the startup of the test.
--Give the candidate time to discover the situation.
--Do not built in traps or tricky situations that hardly ever occur in real life.
--Be clear and open about the role and activities of the observers. Attention points can be:
++|(((
++The interview should follow a biographical approach, which helps the assessor to understand the career path of the participant.
--* Observers write also about positive points.
--* Observers are silent, because they keep a distance.
--* Observers will only stop the test in case of danger or overtime.
++Ensure that all assessors use the same path/structure. The variations should be limited. Continue questioning untill you reach the desired focus. Use simple open questions (e.g. who, when, where, why, how, for how long, …). Don't judge in your answers or sub questions. Listen actively, show interest, listen to what is said in between the lines, ask deeper if needed, but give the candidate the lead.
++Disruptive factors should be minimized by a high degree of standardization.
++)))
++
  == Traps ==
--If the candidate needs support, the assistant must be trained to limit the intervention to what the candidate requires and not (as we would do in reality) to take over the decision-making process or be proactive.
--There is a risk that the assessor is biased. That is why assessors should be professionals from the field of competences being assessed. Assessors have to be aware that there are different methods to perform a specific task and should take distance from one prefered method, for as far as the goal is reached.
++|(((
++The personal interests and opinions of the interviewer can interfere with the interview process (bias). To avoid this distortion, the interviewer has to make himself aware of his / her perception tendencies. Attractiveness, size, eloquence, humor, clothing style, nervousness and gender
++of the participant can influence the assessment of the assessors, even though they have no information about the tested competence. The context can also influence the candidate.
++
++It should be noted that this interview should not assume the role of an oral test, even though the boundaries might be blurry at some points.
++
++There is a risk that, especially in sub questions, the assessor suggests the answers unconsciously. The direction of the interview is depending on the candidate and the sub questions of the assessors.
++)))
++
  == Scoring Tools ==
--Observing can be done through a list of observable criteria. The criteria should be derived from the sectoral layer skills, in other words, they are a concretisation of the visible, observable result of the skill in a specific situation.
--As the situation is always identical, the scoring tool can be very specific and leave little room for interpretation.
--The final decision is made based on the link of the criteria with the competence and by comparing the observations of the different assessors.
++Before the interview, the assessor determines which
++answers to the questions receive positive and which answers receive negative points. The "positive indicators" and the "negative indicators" can be compared in a table. The "negative indicators" can also be divided into "minor negative indicators" and "decisive negative indicators". The respective extent to which the indicators match with the answers is rated on a scale of 0-4 (0 = no evidence; 1 = poor; 2 = areas of concern; 3 = satisfactory; 4 = good to excellent). A good list of what is expected needs to be used. The list should reflect the skills assessed.
++
++
++If the method is used as backup, scoring lists from other methods can be used.
++
  ----
  = Implementation =
@@ -59,40
          +59,48 @@
  == Information for Standard ==
--The standard must describe the specific situations, incentives and expected complexity of the skills to be assessed.
++If the interview is foreseen in the assessment, the context, the focus, the time and criteria are described. If the method is used as backup, it should be mentioned with other methods.
  == Development ==
--The development of an observation in a simulated environment starts with the analysis of the skills that need to be evaluated. Since not every skill can be tested in all variations, representative situations are chosen to reflect the mastery of the general skill. The skills are built into a well-chosen scenario that reflects a real-life experience, but also integrates behavioural incentives and choices. The candidate is asked to perform a task, but the environment limits or alters the way the task is performed. In this way, the candidate must make his/her own decisions.
--The activities should reflect different contexts. Often a skill or behavior is built in twice to improve reliability and avoid "false positives".
--Assessment facilities must be tested and updated before they are used with "real" candidates.
++|(((
++The main development is in the training of the assessors to use the method. But some scheme with questions and sub questions can support the assessors.
++The method is based on an interviewing technique using principles of the STARR method:
++
++* S(ituation): What was the situation? - description of a past work situation
++* T(ask): What was your task? - clarification of the responsibilities of the candidate
++* A(ction): What actions did you take, what did you do? - explanation of the performed action
++* R(esult): What was the result, what happened? - statement about the results
++* R(eflection): What did you learn? - evaluation of the situation from his/her perspective today.
++)))
++|(((
++
++)))
++
  == Needs/Set-Up ==
--This is an observation in a “real life” professional setting. It must be organized as a normal day in the life of the candidate (= working day). One assessor could be acting as a “colleague” the other would assess from a distance. There could also be trained “colleagues” (must not have an assessor qualification), who “work with” the candidate in the observation environment. This is only necessary when a colleague is “physically” necessary to assess the competence at hand. One assessor can't oversee all activities, idealy there are at least two assessors, one who is observing from a distance and a second one observing close.
--Technical competence is relatively easy to assess. Knowledge behind the action can be assessed in most cases, if the test is prepared in the proper way. Competences are tested in the “group” working environment, as it is in reality. Several competences can almost always be assessed at one time. The proper atmosphere is very important.
--The assessments could be done at educational institutions with the necessary equipment.
++The interview needs to be conducted by two assessors in order to avoid bias of the result (sympathy etc.) or might even require a written transcript for evaluation. It needs a quiet room (with a table, chairs, ...) and plenty of time.
  == Requirements for Assessors ==
--Assessors need competences for valid observations, such as those that can be acquired in observer training courses. They should have a basic knowledge of diagnostics, be able to deal with perceptual effects (e.g. errors of observation and assessment) and be able to recognize their own subjectivity. A professional competence is essential for the evaluation of the candidate's performance against the background of the assessment standard. It is also needed to construct a work situation appropriate to the competences to be assessed.
++The assessor needs basic skills in conducting unbiased interviews. He needs professional skills in order to be able to deduce appropriate competencies from the candidate's report.
  == Examples ==
--For the skill "Working on heights" a candidate should perform several activities on ladders, scaffolding, … Based on a checklist, his/her behaviour is observed.
++The situation for the interview, which is assumed, may be the observance of safety precautions during the planning of previous light settings.
  == In Combination with ==
--This Method can be combined with a criterion focused interviews to fill the gaps or skills that have not been observed (not negative or positive). It can be combined with a multiple choice or open answer test for knowledge that is not made visible in practice.
++|A post-box exercise or role play would be required in addition. Often a competence focussed interview is used as backup if other test didn't show certain competences (not positive or negative).
  = References/Notes =
--* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evc-toolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]  (last 17.08.2020)
--* Jhpiego (2011): Simulation Training for Educators of Health Care Workers. Online: [[http:~~/~~/reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf>>http://reprolineplus.org/system/files/resources/simulation_facilitatorsguide.pdf]]  (last 05.08.2020)
--* Multiprofessional Faculty Development (2012): Teaching and Learning in Simulated Environments. Online: [[https:~~/~~/faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments>>https://faculty.londondeanery.ac.uk/e-learning/teaching-clinical-skills/teaching-and-learning-in-simulated-environments]]  (last 05.08.2020)
--* Scottish Qualifications Authority (2019): Guide to Assessment. Online: [[https:~~/~~/www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf>>https://www.sqa.org.uk/files_ccc/Guide_To_Assessment.pdf]]  (05.08.2020)
--* Vincent-Lambert, C. / Bogossian, F. (2006): A guide for the assessment of
--* clinical competence using simulation. Online: [[https:~~/~~/pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf>>https://pdfs.semanticscholar.org/bda7/dae4871a49e19fd2cc186823379518e39192.pdf]]  (last 05.08.2020)
++* Catalogus Assessmentmethodes voor EVC, Agentschap Hoger Onderwijs, volwassenenonderwijs, Kwalificaties en Studietoelagen, Ministery of education and training of the Flemish community (2015). Online: [[http:~~/~~/www.erkennenvancompetenties.be/evc-professionals/evctoolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf>>http://www.erkennenvancompetenties.be/evc-professionals/evctoolbox/bestanden/catalogus-assessmentmethodes-evc-2015.pdf]]. (last 17.08.2020)
++* ISC Professional (2016): Competency-Based Interviews. Online: [[https:~~/~~/www.interview-skills.co.uk/freeinformation/interview-guide/competency-basedinterviews>>https://www.interview-skills.co.uk/freeinformation/interview-guide/competency-basedinterviews]] (last 27.7.2020)
++* Obermann, C. (2018): Assessment Center. Entwicklung, Durchführung, Trends. Mit neuen originalen AC-Übungen.
++6., vollständig überarb. u. erw. Aufl. Wiesbaden: Springer Fachmedien.
++* Schmidt, F. L. & Hunter, J. E. (2000). Messbare Personenmerkmale:Stabilität, Variabilität und Validität zur Vorhersage zukünftiger Berufsleistung und berufsbezogenen Lernens. In: M. Kleinmann & B. Strauß (Hrsg.), Potentialfeststellung und Personalentwicklung (S. 15–43). Göttingen: Verlag für Angewandte Psychologie.
  == AT ==

Changes for page Criterion Based Interview

Summary

Details

TeBeVAT Menu