Bundesinstitut für Berufsbildung (BIBB)

Why competence diagnosis?

Even though the idea of a PISA study on vocational education and training (VET) has become somewhat jaded and has been the target of criticism (Bundesanzeiger No 104/2009), it still serves as a reminder of the value, in terms of education policy and economics, of discovering whether and in what ways investments in VET are cost-effective. The collection of contextual data, for example on training design in companies and vocational colleges, also permits a remarkably detailed analysis of the prerequisites and conditions for good and not-so-good training structures. This in turn can serve as the basis for strategic and political conclusions on the organisation and design of training processes. Large-scale competence diagnosis represents a formidable challenge to VET researchers. Two questions are at the heart of this article, which reports on the KOMET project conducted at the Bremen-based Institute for Vocational Education and Training (see box).

Key data on the KOMET project

Aim: To measure the occupational competence and professional commitment of electronic engineers

Test subjects: Since 2007, 700 test subjects in Hessen and Bremen from the dual system of vocational training and technical colleges; 800 test subjects in Peking from vocational sandwich courses and higher technical colleges, plus 40 instructors.

Consortium: Hessen Ministry of Education, the Bremen Senator for Education and the RIBB, Beijing.
Academic support by the I:BB Research Group at the University of Bremen, headed by Professor FELIX RAUNER; psychometric testing by Dr BIRGIT ERDWIEN of Erdwien Consulting, Bremen, Dr THOMAS MARTENS of DIPF, Frankfurt, and Dr LARS HEINEMANN of the I:BB at the University of Bremen; instructors, lecturers in specialised instructional techniques and instructor training: Hessen steering group and Bremen steering group as well as 30 assessors from Hessen and Bremen and 30 from Beijing.

1. Can the theoretical and standards-based KOMET competence model (see grid on page 25) be transformed into a psychometric model, and can the latter be used to measure vocational competence?
2. What experiences were gained during the three years of practical testing based on the KOMET methods?

Top of the page

What can and cannot be tested using large-scale competence diagnosis?

The establishment of the German Research Foundation's Priority Programme on Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes saw empirical educational researchers reach agreement, if they had not done so before, on a pragmatic definition of competence as a basis for empirical skills research (see KLIEME and LEUTNER, 2006). According to that definition, competence is a cognitive disposition to perform in a specific way in a specific context, since areas of competence can, to a certain extent, be generalised by extension to similar situations. Motivational and affective factors are disregarded (HARTIG and KLIEME, 2006, p. 18). 
On the basis of this definition, there is a distinction in vocational education and training between the cataloguing of areas of competence on the one hand and the checking of skills on the other.

• Examination (within the meaning of the German Vocational Training Act (Berufsbildungsgesetz)) of the skill requirements defined in the occupational profiles is intended to check all of the skills that are relevant to a person's aptitude to practise a particular occupation.
• The measurement of occupational competence in the sense of cognitive potential is intended to determine competence levels or profiles, particularly in the context of a comparison between different test groups.

The definition of the function of large-scale competence diagnosis makes it advisable to begin by highlighting what its methods cannot be used to measure (see Table 1).

Table 1 Aspects of ability and learning that cannot be measured by large-scale competence diagnosis

Top of the page

Measuring occupational competence

CONCEPTUAL FRAMEWORK

The measurement of occupational competence presupposes a theoretical and standards-based competence model that can be developed into a measurement model based on psychometric criteria (see MARTENS and ROST, 2009, pp. 95 et seq.). Competence models have the following functions: 

• to operationalise the fundamental criteria that have to be met in the context of problem-solving in the workplace as well as the associated principles and objectives of vocational education and training, and
• to provide sufficiently concrete guidelines for the formulation of test assignments.

For the identification of the basic criteria for work-related problem-solving in the KOMET project, a level of abstraction was chosen that makes it possible to define these criteria on an interdisciplinary basis. An analysis of occupational profiles and training curricula and of marking guides in cases where test assignments were used to assess the development of occupational competence in the GAB pilot project (see BREMER and HAASLER, 2004) resulted in the identification of eight criteria which occupational problem-solving must satisfy (see Table 2 next page). It is therefore logical to incorporate these criteria into a competence model for vocational training so that the reality of the workplace and the associated principles and objectives of vocational training can be reflected in the formulation of test assignments and in the assessment of their outcomes.

Table 2 Criteria for measuring occupational competence

THE TWO-DIMENSIONAL KOMET COMPETENCE MODEL

The two-dimensional KOMET competence model distinguishes between the expectations dimension (competence levels) and the content dimension. The eight competence criteria listed in Table 2 are assigned to the skill levels, which are of fundamental importance in assessing the proper performance of occupational tasks (see Figure 1). For the modelling of the expectations dimension (skill levels), the KOMET competence model is based on the four-stage scientific and technical literacy model proposed by BYBEE (1997), which was also used in the PISA project: 

• Nominal scientific and technological literacy: at this first level of competence, apprentices possess superficial conceptual knowledge which is not yet applicable in practice in terms of occupational ability. Their understanding of occupational terminology scarcely goes beyond the colloquial use of specialised terms. Since the level of occupational competence is not reached at this stage, it is not included in the grid. 
  
•  Functional literacy: at this stage, instrumental abilities in a discipline are based on the requisite elementary subject knowledge and skills, although their connections and their

• occupational applications have not yet been fully assimilated. 'Specialisation' amounts to non-contextual specialised knowledge and corresponding skills.

• Procedural literacy: occupational tasks are interpreted and addressed in terms of their relationship to work processes and workplace situations. Aspects such as economic viability and customer and process focus are now taken into account.

• Multidimensional literacy: at this level of competence, occupational tasks are considered in their full complexity with due regard to the diverse operational and social conditions in which they are performed and to divergent requirements in terms of the work process and its outcome.

The selection of typical substantive elements of an occupation can serve to decode the content dimension of occupational competence in terms of structures based on the development continuum 'from novice to expert' in a given discipline. Thereafter, a distinction is made between four stages of learning, in accordance with which the occupational activity and learning areas are structured on the basis of a development continuum into areas for beginners, advanced beginners, advanced trainees and experts. On this basis, content can be developed for the formulation of test assignments by reference to the subject's stage of training (see BECKER, 2009, pp. 241-242, and RAUNER et al., 2009(a), chapter 2.1). The test assignments developed and set in the framework of the KOMET project related to the developmental stage of readiness for employment.

TEST ASSIGNMENTS

The format of test assignments is based systematically on occupational practice and on the principle of a training system geared to procedural and creative competence. This gives rise to the following three fundamental requirements for the design of test assignments:

1. The degree of complexity must be set at such a level that contextual understanding - and not just a collection of distinct skills - can be measured.
2. Open-ended test assignments are necessary, since specialists must choose between alternative solutions when performing occupational tasks.
3. The content of test assignments must be characteristic and representative of the occupation in question. It is not necessary to cover the whole occupational profile in all of its disciplinary breadth in test assignments.
   

The assessment of open-ended test assignments with the aid of checklist items that have been used to operationalise the eight skill criteria requires the description of a feasible area within which the various candidate solutions can be situated. In the performance of these test assignments, the level of occupational competence is revealed in the way in which the whole feasible area covering the potential solutions is exhausted in terms of its eight competence dimensions.

A typical test assignment comprises a realistic description of a scenario. The specifications for professional problem-solving can be derived from this.

Example of a test assignment One test assignment, for example, describes the basic conditions for the conversion of an existing storage area into a drying area for painted machine parts, using a sketch to illustrate the scenario. Task: Draw up a concrete proposal for the conversion and the electrical wiring of the area. Present your design with the aid of practice-related documentation.
Provide a comprehensive and detailed explanation of your proposed solution. Should you have any questions, for example to the contractor, the user or another subcontractor, please write these down in preparation for consultations. Resources: All the usual resources, such as books of tables, specialised reference books, your own notes and your own pocket calculator, may be used to assist you in the performance of this task.

The grading of test results is ability-based. If a test subject does not go beyond the presentation and explanation of afeasible solution, his or her level of competence differs from that of another test subject who is able to develop a solution that also takes account of criteria with which the second and third levels of competence are defined. With the aid of 40 checklist items - five for each skill criterion - the assessors rate the solutions on the basis of the grading scale.

Figure   Structure of the expectations dimension: criteria and levels of occupational competence

Source: RAUNER et al., 2009(a), p. 91.

Top of the page

Psychometric validation of the competence and measurement model

The basic prerequisite for a model designed to measure open-ended test assignments is a high degree of consistency in the assessors' rating of solutions. In the KOMET project, this goal has been achieved in all of the subprojects conducted to date. To validate the psychometric quality of the competence and measurement model, an exploratory factoring analysis was conducted in the first instance in which, on the basis of the correlations between the 40 checklist items, those items were gathered into implicit structures, known as factors. On the whole, the outcome of the factoring analysis was highly consistent with the initial theoretical assumptions (for details, see Erdwien and Martens, 2009).
The assumptions underlying the theoretical competence model, namely that better performance in terms of procedural literacy cannot be achieved until functional literacy is firmly enough established and that a substantial degree of creative competence will not emerge until functional and procedural literacy are sufficiently well established, were also empirically conformed (op. cit.)

Problems and unresolved issues

• The influence of the motivating power of tests on test results was  underestimated. The findings of a study on test-driven motivation suggest that the timing of tests should be altered.
• A case study supports the hypothesis that the test venue (company premises or college) affects the problem-solving horizons of test subjects and therefore influences test results.
• The measurement of workers' identification with their occupation and their commitment to it and the collection of contextual data require an adjustment of national survey instruments for the purpose of international comparative studies.
  
  

 

Top of the page

Experience with large-scale competence diagnosis and future prospects

The experience gathered in three years of piloting the preparation, implementation and evaluation of cross-sectional and longitudinal studies involving 700 test subjects in 2008 and 1 500 in 2009 may be summed up as follows:

• Complex open-ended test assignments have proved their worth. The individual assignments, which test subjects are given a maximum of 120 minutes to complete, serve to measure occupational competence and, in the case of longitudinal studies, the development of occupational competence in realistic conditions, with reference to the principles and objectives of vocational education and in a manner that permits international comparison.
• The KOMET competence and measurement model lends itself to comparative studies in which groups of test subjects are drawn from various training systems (the test subjects have hitherto come from the dual system of vocational education and training, technical colleges and specialised vocational colleges).
• Educationalists from the relevant specialised disciplines estimate that it would be necessary to amend some 15 to 20 per cent of the checklist items in order to adapt the measurement model for occupations in the realm of personal service provision.
• The collection of data on identification with an occupation, commitment and the design of in-company and college training makes it possible to develop recommendations on vocational-training practice.

The KOMET project planning provides for the involvement of additional occupations and occupational fields in 2010.
One of the priorities of the project lies in the introduction of project-type forms of learning and self-appraisal methods for apprentices, instructors and trainers which are based on the KOMET competence and measurement model (see Katzenmeyer et al., 2009).
The results of the project suggest the need for gradual extension of VET research in the realm of large-scale competence diagnosis.

Top of the page

Bibliography

• BECKER, M.: Kompetenzmodell zur Erfassung beruflicher Kompetenz im Berufsfeld Fahrzeugtechnik.
  In: FENZL, C. u. a. (Hrsg.): Berufsarbeit von morgen in gewerblich-technischen Domänen. Forschungsansätze und Ausbildungskonzepte für die berufliche Bildung. Bielefeld 2009, S. 239-245
• BREMER, R.; HAASLER, B.: Analyse der Entwicklung fachlicher Kompetenz und beruflicher Identität in der beruflichen Erstausbildung. In: ZfPäd 50 (2004) 2, S. 162-181
• BYBEE, R. W.: Achieving scientific literacy: from purposes to practices. Portsmouth, NH.1997
• ERDWIEN, B.; MARTENS, TH.: Die empirische Qualität des Kompetenzmodells und des Ratingverfahrens.
  In: RAUNER, F. u. a. (Hrsg.): Messen beruflicher Kompetenzen. Band II. Ergebnisse KOMET 2008. 2. Aufl. Münster 2009, S. 62-82
• HARTIG, J.; KLIEME, E.: Kompetenz und Kompetenzdiagnostik.
  In: SCHWEIZER, K. (Hrsg.): Leistung und Leistungsdiagnostik. Berlin 2006, S. 127-143
• HEINEMANN, L.; RAUNER, F.: Engagement und Ausbildungsorganisation. i:BB. Bremen 2009
• KATZENMEYER, R. u.a.: Das KOMET-Kompetenzmodell in der Unterrichtspraxis.
  In: RAUNER, F. u. a. (Hrsg): Messen beruflicher Kompetenzen. Band II. Ergebnisse KOMET 2008. 2. Aufl., Münster 2009, S. 161-205
• KLIEME, E; LEUTNER, D: Kompetenzmodelle zur Erfassung individueller Kernergebnisse und zur Bilanzierung von Bildungsprozessen. Beschreibung eines neu eingerichteten Schwerpunktprogramms der DFG.
  In: ZfPäd 52 (2006) 6, S. 876-903
• MARTENS,TH.; ROST, J.:  Zum Zusammenhang von Struktur und Modellierung beruflicher Kompetenzen.
  In: RAUNER, F. u. a.: Messen beruflicher Kompetenzen. Grundlagen und Konzeption des KOMET-Projekts. Bd. I. 2. Auflage. Berlin 2009a
• RAUNER, F. u. a.:  Messen beruflicher Kompetenzen. Grundlagen und Konzeption des KOMET-Projekts. Bd. I. 2. Auflage. Berlin 2009a
• RAUNER, F. u. a.: Messen beruflicher Kompetenzen. Ergebnisse KOMET 2008. Bd. II. 2. Auflage. Reihe Bildung und Arbeitswelt. Berlin 2009b

Last modified on: October 20, 2010