Bundesinstitut für Berufsbildung (BIBB)

IT examination: The debate continues

BIBB using online survey to determine opinions on final examinations for the IT occupations

The model for the final examination in the four IT occupations has been a source of heated debate. Six years after it was launched and the third year after the first regular round of examinations, the waves are not quite as high as they were at first, but the debate over the pros and cons of the new forms of examination have yet to recede completely. This is also demonstrated by the lively response to the online survey carried out by BIBB on the IT final examination.

The examination system in the system of dual vocational training has changed significantly over the last six years, especially as a result of the launch of more innovative, practically and process-oriented examination procedures. With these an attempt is being made to examine employability skills in a more discriminating, revealing way. Among these new developments, the examination forms "company project work" and "holistic assignment" have attracted special attention. Both forms of examinations are elements of the final examinations in the four IT (information and technology) occupations created in 1997. Diagram 1 shows the structure of the IT final examination.

Diagram 1: Structure of the final examination in the IT occupations

The first regular final examinations based on the new structure for the IT occupations took place in 2000. At the same time as this round of examinations was taking place, the first evaluation was carried out of the examination structure and its implementation. This showed that all of the parties involved in the examination procedure accepted the model per se.

At the same time, the evaluation also revealed various critical aspects to the new examination forms, inter alia the implementation of the holistic strategy in the written examination assignments and assuring comparability in the project work.

A second evaluation of the IT final examination in the summer of this year is looking at whether and what has changed to the positive and what has changed to the negative in the intervening period and what has brought these changes about.

In this context BIBB will be carrying out an online survey on the IT final examinations between March 15 and July 30, 2003. The online survey for the most part seeks to shed light on experience with and attitudes towards the examination model as well as the two forms of examination, project work and holistic assignment, to obtain a general picture of the atmosphere and generate information on problems relating to the structure of the examination which have been overcome, which have not been overcome or even new problems with the examination structure. At the same time an effort was made to generate data on the structure of companies providing training in the IT occupations.

How was the online survey designed?

The start-up site for the online survey could be accessed either via the BIBB homepage or the URL http://www.bibb.de/itforum/befragung. At the heart of the start-up site was a so-called opening questionnaire inviting interested persons to issue free statements as they see fit on the IT examination. In addition, questionnaires aimed at certain target groups can also be called up on the start-up site (teachers and trainers, trainees, vocational school teachers and examiners). In addition to the possibility to freely state one's opinion, these forms also contain questions on the examination and background data. To attain as high a response rate as possible on these target group-specific questionnaires, a notice calling attention to these questionnaires pops up when the initial questionnaire is sent off. References to additional literature, BIBB projects and initiatives relating to the topic along with contacts rounded off the spectrum (cf. annex - set-up of the start-up page for the online survey).

The online questionnaire was designed to be an explorative study. It is directed at everyone involved in a final IT examination or who is interested in such. It did not restrict the group of people it is aimed at or access rights by means of a password. This was done deliberately - in spite of the risk of receiving "unusable responses" - in order to reach as many interested persons as possible. Procedures were developed to check internal plausibility, however, in an effort to avoid "bogus" responses.

And what was the response to the online survey?

The online survey was received by a large group of persons through press releases, information flyers and with the support of numerous associations and institutions which drew attention to the online survey on their homesites and in other media. These initiatives led to a very encouraging response to the online survey: over 13,000 hits were registered on the start-up site for the survey between March and the end of July.
The following statistics give some idea of the responses: 94 teachers at vocational training schools, 252 company trainers, 287 examiners and 1,536 trainers took advantage of the questionnaires for their specific target groups. On top of this, 1,296 opening questionnaires were returned; almost half of these came from the trainees. Thus a total of 3,465 valid responses were received to the online survey.

About two-thirds of all responses contained in part detailed statements on the examination model, on one or both forms of examination, comments on how the examination was organised, a description of experience with the examinations and similar. Only a small percentage of these statements were exclusively negative. In most cases there was a critical but constructive analysis of aspects found wanting, while in many cases suggestions for improvements were made. The number of statements focusing on the strengths of the examination design or the forms of examinations or describing positive experience was significantly lower.

What are the main messages in the statements?

The evaluation of numerous statements and the responses to the mandatory questions have not yet been completed in its entirety. Nevertheless certain core topics regarding which a large number of statements were made can be identified already at this point.

Here we should once again expressly note that the online survey was explorative in nature and thus makes no claim to being representative, complete or generalisable. At issue, rather, were individual, subjective experience, impressions, and assessments of the atmosphere and mood. Hypotheses were in turn generated on this basis which are to be examined in the further course of the evaluation study.

Core topics on project work

a) Lack of transparency in processes and criteria

Trainers, trainees, teachers and examiners all criticised the fact that there is still no uniform understanding of which requirements a contract must meet in order to be suitable for an examination of company project work even in the fourth year following the first regular final examination. The reasons for the approval or rejection of a project application are accordingly difficult to understand, which in particular makes it almost impossible for trainers to profit from past experience.
Both trainers and trainees repeatedly stated that they had the impression that requirements applying to the projects had in the meantime reached a level which was no longer compatible with the time available for project processing and documentation. It is held that the solution to this problem is to be found in a uniform national code of requirements which are spelled out in reference projects wherever this is possible.

The standards which apply to the documentation, the presentation and the technical discussion part of examinations were comparily criticized. Many participants felt that skills were less important than formal matters with respect to the documentation, and that in the presentation the use of equipment was more important than creating a content with an interesting structure which was internally consistent and which conformed with the interests of the target group. In addition there is still a good deal of uncertainty as to what the subject of the presentation is supposed to be - the job contract under processing, the processing procedure itself or the result attained. Here as well the majority of the persons surveyed were in favour of creating uniform guidelines. The technical discussion was assigned a high level of importance by the examiners as well as the persons being tested when it came to demonstrated in-depth knowledge. There was still disagreement, however, on whether or not and to what extent the technical discussion could remove people from the project topic.

b) Role of documentation, presentation and technical discussion

The initial evaluation produced a very heterogeneous picture regarding how project documentation, project presentation and the technical discussion were to be weighted in examination part A. It appears now, however, that a uniform line is beginning to crystallise. The majority of persons responding to the survey who addressed this point were in favour of a much lower weight being assigned to the documentation, and in some cases were even in favour of documentation not receiving any weight at all. Presentations and technical discussions are to be significantly expanded - including in terms of time, with most of the respondents suggesting a time totalling 45 to 60 minutes.

Core topics on comprehensive assignments

a) Seeking a uniform level

While the current design of the comprehensive tasks combined with the possibility to leave out one of the six steps met with an overwhelmingly positive response, the content-related design of the written examination assignments continued to be a stumbling block. It was first of all criticised that thus far no comprehensive assignments have been developed which cover a discriminating spectrum. Instead, the comprehensive assignments have addressed topics which vary widely from one examination to another, while the respectively selected topics have been evaluated very narrowly and in an isolated manner and all other content of relevance to the trade have been pretty much ignored. Secondly criticism focused on the criticism of the level of requirements and difficulty in the comprehensive assignments, which vary greatly from one examination to the other. Numerous trainees in addition expressed the fear that this could have a negative effect on the vocational degree attained. For this reason, many respondents were in favour of a constant level of requirements commensurate with the qualifications of skilled workers.

b) Demarcate the lines between occupations more clearly

Many of the respondents felt that the distinction made between core and technical qualifications did not make any sense; here there was considerable opposition to the comprehensive concept. The fact that qualifications are attained in different occupations while the lines separating these occupations are once again lost in whole or in part in the examination through the comprehensive assignment II (this is the same for all four IT occupations and is aimed at checking general core qualifications common to all occupations) met with even more criticism. It was admitted that there is a certain overlapping between the qualification profiles which can be called core qualifications, but these core qualifications are reflected in the processes contained in specific vocations. Given this fact, almost all of the respondents identified this aspect and urged that the comprehensive assignments be structured along the lines of specific vocations.

Flow of information: a key topic

The lack of transparency with regard to requirements and criteria applying to project work are an additional key topic which the respondents analysed: the conveyance of information. One reason explaining the numerous problems which have cropped up in the IT final examination in the past and at present is in the view of the majority of the respondents the lack of coordination between the training companies, vocational schools and the chambers. Coordination between companies and vocational schools in preparing for the examination, for example, was said to be in need of improvement. Without these - according to many respondents - contents, techniques and methods of relevance to the examination would either be duplicated or not conveyed at all. Deficits in the flow of information between the chamber and the company were frequently beset by a lack of clarity regarding deadlines for the filing of applications and execution of project work or the form which applications are supposed to use (online or offline).

A look ahead

The online survey, which was initially included in the study design in the second evaluation of the IT final examination on an experimental basis, has proven to be a suitable tool for identifying fields of topics and problems. This will be explored in the further course of the evaluation studies together with other innovative methods to test the hypotheses derived on cause-effect correlations and the type of problems identified (content-related, organisational, structural or conceptual).
These further research methods include
- a complete survey, recently completed, of all chambers of commerce in the Federal Republic of Germany to determine basic data of relevance to training and examinations.
- interviews with experts and
- analyses of comprehensive tasks derived from criteria used in this year's summer examination.

A detailed report on the results of the evaluation is planned for autumn 2003.

Author

Margit Ebbinghaus
Research assistant in the section for "Quality standards, certifications and examinations" at BIBB

Literature on the topic

• Borch, H., Weißmann, H.: IT-Weiterbildung hat Niveau(s). Das neue IT-Weiterbildungssystem für Facharbeiter und Seiteneinsteiger. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2002
• Borch, H., Weißmann, H. (ed.): IT-Berufe machen karriere. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2002
• Selka, R.: Neue Berufe brauchen neue Konzepte. Best practice in IT- und Medienberufen. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2002
• Bott, P., Hall, A., Schade, H.-J.: Qualifikationsanforderungen im IT-Bereich: Wunsch und Wirklichkeit. Ergebnisse einer Inserentennachbefragung im Rahmen des Früherkennungssystems Qualifikationsentwicklung. Volume 1. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2000
• EBBINGHAUS, Margit; GÖRMAR, Gunda; STÖHR, Andreas: Evaluiert: Projektarbeit und Ganzheitliche Aufgaben. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2001
• Ebbinghaus, M.: Mechatroniker-Prüfung - Anspruch und Wirklichkeit. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2003
• Mechnich, G., Ebbinghaus, M.: Prüfen in einem "neuen" Berufsfeld - Mediengestalter / Mediengestalterin Bild und Ton. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2003
• Ebbinghaus, M., Walter, G., Schmidt, U.: Kompetenzen erkennen - planvoll orientieren. Berufscasting: Ein berufsorientierendes Assessmentverfahren für Jugendliche. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2003
• Ebbinghaus, M.: Gestaltungoffene Abschlussprüfung. Ergebnisse einer Prüferbefragung im Ausbildungsberuf mediengestalter / Mediengestalterin für Digital- und Printmedien. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2002
• Borch, H., Frackmann, M., Weißmann, H.: Mechatroniker / Mechatronikerin - Best practice. Gestaltung der Ausbildung - Umsetzungsbeispiele. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2001
• Borch, H., Weißmann, H.: Mechatroniker / Mechatronikerin - Ein neuer staatlich anerkannter Ausbildungsberuf. Bundesinstitut für Berufsbildung, Der Generalsekretär (ed.). Bielefeld: Bertelsmann, 2000