This cross-sectional, evaluative study was carried out in two phases. At the beginning of the course, participants from all three countries filled out the Self-Assessment Instrument of Competencies for Public Health Leaders (SAIC-PHL) [24]. At the end, they again filled out an evaluation questionnaire and the SAIC-PHL.
Course description
The Leadership in Public Health courseFootnote 1 was designed by international experts to introduce diverse European perspectives of leadership in the modern public health environment in Europe. Public health is broadly made up of a number of specific disciplines such as methods in public health; population health and its social and economic determinants; population health and its material-physical, radiological, chemical, and biological environmental determinants; health policy; economics; organizational theory and management; health promotion; health education; health protection; and disease prevention and ethics [25]. A starting point was to conceptualize public health in a way that it was relevant to all European states. For this reason, the project adopted the definition of public health in line with the European Public Health Operations (EPHOs) which constitute, “a set of fundamental actions that address determinants of health, and maintain and protect population health through organized efforts of society [26].” It was built around a thematic framework of public health leadership competencies, based on a systematic literature review [27], consisting of 52 competencies distributed among eight domains: Systems Thinking, Political Leadership, Building and Leading Interdisciplinary Teams, Leadership and Communication, Leading Change, Emotional Intelligence and Leadership in Team-based Organisations, Leadership Organisational Learning and Development and Ethics and Professionalism. After being first piloted by Sheffield University in the UK, the course was implemented at Maastricht University (Netherlands), Kaunas University (Lithuania), and the University of Graz (Austria) which were partners in the EU Erasmus Curriculum Development project “Leaders for Public health in Europe.” It was a part-time course, delivered over a period of 8 weeks. The official language was English. PBL was used as the instructional model and implemented as blended learning. The course began with one and a half days of face-to-face learning which included introductions to blended learning and PBL, tutorial group meetings for the first and second PBL task, and a lecture. All other tutorial group meetings and lectures were delivered online during six half-day sessions within a period of 8 weeks. The online sessions were interactive. Participants could interact by using the microphone or the chat function.
Collaboration is one of the key learning principles of PBL and played a central role during the tutorial group meetings. During these meetings, participants were continuously in interaction with each other. Knowledge and experiences were shared, like in proper face-to-face PBL meetings. All participants attended the online lectures as one group. In some lectures, small group events were organized: Participants were divided into different online breakout rooms for discussions and afterwards plenary reported the results of that discussion in the main online lecture hall.
The taught content was based on the competency domains and included systems thinking, political leadership, collaborative leadership, building and leading interdisciplinary teams, leadership and communication, leading change, emotional intelligence and leadership in teams, and leadership, organizational learning, and development. These topics correspond with domains and competencies associated with leadership in the public health domain and cover the topics that were identified in the literature as most important and relevant for public health leaders [27].
Each session was delivered by a teacher(s) responsible for one of these components. Teachers were from the UK, Austria, Lithuania, and the Netherlands and represented various academic fields: public health, psychology, nursing, political science, education, and social science. All teachers underwent PBL training and blended learning training prior to the delivery of the course. They also served as tutors in the online tutorial groups.
A virtual learning environment was constructed which contained announcements, course information including course handbook, information on e-learning, and information for the sessions including all teaching materials, such as assignments, briefs, hand-outs, additional references, and literature cited or used during each session. There was also a discussion board, which was used for informal communication between participants. E-mail was used for questions to the course coordinator.
Participants
Nineteen participants (4 males, 15 females) completed the course: eight from Maastricht University (The Netherlands), five from Kaunas University (Lithuania), and six from the Medical University of Graz (Austria). The participating universities offered the existent courses in which this leadership course in a blended learning format could be included. The participants from Maastricht followed the European Public Health program and were used to PBL tutorial groups and familiar to some online practices. Participants from Kaunas followed a public health PhD program and had no prior experience of PBL or online learning, but they did have experience of team-based learning. Participants from Graz were working professionals, following a Master’s Program in Health and Nursing Science and were usually taught in lectures and small group seminars, but also had some experience of PBL as well as some experience with web-based-training and online lectures.
Instruments
The Evaluation Questionnaire consisted of 54 items from several existing scales. General satisfaction with course and instructor quality was measured with three items [28]. Thirteen items were used for measuring instructiveness, productivity of tutorial groups, applicability of new knowledge, and difficulty of the course. Tutorial group functioning was measured by six subscales: elaboration, interaction, motivation, and sponging, cohesion, and withdrawing [29]. To evaluate tutor functioning, four subscales were used [30]: stimulating constructive/active learning, stimulating self-directed learning, stimulating contextual learning, and stimulating collaborative learning. Motivation of the tutor to fulfill this role and stimulating professional behavior by the tutor are two single-item scales. Quality of different aspects of the e-learning was evaluated with three subscales [10]: Evaluation of e-teaching, evaluation of e-resources, and interactions between learners consisted. Items use a 5-point Likert scale ranging from strongly disagree to strongly agree or a 10-point rating scale. One additional item asked to rate their computer skills on a 5-point rating scale ranging from “very poor” to “excellent.”
The Self-Assessment Instrument of Competencies for Public Health Leaders (SAIC-PHL) [27] was used to measure learners’ own perceptions of their competencies before and after the course. This consisted of 52 items describing competencies essential for public health leaders. These competency descriptions are developed based on a literature review and refined and validated in a consensus development panel and two rounds of a Delphi survey [27]. For each competency, learners had to assess how well they thought they were doing on a 5-point scale ranging from “acting as a novice” to “acting as an expert.” This scale has been adapted from Dreyfus and Dreyfus [31]. The items were organized into eight competency domains of the public health leadership framework [27], reflected in the eight subscales of the SAIC-PHL.
Data collection and analyses
All data were collected online. SPSS version 19 was used to analyze the data. Cronbach’s alpha was calculated for the different scales from the Evaluation Questionnaire and the SAIC-PHL to check whether it was acceptable to use scale scores (alpha from .70 considered as acceptable, from .80 as good). We also explored the results on scales with lower alpha scores, while interpreting the results with more caution. Descriptive statistics for items and scales of the Evaluation Questionnaire, in the form of percentages and means, were used to examine the participants’ evaluations of the course. For negatively formulated items and scales consisting of only negatively formulated items, smaller means were interpreted as more positive. For scales containing both negatively and positively formulated items, the negatively formulated items were recoded before scale scores were calculated. For the SAIC-PHL, pretest and posttest scores were compared with paired t tests.
Differences in evaluation scores and in mean gains on the SAIC-PHL between participants from different locations were analyzed with non-parametric Kruskal-Wallis tests and post hoc Mann-Whitney tests. Non-parametric tests were used, because the sample sizes were small due to splitting up the sample into three groups for these analyses. To correct for multiple testing, Bonferroni-corrected alpha levels were used with the paired t tests, Kruskal-Wallis tests, and the Mann-Whitney tests. For the Kruskal-Wallis tests, Bonferroni correction was applied for each instrument separately. For the Mann-Whitney tests, the correction was applied for each pair of post hoc tests after a significant Kruskal-Wallis test. Bonferroni correction was also applied for the eight paired t tests computed for the SAIC-PHL.