Is blended learning and problem-based learning course design suited to develop future public health leaders? An explorative European study
Received: 10 July 2017
Accepted: 6 March 2018
Published: 1 June 2018
Abstract
Background
Public health leaders are confronted with complex problems, and developing effective leadership competencies is essential. The teaching of leadership is still not common in public health training programs around the world. A reconceptualization of professional training is needed and can benefit from innovative educational approaches. Our aim was to explore learners’ perceptions of the effectiveness and appeal of a public health leadership course using problem-based, blended learning methods that used virtual learning environment technologies.
Case presentation
In this cross-sectional evaluative study, the Self-Assessment Instrument of Competencies for Public Health Leaders was administered before and after an online, blended-learning, problem-based (PBL) leadership course. An evaluation questionnaire was also used to measure perceptions of blended learning, problem-based learning, and tutor functioning among 19 public health professionals from The Netherlands (n = 8), Lithuania (n = 5), and Austria (n = 6).
Participants showed overall satisfaction and knowledge gains related to public health leadership competencies in six of eight measured areas, especially Political Leadership and Systems Thinking. Some perceptions of blended learning and PBL varied between the institutions. This might have been caused by lack of experience of the educational approaches, differing professional backgrounds, inexperience of communicating in the online setting, and different expectations towards the course.
Conclusions
Blended, problem-based learning might be an effective way to develop leadership competencies among public health professionals in international and interdisciplinary context.
Keywords
Background
Public health leaders in many countries are faced with the challenges like the aging populations with chronic disease and major issues regarding emerging or re-emerging infectious diseases, including uncontrolled epidemics, distrust towards vaccination, or antibiotic resistance. There is little doubt that effective public health leadership is essential given the significant financial pressures on health services and the need to deliver more with less resources [1]. However, few educational courses deliver the necessary competences and specific training in leadership [2–4]. A recent debate on public health leadership featured in The Lancet pointed out that leadership is still not common in most public health training programs at undergraduate, postgraduate, and continuing professional development level and asserted that every public health organization should be engaged in developing more leaders at every level [4, 5]. There is a need for substantial investment in leadership training for public health professionals [6]. This raises a question of how higher education institutions can provide “content and context to initiate a major reconsideration of working and learning patterns which incorporate novel forms, based on the principles of interprofessional collaboration and transcend the confines of the classroom” [7].
Blended learning, which is a combination of face-to-face and online learning [8], improves access to education for learners who need to organize their education around professional roles or domestic responsibilities or live far from universities [8–11]. Further, it is possible to provide the same educational program to learners in different countries [11, 12] and enable learning in groups despite geographical boundaries [13], which is important in the context of globalization in higher education and continuing education [14, 15]. This should facilitate multidisciplinary learning and nurture a spirit of teamwork, particularly with regard to leadership skills [16]. Blended learning has proven to be at least as effective and equally satisfying for learners as traditional learning methods [17–19].
Problem-based learning (PBL) [20] is widely used in medical education and stimulates the development of leadership competencies as learners are self-directed and collaborate in small groups to work on authentic, complex tasks to explore problems and consider possible solutions [21, 22]. In response to the need to develop effective public health leaders, we explored the perceptions of the effectiveness and appeal of a newly developed public health leadership course using problem-based, blended-learning methods using virtual learning environment technologies. It was developed by international experts in public health and implemented in an international context. As diversity between learners from different countries may influence the adaptation and use of educational innovations [23], we also investigated possible differences between three different European academic settings: the Netherlands, Austria, and Lithuania.
Case presentation
This cross-sectional, evaluative study was carried out in two phases. At the beginning of the course, participants from all three countries filled out the Self-Assessment Instrument of Competencies for Public Health Leaders (SAIC-PHL) [24]. At the end, they again filled out an evaluation questionnaire and the SAIC-PHL.
Course description
The Leadership in Public Health course1 was designed by international experts to introduce diverse European perspectives of leadership in the modern public health environment in Europe. Public health is broadly made up of a number of specific disciplines such as methods in public health; population health and its social and economic determinants; population health and its material-physical, radiological, chemical, and biological environmental determinants; health policy; economics; organizational theory and management; health promotion; health education; health protection; and disease prevention and ethics [25]. A starting point was to conceptualize public health in a way that it was relevant to all European states. For this reason, the project adopted the definition of public health in line with the European Public Health Operations (EPHOs) which constitute, “a set of fundamental actions that address determinants of health, and maintain and protect population health through organized efforts of society [26].” It was built around a thematic framework of public health leadership competencies, based on a systematic literature review [27], consisting of 52 competencies distributed among eight domains: Systems Thinking, Political Leadership, Building and Leading Interdisciplinary Teams, Leadership and Communication, Leading Change, Emotional Intelligence and Leadership in Team-based Organisations, Leadership Organisational Learning and Development and Ethics and Professionalism. After being first piloted by Sheffield University in the UK, the course was implemented at Maastricht University (Netherlands), Kaunas University (Lithuania), and the University of Graz (Austria) which were partners in the EU Erasmus Curriculum Development project “Leaders for Public health in Europe.” It was a part-time course, delivered over a period of 8 weeks. The official language was English. PBL was used as the instructional model and implemented as blended learning. The course began with one and a half days of face-to-face learning which included introductions to blended learning and PBL, tutorial group meetings for the first and second PBL task, and a lecture. All other tutorial group meetings and lectures were delivered online during six half-day sessions within a period of 8 weeks. The online sessions were interactive. Participants could interact by using the microphone or the chat function.
Collaboration is one of the key learning principles of PBL and played a central role during the tutorial group meetings. During these meetings, participants were continuously in interaction with each other. Knowledge and experiences were shared, like in proper face-to-face PBL meetings. All participants attended the online lectures as one group. In some lectures, small group events were organized: Participants were divided into different online breakout rooms for discussions and afterwards plenary reported the results of that discussion in the main online lecture hall.
The taught content was based on the competency domains and included systems thinking, political leadership, collaborative leadership, building and leading interdisciplinary teams, leadership and communication, leading change, emotional intelligence and leadership in teams, and leadership, organizational learning, and development. These topics correspond with domains and competencies associated with leadership in the public health domain and cover the topics that were identified in the literature as most important and relevant for public health leaders [27].
Each session was delivered by a teacher(s) responsible for one of these components. Teachers were from the UK, Austria, Lithuania, and the Netherlands and represented various academic fields: public health, psychology, nursing, political science, education, and social science. All teachers underwent PBL training and blended learning training prior to the delivery of the course. They also served as tutors in the online tutorial groups.
A virtual learning environment was constructed which contained announcements, course information including course handbook, information on e-learning, and information for the sessions including all teaching materials, such as assignments, briefs, hand-outs, additional references, and literature cited or used during each session. There was also a discussion board, which was used for informal communication between participants. E-mail was used for questions to the course coordinator.
Participants
Nineteen participants (4 males, 15 females) completed the course: eight from Maastricht University (The Netherlands), five from Kaunas University (Lithuania), and six from the Medical University of Graz (Austria). The participating universities offered the existent courses in which this leadership course in a blended learning format could be included. The participants from Maastricht followed the European Public Health program and were used to PBL tutorial groups and familiar to some online practices. Participants from Kaunas followed a public health PhD program and had no prior experience of PBL or online learning, but they did have experience of team-based learning. Participants from Graz were working professionals, following a Master’s Program in Health and Nursing Science and were usually taught in lectures and small group seminars, but also had some experience of PBL as well as some experience with web-based-training and online lectures.
Instruments
The Evaluation Questionnaire consisted of 54 items from several existing scales. General satisfaction with course and instructor quality was measured with three items [28]. Thirteen items were used for measuring instructiveness, productivity of tutorial groups, applicability of new knowledge, and difficulty of the course. Tutorial group functioning was measured by six subscales: elaboration, interaction, motivation, and sponging, cohesion, and withdrawing [29]. To evaluate tutor functioning, four subscales were used [30]: stimulating constructive/active learning, stimulating self-directed learning, stimulating contextual learning, and stimulating collaborative learning. Motivation of the tutor to fulfill this role and stimulating professional behavior by the tutor are two single-item scales. Quality of different aspects of the e-learning was evaluated with three subscales [10]: Evaluation of e-teaching, evaluation of e-resources, and interactions between learners consisted. Items use a 5-point Likert scale ranging from strongly disagree to strongly agree or a 10-point rating scale. One additional item asked to rate their computer skills on a 5-point rating scale ranging from “very poor” to “excellent.”
The Self-Assessment Instrument of Competencies for Public Health Leaders (SAIC-PHL) [27] was used to measure learners’ own perceptions of their competencies before and after the course. This consisted of 52 items describing competencies essential for public health leaders. These competency descriptions are developed based on a literature review and refined and validated in a consensus development panel and two rounds of a Delphi survey [27]. For each competency, learners had to assess how well they thought they were doing on a 5-point scale ranging from “acting as a novice” to “acting as an expert.” This scale has been adapted from Dreyfus and Dreyfus [31]. The items were organized into eight competency domains of the public health leadership framework [27], reflected in the eight subscales of the SAIC-PHL.
Data collection and analyses
All data were collected online. SPSS version 19 was used to analyze the data. Cronbach’s alpha was calculated for the different scales from the Evaluation Questionnaire and the SAIC-PHL to check whether it was acceptable to use scale scores (alpha from .70 considered as acceptable, from .80 as good). We also explored the results on scales with lower alpha scores, while interpreting the results with more caution. Descriptive statistics for items and scales of the Evaluation Questionnaire, in the form of percentages and means, were used to examine the participants’ evaluations of the course. For negatively formulated items and scales consisting of only negatively formulated items, smaller means were interpreted as more positive. For scales containing both negatively and positively formulated items, the negatively formulated items were recoded before scale scores were calculated. For the SAIC-PHL, pretest and posttest scores were compared with paired t tests.
Differences in evaluation scores and in mean gains on the SAIC-PHL between participants from different locations were analyzed with non-parametric Kruskal-Wallis tests and post hoc Mann-Whitney tests. Non-parametric tests were used, because the sample sizes were small due to splitting up the sample into three groups for these analyses. To correct for multiple testing, Bonferroni-corrected alpha levels were used with the paired t tests, Kruskal-Wallis tests, and the Mann-Whitney tests. For the Kruskal-Wallis tests, Bonferroni correction was applied for each instrument separately. For the Mann-Whitney tests, the correction was applied for each pair of post hoc tests after a significant Kruskal-Wallis test. Bonferroni correction was also applied for the eight paired t tests computed for the SAIC-PHL.
Results
Cronbach’s alphas for the scales from the self-assessment of competencies for public health leaders on pretest and posttest
N of items | α pretest | α posttest | |
|---|---|---|---|
Systems Thinking | 7 | .89 | .91 |
Political Leadership | 8 | .94 | .95 |
Inspiring and Motivating Others | 7 | .93 | .96 |
Building and Leading Interdisciplinary Teams | 5 | .82 | .95 |
Leadership and Effective Communication | 7 | .86 | .89 |
Leading Change | 6 | .95 | .94 |
Emotional Intelligence and Leadership in Teams | 6 | .91 | .94 |
Ethics and Professionalism | 6 | .95 | .93 |
Evaluation of the course
General satisfaction
The course content was just right according to 15 of the participants (79%), while two of them (11%) thought contents were easy and two others (11%) found them difficult. The average grade given to different aspects of the course ranged from 6.79 (SD = 2.66) to 7.79 (SD = 2.04) on a scale from 1 to 10. Because these means are on the positive half of the scale, these results indicate that on average participants were satisfied. Scores for the items using a 5-point scale differed between 3.47 (SD = 1.07) and 3.95 (SD = 0.97); they were on the positive side of the neutral value of 3. Means for the two negatively formulated aspects were 1.74 (SD = 0.93) and 2.05 (SD = 1.35), indicating positive evaluations. On the scale “Global rating of course and instructor quality,” the score was 3.53 (SD = 1.04).
Student evaluation of course on leadership: results of Kruskal-Wallis comparing results at different locations
Evaluation items | H | p |
|---|---|---|
The organization of the blocka | 3.64 | .16 |
The instructiveness of the blocka | 2.43 | .31 |
The link of the block with their prior knowledgea | 0.79 | .69 |
The productivity of the tutorial groupa | 9.09 | .0052 |
The link between the block and the assessmenta | 3.74 | .16 |
The objectives of the course were clearb | 4.74 | .08 |
The literature fitted to the objectives of the courseb | 6.55 | .03 |
I can apply what I have learned in my daily workb | 2.88 | .23 |
The e-lectures were instructiveb | 7.35 | .02 |
The tasks in the course handbook were instructiveb | 6.27 | .04 |
There were problems in collaboration because of differences in cultural backgroundb | 5.28 | .07 |
There were difficulties because different universities were involvedb | 5.18 | .07 |
Difficulty of the block contentsb | 1.46 | .62 |
Scale “global rating of course and instructor quality”b | 9.71 | .0031 |
Group functioning
The means on functioning of the tutorial group were 3.56 (SD = 0.79) for interaction, 3.58 (SD = 1.07) for motivation, and 3.87 (SD = 0.78) for elaboration. Also, on cohesion, the means (3.11, SD = 1.37 and 4.26, SD = 0.65) were on the positive side of the scale or close to the neutral value of 3. For the scale on negative aspects of group functioning (i.e., sponging) the mean score was 2.58 (SD = 1.02); a lower score is a positive outcome on this scale. For the two items on withdrawing, the means were 3.58 (SD = 1.22) and 2.32 (SD = .82).
Evaluation of functioning of the tutorial group and tutor functioning: descriptive statistics and results of Kruskal-Wallis tests and Mann-Whitney tests for different locations
M (SD) Maastricht | M (SD) Kaunas | M (SD) Graz | KW | MW Maastricht-Kaunas | MW Graz-Maastricht | MW Graz-Kaunas | |
|---|---|---|---|---|---|---|---|
Scales | H (p) | U (p) | U (p) | U (p) | |||
Functioning of tutorial group | |||||||
Elaboration | 3.44 (0.68) | 3.70 (0.45) | 4.58 (0.66) | 6.61 (.03) | |||
Interaction | 3.00 (0.59) | 3.47 (0.38) | 4.39 (0.53) | 11.34 (.0005) | 10.50 (.17) | 1.00 (.0013) | 2.00 (.02) |
Motivation | 3.31 (0.88) | 2.90 (1.19) | 4.50 (0.55) | 7.67 (.01) | |||
Sponging | 2.75 (0.76) | 3.50 (0.71) | 1.58 (0.66) | 10.90 (.001) | 9.50 (.13) | 5.50 (.0097) | .00 (.0043) |
Cohesion_item 1 | 4.38 (0.52) | 3.80 (0.45) | 4.50 (0.84) | 3.94 (.14) | |||
Cohesion_item 2 | 3.25 (1.28) | 3.40 (1.34) | 2.67 (1.63) | 0.86 (.68) | |||
Withdrawing_item 1 | 4.00 (1.07) | 4.20 (0.84) | 2.50 (1.05) | 6.70 (.03) | |||
Withdrawing_item 2 | 2.63 (0.74) | 2.80 (0.45) | 1.50 (0.55) | 9.31 (.0047) | 18.50 (.84) | 6.00 (.02) | 1.50 (.0087) |
Tutor functioning | |||||||
Stimulating constructive/active learning | 3.38 (1.16) | 3.47 (0.56) | 4.50 (0.41) | 7.50 (.02) | |||
Stimulating self-directed learning | 3.13 (1.16) | 3.40 (0.65) | 3.92 (0.74) | 1.86 (.41) | |||
Stimulating contextual learning | 3.00 (0.80) | 3.10 (0.82) | 4.33 (0.52) | 8.92 (.0064) | 18.50 (.84) | 4.00 (.0073) | 2.00 (.01) |
Stimulating collaborative learning | 2.31 (1.16) | 3.40 (0.65) | 3.92 (0.80) | 6.93 (.02) | |||
Motivation of the tutor | 3.38 (0.74) | 3.40 (0.55) | 4.50 (0.84) | 6.28 (.04) | |||
Stimulating professional behavior | 2.63 (0.92) | 3.20 (0.84) | 4.33 (0.82) | 8.28 (.00832) | 13.00 (.36) | 4.00 (.0087) | 5.00 (.07) |
Tutor functioning
The means for the tutor evaluation scales ranged from 3.11 (SD = 1.15) for stimulating collaborative learning to 3.75 (SD = 0.95) for stimulating constructive/active learning, indicating neutral to positive perceptions. The Bonferroni-corrected alpha for the Kruskal-Wallis tests of tutor functioning was .0083 (six tests conducted). Differences between locations were found on stimulating contextual learning and stimulating professional behavior (see Table 3). Participants from Graz were more positive about the tutors than participants from Maastricht on stimulating contextual learning and stimulating professional behavior. They were also more positive about stimulating contextual learning compared to participants from Kaunas.
Quality of blended learning
The mean regarding perceived quality of e-teaching was 3.61 (SD = 0.84), regarding quality of e-resources 3.49 (SD = .94), and regarding online student interaction 3.54 (SD = 0.71). The Bonferroni-corrected alpha for the Kruskal-Wallis tests of quality of blended learning was .017 (three tests conducted). Participants from Graz were more satisfied with the e-learning resources (H [2] = 9.23, p .0045; M = 4.33, SD = 0.63) than the participants from Kaunas (M = 2.60, SD = 0.86, U = 1.00, p = .01). On the other two scales, there were no differences.
Required computer skills
The average rating for own computer skills was 4.16 (SD = 0.90), ranging from poor to excellent. Because only one Kruskal-Wallis test was performed for required skills, alpha was not adjusted. Differences between the universities for computer skills were found, H [2] = 10.18, p = .003, indicating that participants from Graz (M = 3.17, SD = 0.75) rated their computer skills as poorer than those of the other participants (MMaastricht = 4.63, SD = 0.52, U = 3.00, p = .005; MKaunas = 4.60, SD = 0.55, U = 2.00, p = .02). No differences were found between participants from Maastricht and Kaunas, U = 19.50, p = 1.00.
Self-assessment of competencies for public health leaders
Self-assessment of competencies on leadership: descriptive statistics for pretest, posttest, and learning gains and statistics for paired t tests
Leadership domain (topic of teaching) | M (SD) pretest | M (SD) posttest | M (SD) gains | t (df = 18) | p |
|---|---|---|---|---|---|
Systems Thinking | 2.28 (0.65) | 3.26 (0.59) | 0.98 (0.69) | 6.21 | .0000 |
Political Leadership | 1.91 (0.83) | 2.95 (0.75) | 1.04 (0.56) | 8.10 | .0000 |
Inspiring and Motivating Others | 2.29 (0.74) | 3.15 (0.81) | 0.86 (0.76) | 4.92 | .0001 |
Building and Leading Interdisciplinary Teams | 2.49 (0.57) | 3.21 (0.84) | 0.72 (0.53) | 5.84 | .0000 |
Leadership and Effective Communication | 2.59 (0.63) | 3.11 (0.73) | 0.51 (0.59) | 3.79 | .0013 |
Leading Change | 2.17 (0.92) | 2.89 (0.82) | 0.73 (0.83) | 3.82 | .0013 |
Emotional Intelligence and Leadership in Teams | 3.02 (0.84) | 3.28 (0.74) | 0.26 (0.96) | 1.20 | .25 |
Ethics and Professionalism | 2.81 (0.89) | 3.10 (0.82) | 0.29 (0.94) | 1.35 | .20 |
Discussion and conclusions
The study showed that participants were overall positive about the effectiveness and delivery of the public health leadership course using blended learning and PBL methodologies. It seems that they valued both group functioning and tutor functioning in PBL as well as different aspects of blended learning (online interaction, e-teaching, e-resources). The self-reported level of leadership competencies increased over the period of the course for participants at all three locations. They gained most in the area of Political Leadership, Systems Thinking, and Inspiring and motivating others. The perceptions of blended learning and PBL partly varied between the participating locations. Differences in educational background between learners from different countries might have influence the use of educational innovations [23] as well as the perceptions of blended learning curricula [32, 33]. Keller and colleagues reported that in their study of blended education delivered in Lithuania, Sweden, and Norway, the students from Lithuania evaluated the virtual learning most positively, which is not confirmed in our study with participants from Kaunas University. We consider several possible explanations for the differences we found between evaluations of participants at different locations.
Lack of experience with PBL and online learning might have led to less positive evaluations of the course. Furthermore, participants’ expectations of the course might have differed from their experiences during the course due to perceived pressure and insecurity resulting from the need to perform in the group online. Discrepancies between expectations and actual experiences might have negatively affected their evaluation of the course [34]. The demanding nature of blended learning may negatively affect student reactions to these courses [17, 35], and lower activity of some participants in online discussions might be due to the fact that they were not used to actively participate in their learning at their home institution. Furthermore, the participants had different professional status. Participants who were clinical workers might have seen the relevance of the content more than participants who were mainly master students and PhD students. While the participants might have experienced the tension between the leadership theory and public health content, which constituted a kind of comfort zone for them, the resulting reflection contributed to a positive learning experience and understanding. In order to successfully implement effective public health interventions and change, collaboration with and participation of diverse groups of stakeholders is vital.
Although the results of the study can be direction setting for the blended PBL course designers, there are limitations to this study. The sample size of our study was small and selected conveniently using the available cohorts of students. The results could have been different if we offered the course on the open enrollment basis. It is a strength of our study that a pretest-posttest design was used for measuring competency gains, although based on self-assessment. Using objective learning data and standard assessment of participants’ competencies could be an interesting addition for future studies as well as following the impact of the course of a longer period of time and within an experimental design.
Taken together, this study suggests that organizing continuing education for professionals in an international context has potential [15]. Course designers might benefit from our results when designing blended learning courses. Bringing together international experts in the field as teachers in a blended course, with ample opportunities for group work and discussions among participants, contributes to the development of professional competencies, in our case on leadership in public health [13]. Differences between the evaluations of participants from universities from different countries point to the relevance of participants’ expectations, previous learning experiences, and educational context. The findings may also suggest that although the blended learning format of course delivery was well accepted in this small-scale study, there is a need to support students’ shift to the more active independent mode of learning including coaching, mentoring, and personal development planning. Such approaches are successfully included in other hybrid inline leadership courses, for example, University of North Carolina and Gillings School of Global Public Health Doctoral Program in Health Leadership [36]. The course applies modern technology and flexible teaching approaches with the emphasis on development of leadership competencies for experienced health professionals working full-time anywhere in the world. Similar to our course, leadership is learned through interaction, debate and collaboration, and mentorship which is a key to lifelong learning. Future research on best ways to support the learning process may also be relevant to the other academic environments. We recommend investing in training problem-based learning and blended learning skills for both students and staff to prevent that learning of these skills interferes with studying the course content.
Conclusion
It seems that problem-based learning and blended learning can be an effective way to develop public health leadership competencies among professionals in international and interdisciplinary context if the specificity and educational background of the learners is properly addressed.
The study “Leaders for European Public Health” (LEPHIE) Project was carried out with the support of the European Commission Lifelong Learning Program.
Declarations
Acknowledgements
Not applicable
Funding
ERASMUS Multilateral Curriculum Development project: Leaders for European Public Health (LEPHIE). Project no. 510176-LLP-1-2010-1-NL-ERASMUS-ECDCE—supported by the Lifelong Learning Program of the European Commission. Funding was received for developing the course. The funding body did not have a role in design of the study and collection, analysis, and interpretation of data, nor in writing the manuscript.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Authors’ contributions
All authors but IS contributed to the conception and the design of the study (KK, NdJ, CL, LS, TS, SO’C, JvM, KC). The development of the measurement instruments was done by KK, NdJ, TS, SO’C, JvM, and KC. The data collection at location was realized by NdJ, CL, LS, and TS. Data analyses was conducted by IS, KK, and JvM. In interpreting the data and writing the paper, KK, IS, and KC took the lead. All authors read and approved the final manuscript.
Ethics approval and consent to participate
We invited the participants of the course to fill out the evaluation questionnaire within the framework of university satisfaction evaluations, for which we did not ask for the ethical clearance. Data collection was done according to the Helsinki Declaration, and informed consent was obtained.
Consent for publication
Not applicable
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
- Marmot M, Friel S, Bell R, Houweling TA, Taylor S. Health CoSDo. Closing the gap in a generation: health equity through action on the social determinants of health. Lancet. 2008;372(9650):1661–9.View ArticlePubMedGoogle Scholar
- Bjegovic-Mikanovic V, Vukovic D, Otok R, Czabanowska K, Laaser U. Education and training of public health professionals in the European region: variation and convergence. Int J Public Health. 2013;58(6):801–10.View ArticlePubMedGoogle Scholar
- Czabanowska K, Smith T, Stankunas M, Avery M, Otok R. Transforming public health specialists into public health leaders. The role of LEPHIE project. Lancet. 2013;381(9865):449–50.View ArticlePubMedGoogle Scholar
- Day M, Shickle D, Smith K, Zakariasen K, Oliver T, Moskol J. Time for heroes: public health leadership in the 21st century. Lancet. 2012;380(9849):1205–6.View ArticlePubMedGoogle Scholar
- Day M, Shickle D, Smith K, Zakariasen K, Oliver T, Moskol J. Transforming public health specialists into public health leaders—authors’ reply. Lancet. 2013;381(9865):450.View ArticlePubMedGoogle Scholar
- Czabanowska K, Rethmeier KA, Lueddeke G, et al. Public health in the 21st century: working differently means leading and learning differently. Eur J Public Health. 2014;24(6):1047–52.View ArticlePubMedGoogle Scholar
- Lueddeke GR, Dayrit MM. Transforming medical education for the 21st century: megatrends, priorities and change. Radcliffe; 2012.Google Scholar
- Graham CR. Emerging practice and research in blended learning. Chapter 21. In: Moore MG, editor. Handbook of distance education, third edition. Routledge; p. 333-50.Google Scholar
- Beckmann EA. Learners on the move: mobile modalities in development studies. Distance Education. 2010;31(2):159–73.View ArticleGoogle Scholar
- Ginns P, Ellis R. Quality in blended learning: exploring the relationships between on-line and face-to-face teaching and learning. Internet High Educ. 2007;10(1):53–64.View ArticleGoogle Scholar
- Graham C. Blended learning systems: definition, current trends, and future directions, The handbook of blended learning: global perspectives, local design. San Francisco: John Wiley & Sons; 2006. p. 3–21.Google Scholar
- Altbach PG, Knight J. The internationalization of higher education: motivations and realities. J Stud Int Educ. 2007;11(3-4):290–305.View ArticleGoogle Scholar
- Ziegler MF, Paulus T, Woodside M. Understanding informal group learning in online communities through discourse analysis. Adult Educ Q. 2014;64(1):60–78.View ArticleGoogle Scholar
- Bowman L. Book review: handbook of adult and continuing education. Adult Educ Q. 2012;62(4):413–5.View ArticleGoogle Scholar
- Coryell JE. Collaborative, comparative inquiry and transformative cross-cultural adult learning and teaching: a western educator metanarrative and inspiring a global vision. Adult Educ Q. 2013;62(4):299–320.View ArticleGoogle Scholar
- Suzuki Y, Niwa M, Shibata T, et al. Internet-based problem-based learning: international collaborative learning experiences, Problem-based learning in eLearning breakthroughs; 2007. p. 131–46.Google Scholar
- Spanjers IA, Könings KD, Leppink J, et al. The promised land of blended learning: quizzes as a moderator. Educational Res Rev. 2015;15:59-74.Google Scholar
- Wutoh R, Boren SA, Balas EA. ELearning: a review of Internet-based continuing medical education. J Contin Educ Health Prof. 2004;24(1):20–30.View ArticlePubMedGoogle Scholar
- Könings KD, Popa D, Gerken M, et al. Improving supervision for students at a distance: videoconferencing for group meetings, Innovations in education and teaching international; 2015(ahead-of-print). p. 1–12.Google Scholar
- Mustard JF, Neufeld VR, Walsh WJ, Cochran J. New trends in health sciences education, research, and services: the McMaster experience. New York: Praeger Publishers; 1982.Google Scholar
- Dolmans DH, De Grave W, Wolfhagen IH, Van Der Vleuten CP. Problem-based learning: future challenges for educational practice and research. Med Educ. 2005;39(7):732–41.View ArticlePubMedGoogle Scholar
- Dolmans D, Michaelsen L, Van Merrienboer J, van der Vleuten C. Should we choose between problem-based learning and team-based learning? No, combine the best of both worlds! Med Teach. 2015;37(4):354–9.View ArticlePubMedGoogle Scholar
- Jippes M, Majoor GD. Influence of national culture on the adoption of integrated and problem-based curricula in Europe. Med Educ. 2008;42(3):279–85.View ArticlePubMedGoogle Scholar
- Czabanowska K, Smith T, O'Connor S, Sumskas L, Otok R, Brand H. Identifying the leadership competencies for European public health professionals. Eur J Pub Health. 2012;22:91.Google Scholar
- Birt CF, A.. European core competences for MPH education (ECCMPHE). Brussels: ASPHER; 2011.Google Scholar
- World Health Organization. European action plan for strengthening public health capacities and services. Copenhagen: Regional Committee for Europe; 2012. http://www.euro.who.int/en/health-topics/Health-systems/public-health-services/publications/2012/european-action-plan-for-strengthening-public-health-capacities-and-services.
- Czabanowska K, Smith T, Könings KD, et al. In search for a public health leadership competency framework to support leadership curriculum—a consensus study. Eur J Public Health. 2013:24(5):850–56.Google Scholar
- Frick TW, Chadha R, Watson C, Zlatkovska E. Improving course evaluations to improve instruction and complex learning in higher education. Educ Technol Res Dev. 2010;58(2):115–36.View ArticleGoogle Scholar
- Dolmans DH, Wolfhagen IH, Van der Vleuten CP. Thinking about student thinking: motivational and cognitive processes influencing tutorial groups. Acad Med. 1998;73(10):S22–4.View ArticlePubMedGoogle Scholar
- Dolmans DH, Ginns P. A short questionnaire to evaluate the effectiveness of tutors in PBL: validity and reliability. Med Teach. 2005;27(6):534–8.View ArticlePubMedGoogle Scholar
- Dreyfus SE, Dreyfus, HL. A five-stage model of the mental activities involved in directed skill acquisition (No. ORC-80-2). California Univ Berkeley Operations Research Center. 1980.Google Scholar
- Keller C. User acceptance of virtual learning environments: a case study from three northern European universities. Commun Assoc Inf Syst. 2009;25(1):38.Google Scholar
- Keller C, Hrastinski S, Carlsson S. "Students' Acceptance of E-Learning Environments: A Comparative Study in Sweden and Lithuania" (2007). ECIS 2007 Proceedings. 40. http://aisel.aisnet.org/ecis2007/40.
- Könings KD, Brand-Gruwel S, van Merriënboer JJ, Broers NJ. Does a new learning environment come up to students’ expectations? A longitudinal study. J Educ Psychol. 2008;100(3):535–48.View ArticleGoogle Scholar
- Sitzmann T, Kraiger K, Stewart D, Wisher R. The comparative effectiveness of web-based and classroom instruction: a meta-analysis. Pers Psychol. 2006;59(3):623–64.View ArticleGoogle Scholar
- Health UGSoGP. Public health leadership program. https://sph.unc.edu/phlp/phlp/.
