The Design and Evaluation of Online Faculty Development for Effective Graduate Supervision

Correspondence: Michele Jacobsen University of Calgary Email: dmjacobs at ucalgary.ca Abstract This design-based research aims to improve the quality of graduate supervision using a Massive Open Online Course (MOOC). The Quality Graduate Supervision MOOC brings interdisciplinary faculty, postdoctoral scholars, and expert supervisors together in an online learning community to discuss and consider effective supervision practice, strategies for relationship building, supports for academic writing, mentoring for diverse careers, and how to combine excellence and wellness. The survey, interview, and system data were analyzed to inform and assess the design and development of the QGS MOOC, to gain insights into learner experience and engagement, and to assess the impact of the online learning community on graduate supervision practices. Through ongoing design and evaluation of this online learning course for graduate supervisors, the research team found the learning community influenced faculty members’ awareness, collective knowledge building, goal setting, and actions for graduate supervision practice. We present results from our evaluation of the design components in the QGS MOOC, the learning benefits for supervisors, impacts on graduate supervision practice, and make several recommendations for research and practice.


Introduction
Graduate supervision is a complex and demanding part of a professor's research and teaching responsibilities (Walker et al., 2008). Graduate supervisors serve many roles, from coach, mentor, teacher, guide, and critic. Supervisors need to be flexible and adapt to what individual students need at various stages of their program. The most effective graduate supervisors combine their passion for research and teaching in their discipline with a commitment to student mentorship. Quality graduate supervision pedagogies are a key factor in student success (Adkins, 2009;Kamler & Thomson, 2014;Skarakis-Doyle & McIntyre, 2008), while marginal supervision is associated with high doctoral student dropout rates in Canada and Asia (Ali et al., 2017;Erichsen et al., 2014). The quality of supervision has an impact on a graduate student's learning and development as a new researcher (Beaudin et al., 2015;Bloom et al, 2007;Williams, 2005), their writing and defense of the thesis or dissertation (CAGS, 2018a;Ciampa & Wolfe, 2019), and timely completion of their degree.
Most supervisors want to be effective at mentoring and guiding the next generation of researchers and leaders. However, academics tend to develop graduate supervision practices on the job as opposed to being actively and intentionally educated as a research mentor (CAGS, 2018b;Eley & Jennings, 2005;Wisker, 2012). Some institutions require an orientation before granting supervisory privileges; however, the type of short seminars usually on offer fails to address the scope and variety of supervisory practices and understates the potential challenges facing students and supervisors (McAlpine, 2017;Wisker, 2012). Academics welcome opportunities to develop their supervision practices; however, this enthusiasm may not translate into time for workshops or seminars on campus, even if such events fit their schedule. Faculty attendance in scheduled, on-campus workshops are influenced by: 1) limited time, which means faculty may not prioritize a scheduled workshop; 2) intense workload, scheduled seminars often overlap with other commitments; and 3) physical location, with transit between buildings or sites, paying for parking, arranging childcare, and meeting in person as barriers.
Academics need timely and flexible access to quality faculty development on graduate supervision. Complex issues and situations can arise while working with diverse students (Ali et al., 2017;Roumell & Bollinger, 2017;Saleem & Mehmood, 2018). A lack of supervisor training can result in professors being unprepared to manage challenges in supervision, such as remote graduate supervision (McAlpine & McKinnon, 2013;Kumar & Johnson, 2019;Kumar & Coe, 2017;Maor, et al, 2016). Considering the rapid adjustment to online supervision with the onset of COVID19 in 2020, providing appropriate and responsive supports to both supervisors and students is a pressing concern in higher education.
Massive Open Online Courses (MOOCs) provide a workable solution to transcend temporal and geospatial restrictions of scheduled on-campus workshops and address the need for flexible, accessible, and interdisciplinary faculty professional development on graduate supervision (Alharbi & Jacobsen, 2018;CAGS, 2018a). A MOOC can offer a blend of instructor-guided and self-directed learning opportunities that are relevant and tailored to faculty development needs and goals during times that work best for individual schedules. For example, Alharbi and Jacobsen's (2018) QGS miniMOOC pilot increased opportunities for meaningful discussions and sharing among diverse colleagues about graduate supervision. Sommer et al. (2019) transformed a workshop for researchers into a MOOC to support more learners than could have been included face-to-face. Salmon et al. (2015) designed a MOOC for academics to replace an on-campus workshop on teaching quality.
MOOCs offer an exciting and flexible approach to online learner engagement for both new and experienced academics (Mackness et al., 2013;Moskal et al., 2015;Salmon et al., 2015). Benefits of MOOCs include schedule flexibility, greater enrolment capacity, increased diversity of learners and ideas, expanded time to access and reflect on resources, and curated content well beyond what is usually offered in a short workshop. Well-designed MOOCs can support large-scale faculty development (Rodrigues & Leinster, 2016) that is open and flexible in how and when participants engage with material (Kiers, 2016). Participants appreciate access to online course content on the go or from their home or campus office (Rodrigues & Leinster, 2016). While some learners appreciate less structure in a MOOC, others appreciate designed tasks, discussion boards, and external motivation to gain badges (Salmon et al., 2015) or other micro-credentials, and established timelines for course completion (Mackness et al., 2013;Moskal et al., 2015). The flexibility of MOOCs can support learner autonomy and the degree to which participants choose to engage with content and community.
The Quality Graduate Supervision MOOC is currently being offered at one university with interdisciplinary faculty and postdoctoral scholars to enhance supervision practice using several guided and self-paced learning modules. Our research on the QGS MOOC is guided by three assumptions: 1) graduate supervisors will engage and benefit from a faculty development MOOC, 2) an online learning community can benefit both new and experienced graduate supervisors across disciplines, and 3) that by supporting supervisors' learning and growth, graduate student experiences can be improved (Alharbi & Jacobsen, 2018). Graduate supervision is undertaken differently by faculty across disciplines, such as arts, sciences, medicine, and education (Eley & Jennings, 2005). The QGS MOOC provides an online community of practice in which graduate supervisors from across campus engage in transdisciplinary discussions and collaborations that transcend faculty and department silos and the time and space constraints of scheduled workshops.
Three research questions guide this study: 1) What design components are necessary for a supervision development MOOC?, 2) What types of scaffolding and support enable supervisors to effectively use the MOOC?, and 3) In what ways does the QGS MOOC support and benefit faculty in the ongoing development of graduate supervision practices? We evaluate the impact of six design components in the MOOC and analyze whether this online course influenced supervision practice. In the sections that follow, we describe our research methodology and methods, the design and development of the MOOC, the analysis and integration of data, and summarize study findings and recommendations to inform further research and practice.

Methodology
Design-based research is a participatory educational research methodology well suited for analyzing problems of practice, the collaborative design of solutions with practitioners, and iterative and cyclical evaluation of innovative interventions in authentic learning environments (Jacobsen, 2014;McKenny & Reeves, 2019). A design-based research approach was adopted for the development and evaluation of the MOOC. In phase one, the online course was designed and evaluated with 23 academics as a closed miniMOOC pilot (Alharbi, 2018;Alharbi & Jacobsen, 2019, 2018. In this paper, we report on phase two and the second iteration of the 4 Open/Technology in Education, Society, and Scholarship Association Journal: 2021, Vol. 1(1) 1-23 QGS MOOC (version two or V2) developed in Fall 2019 and offered to academic faculty and postdoctoral scholars in Winter 2020. Multiple and mixed forms of data were collected to address the research questions. Data included 1) survey responses collected at the start (presurvey n= 53) and conclusion (post-survey n= 46) of the MOOC, 2) interviews with seven participants after the course, and 3) LMS data on design components, interaction, and usage patterns throughout the course. Surveys included select responses and open-ended items and were administered online. Interviews were conducted with six MOOC graduates to evaluate learning experiences, to assess what worked well and what needed to change, to capture reflections and impacts on supervision practice, and to inform general design principles for faculty development MOOCs. Online usage metrics and data collected in D2L and Yuja provided insights on participants' and discussion moderators' interaction in online activities in the QGS MOOC V2. Data on the implementation of the course inform our evaluation of the success and impact of the MOOC for faculty development and for improving graduate supervision practice.

Enrollment and Completion Rate
Offered from February to April 2020 at the University of Calgary, the QGS MOOC V2 attracted 107 academic faculty and postdoctoral scholars from twelve faculties and campus service hubs. While the MOOC was designed with academic faculty in mind, we decided to open the course to postdoctoral scholars to address their professional development and learning needs . Originally scheduled for six weeks, we implemented an open-ended completion target given the impact of COVID19 and the rapid pivot to remote teaching and supervision across campus. All participants who indicated a desire to complete the MOOC and earn the Badge of Completion were supported in doing so. Of the 107 participants who enrolled in the course, 50 (47%) completed six modules and earned the QGS MOOC Badge (27 faculty and 23 postdocs). Our completion rate is higher than rates reported for other faculty development MOOCs; for example, Rodrigues & Leinster (2016) report a 37% completion rate, and Salmon et al. (2015) a 17% completion rate. In the next section, we describe the design and development of the QGS MOOC V2 followed by an analysis of data.

Design and Development
Phase one of the course was designed and evaluated as a miniMOOC pilot in 2017 as part of a PhD research project (Alharbi, 2018). Given participants' engagement in a sustained learning community with peers across campus, and the impact of this online learning experience on strengthening supervision practices, the pilot was considered a success (Alharbi & Jacobsen, 2018). In Fall 2019, the second version of the course was designed, then implemented and evaluated in Winter 2020 as part of phase two. The design and development of six modules, activities, resources, and learning tasks in V2 of the QGS MOOC took seven months. Along with weekly webinars, curated resources, engaging activities, and interaction with colleagues from across disciplines, this online learning experience includes the expertise, perspectives, and advice from several award-winning graduate supervisors to inform learning and promote conversations among participants. Fieldwork and surveys with graduate students, faculty members, and subject matter experts informed the development and review of course content, module organization, and navigation design. A month prior to going live, the team carried out a soft launch of the MOOC with three faculty reviewers, chosen for their expertise in online learning and experience with graduate supervision. This review data-informed ongoing revisions and improvements in the QGS MOOC V2 content, navigation, and layout prior to launch. This online course design demonstrates three of four cMOOC characteristics, adapted from Bates (2019) and originally described by Downes (2014): 1) autonomy of the learner (learners participate when and where, learning is self-directed, accountability through reflection and action), 2) diversity of tools, learners, and knowledge (multimedia, multiple disciplines, diverse and emerging content), and 3) interactivity (collaborative learning, open and ongoing communication between learners, emergent knowledge). Given that access to content and activities in the QGS MOOC V2 was done exclusively through an LMS at the University of Calgary, the fourth characteristic, openness, was demonstrated within but not beyond this learning community in the second iteration of the course. Our vision for the ongoing development of QGS MOOC includes expansion and opening the learning community to colleagues at multiple Canadian universities, and eventually expanding to open access in the global community.

Course Content
Design changes in phase two of course development were informed by the research literature and findings from the pilot evaluation (Alharbi, 2018;Alharbi & Jacobsen, 2018). Key module issues, topics, and content are encapsulated in an introductory essay, learning objectives, a weekly webinar, a set of curated resources and materials, weekly discussion forums, and a set of expert supervisor videos. As part of the learning experience in the course, academic faculty document and reflect on current supervision practices, identify strengths, set goals, and commit to actions for their ongoing professional learning and practice as a supervisor in a Taking Action on Supervision project. A description of each MOOC design component is summarized in Table  1. We describe changes to the design from phase one to phase two in Table 2. The rationale and decisions that informed ongoing development are detailed in subsequent sections.

Introduction and Learning Objectives
Provided an introductory narrative for each topic along with weekly learning objectives and related research.

Expert Supervisor Videos
Featured academic faculty from the University of Calgary, recognized for excellence in graduate supervision. Experts included four male and five female supervisors from psychology, biochemistry, engineering, medicine, education, and nursing. Expert supervisor videos from other Canadian Universities were included.

Webinar
Invited panelists and live conversations on weekly topics and issues in a synchronous webinar; supported by course facilitators.

Discussion Forums
Questions, cases, and prompts in discussion forums expanded upon module topics. Course facilitators and discussion moderators guided, promoted, and summarized discussions each week. 6 Open/Technology in Education, Society, and Scholarship Association Journal: 2021, Vol. 1(1) 1-23

Resources
In alignment with weekly topics, curated resources were offered from the Faculty of Graduate Studies, University of Calgary, other Canadian Universities, and online sources (also available in a printable PDF).

Taking Action on Supervision
Involved in an ongoing writing and reflection project. Supervisors are invited to document supervision strengths, reflect on key learnings, set goals for growth and development, and identify actions to be taken on strengthening graduate supervision practice.

Weekly Topics
The original MOOC topics were chosen based on existing literature and an investigation into the needs of graduate supervisors in working with their students (Alharbi, 2018): 1) Introduction to supervision, 2) Supervisors' best practices, 3) Relationship building, 4) Mentoring new researchers, 5) Anticipating and addressing challenges, 6) Promoting excellence and wellness in graduate supervision. In phase two, all six of the modules were revised, remixed, and updated with new content, media, and resources. Participants in the pilot requested a specific module on academic writing with advice on how supervisors can motivate students in their writing processes (Alharbi, 2018). Informed by research on writing preparation and managing the dissertation (Ciampa & Wolfe, 2019;Roberts et al., 2019), a new "Academic Writing" module was developed for V2 of the QGS MOOC. A module on "Mentoring for Diverse Careers," was created for V2 to expand the focus from academic careers to include the many diverse career paths that graduates often pursue beyond academia (Berdahl & Malloy, 2018;McAlpine & Austin, 2018;Flaherty, 2019;Osborne et al., 2014;Sekuler et al., 2013), and the need for change in how supervisors support mentees' career development (McAlpine, 2017;Berdahl et al., 2019;. Given the increased focus on student, faculty, and staff wellbeing and mental health across our campus (University of Calgary, 2015), and more broadly in higher education (Dombroski et al., 2018;Evans et al, 2018;Hawkins, 2019;Wisker & Robinson, 2016), we expanded the sixth module to explicitly address both faculty and student wellness. We curated research and contemporary resources on how graduate supervisors can invest in their self-care and wellness (Cohan, 2019;Freeman & Clare, 2019) and for mentoring others on wellness and self-care (Flaherty, 2019). The QGS MOOC V2, which includes a combination of six revised and two new modules, included these topics in Winter 2020: 1) Best supervision practices, 2) Relationship building, 3) Enhancing academic writing, 4) Addressing and anticipating challenges, 5) Mentoring for diverse careers, and 6) Promoting academic excellence and wellness.

Expert Supervisor Videos
The QGS MOOC V2 included thirty videos of expert supervisors, recognized with awards for excellence in supervision, who shared their approaches, strategies, and experiences working with graduate students. The expert supervisors were five female and four male faculty from across professorial ranks in biochemistry, education, engineering, medicine, nursing, and psychology. The run time for produced video content in V2 is approximately 161 minutes. The development timeframe from capturing raw video, to editing and producing new media content for V2, was approximately two months. Creating the two to six minute instructional videos involved videotaped interviews with expert supervisors using prepared questions and then working with a videographer to create the instructional segments for each module.
Phase two development involved creating a welcome video to introduce the two course facilitators, provide an overview of the course and give advice on getting started with online learning. Another phase two addition was the Graduate Student video to reflect mentee perspectives on effective graduate supervision. Using a vox pop interview format, short interviews were conducted with graduate students and assembled into a composite video. Adding students' perspectives was a recommendation that emerged from the pilot (Alharbi, 2018) as well as from research on variability in graduate students' experiences with their supervisors (McAlpine & McKinnon, 2013).
New to the QGS MOOC V2 was the use of external videos of expert supervisor perspectives, which added approximately 10 minutes of curated content. Also new to V2 was the use of YuJa, the institution's video hosting, and content management system. One advantage of using YuJa was that expert videos were equipped with interactive video transcripts. This built-in universal design mechanism helped enhance the course's accessibility. In addition, the YuJa media player allowed learners to add notes using the Notes feature, which is personal and accessible only to the user. To preview the content, brief, text-based descriptions were added with each video link, along with the video duration in minutes and seconds so participants could manage their time and effort. Lastly, we included a guiding question as a prompt for learners to reflect on while watching videos. Prompts were meant to cue learners to attend to specific content in each video while relating it to their personal experiences; these questions also aligned with the interview questions that expert supervisors responded to when capturing the video.

Discussion Forums
Discussion forums are an interactive asynchronous component of many online courses. Discussion forums in the QGS MOOC were designed to include prompt questions that reflected the topic of the module and encouraged participants to reflect on their practice and dive deep into supervision practices, with discussion moderators who responded to participants (Alharbi & Jacobsen, 2020). When enrollment exceeded 100 participants, we decided to create four discussion groups of approximately 25 participants each to make the discussion threads more manageable, as well as to create four microlearning communities. Participants were expected to participate in their assigned groups, and at the same time, were invited to engage and roam freely in any of the discussion threads that were of interest.

Taking Action on Supervision (TAS)
The Taking Action on Supervision (TAS) project was added to V2 to provide a weekly learning activity that was explicitly linked to the course objectives. For this writing and self-reflection project, participants were invited to write a reflection during each module that 1) captured key learnings, 2) documented present supervision strengths, 3) set goals for developing supervision practices, and 4) identified actions to enable them to achieve their goals. The outcome of the TAS project was ongoing reflective writing about supervision strengths, and key learning and development outcomes. The TAS project served as an overall portfolio of the supervisor's learning in the QGS MOOC V2, their goals, and stated commitments to taking action in their supervision practice. The TAS project forms the basis for further professional learning and development and can be included as part of a graduate supervision portfolio, or a teaching portfolio. Additionally, the weekly reflective writing and TAS portfolio offer both intrinsic and extrinsic learning value for participants. The TAS project served as an accountability component associated with the MOOC Badge, a micro-credential that recognized completion of the course.

Webinars
In phase one, participants in the pilot requested a more real-time or synchronous connection with peers beyond the discussion forums (Alharbi, 2018). Therefore, in phase two we added a weekly synchronous session, comprising a one-hour Webinar with the two course facilitators and several of the expert supervisors from the videos to serve as panelists. In collaboration with the course facilitators, the panelists planned remarks for approximately 20 minutes of presentation on the weekly topic followed by approximately 40 minutes of real-time conversations and interaction with participants. Webinars were recorded and posted online for on-demand access by participants who were unable to join live or who wanted to review the webinar at another time.

Micro-Credential
The QGS MOOC Badge was a new addition to version two. The researchers collaborated with the Taylor Institute for Teaching and Learning to outline what completion of the course entailed so that participants could earn a QGS MOOC Badge or micro-credential. We estimated it would take two to three hours per week, with a total time of about 18 hours to complete the QGS MOOC. As part of creating the Badge, we considered and weighed the learning outcomes, the features of this hybrid instructor-led and self-directed MOOC, and the design components that were essential to the overall learning experience. The criteria for earning the QGS MOOC Badge were to complete four tasks in each of the six modules: 1) Watch the expert videos each week, 2) Participate in the weekly discussion forums, 3) Attend the weekly synchronous session (or review the recording), and 4) Complete the "Taking Action on Supervision" project.

Navigation
To improve navigation and accessibility in phase two, non-developer D2L design templates were used to create a contemporary and consistent look for all content pages in V2 of the QGS MOOC. The templates are designed to meet Universal Design for Learning standards (Rose & Meyer, 2002). The QGS MOOC V2 included a new course homepage banner created with a photo from the University of Calgary Media Relations. A Support link was added to the navigation bar on the course homepage and appears at the top of every page for quick access to help and support resources. YuJa was used for the integration of videos which provided closed captioning and support for visual accessibility. Finally, relevant quotations from the expert supervisor videos offered an accessible and time-effective way to introduce content and to amplify a social connection with speakers and resources each week.

Design and Development Summary
The review of current research on graduate supervision, study findings from the phase one pilot, and review by three subject matter experts prior to launch served to inform the ongoing design and development of the QGS MOOC V2 offered in Winter 2020. Trade-offs between design requirements, resources, and time influenced which changes and improvements were prioritized for V2 (Table 2), and what changes had to be deferred for future development. Outcomes of the current research will also influence the ongoing design in phase three of development. The analysis and evaluation that follows amplify participant experience and perspectives on the design components that enhanced learning in the QGS MOOC.

Evidence of Learner Engagement in Design Components
A majority of post-MOOC survey responses and all of the interviews were from participants who had completed the course. Of the 46 participants who completed the post-MOOC survey, 90% had completed all six modules to earn the QGS MOOC Badge. With regards to assessing learner engagement, we were interested in how participants interacted with various design components and participants' perspectives on the flexibility and accessibility of the online course.

Expert Videos
The majority of survey participants expressed satisfaction with the expert videos and how these provided different approaches to the same supervision issue. Several interview participants mentioned they liked the length and professional quality of the expert videos. One experienced supervisor we interviewed said: One of the things I liked about them too is it was usually two or three, maybe even four people's different perspectives on things. And so it was interesting, they don't all handle things the same way. They don't all necessarily see the same things as important. Sometimes they had quite different approaches, but it...helped me that it was short and made it easier to fit it in and keep my attention.

Asynchronous Discussions
Given the size of the Winter 2020 cohort (n=105), we organized participants into four discussion groups to support active engagement and interaction among members in smaller microlearning communities while at the same time making the discussion forums, threads, and posts more manageable. Discussion activity and interaction in each module were lively, with the number of active participants declining over the duration of the course. The most active discussion, with 276 responses in 22 threads (see Table 3), occurred during the first module in response to the invitation to reflect on keywords to describe effective supervision. This prompt connected what students said about effective supervisors in their video with the words supervisors selected to describe effective supervision. As part of instructor presence in the course, keywords were collected and represented visually in a word cloud to share back the collective knowledge building within the learning community (Figure 1). Frequency is related to word size; thus, the three words selected the most to describe effective supervision were supportive, guide, and mentor. Participants valued the exchange and discussion of ideas and strategies with other supervisors in the forums, and with expert supervisors who served as discussants. As such, many participants noted that they believed the discussions were an essential component of the learning experience. Participants indicated they appreciated the safe learning community that was formed and the relational trust that was cultivated in the discussion forums.

Webinars
Participants reported the webinars were helpful for engagement and interaction with academic peers. For example, an experienced supervisor told us in an interview: I really felt like I was, that I had other people in the room, you know, and when I was listening to others, I felt that we were all in that room together. It just kind of gave that personal, you know, you heard the voice, you saw the facial expression. It was just more relational. I thought the weekly webinars were excellent and the expert videos as well.

Key Words Participants Used To Describe Effective Supervision
Another supervisor spoke to the value of access to different perspectives on the same issue: The Weekly Webinar was fantastic. I really enjoyed being able to connect with everybody that was on the call to be able to learn from the different experts. So whatever expert happened to be speaking at that particular time or on that particular subject. I really enjoyed the chat conversations and, you know, just the conversation in general. ... it was actually really nice to be able to connect with other people on campus. And in this kind of capacity talking about graduate student supervision.
Given the variability in academics' schedules, we anticipated that some participants would be unable or unwilling to attend the live webinars. "I even appreciate that attending webinars was not mandatory and I could review them later as, some days, I was super busy at the assigned time for the webinars". Participants indicated they appreciated the freedom to choose live or to access on-demand recordings.

Micro-Credential
A multipart interview question focused on the QGS MOOC Badge. We were interested in design and impact: 1) whether the addition of the Badge was a motivator for completion, and 2) whether participants valued the Badge for their teaching portfolio, supervision portfolio, their CV, their Annual Progress Report, and so on. Earning the badge was identified as important by 52% of post-survey participants. Several indicated formal recognition of their learning was important for their CV or supervision portfolio, and the micro-credential was valuable for renewal of supervision privileges. One participant explained, "I knew it would be viewed favorably for my APR [Academic Performance Report] and wanted to be able to write something in my next renewal of supervisory privileges in terms of professional development." Another participant expressed, "it is important for developing my graduate supervision profile and continuation of supervisory privileges in the future." One participant said, "first, it is a celebration of achieving my goal, and second, I can use it as evidence of personal growth." Several participants reported the Badge gave them both structure and motivation to complete the entire MOOC.
Twenty-three percent of survey respondents reported earning the badge was not important to them. One participant explained having a reward is nice, but they do not know what a digital badge is or how to use it. Another participant was unsure the badge counted: "I've never seen an academic's CV with the word Digital Badge on it and I have seen a lot of CVs." Several participants explained the badge was not a motivator for completing the MOOC. "I wanted to engage in the learning regardless of whether or not I earned the digital badge." The variety of perspectives on digital badges suggests that learning and goal setting is an individual experience and that a micro-credential may or may not be a motivator or currency for professional learning.

Engagement Within the Learning Community
MOOC participants valued the opportunity to engage with colleagues from across campus in the discussion forums, through the videos, and via interaction in webinars. Supervisors valued exposure to different perspectives, approaches, and strategies for mentoring and guiding graduate students. An experienced supervisor indicated the highlight was conversations with a diverse group of colleagues: "that's one of the things I like best about it, was the fact that there are people from a lot of different faculties, a lot of different programs which have very, very different expectations. And it was nice to see things through their lens a bit." In an interview, an experienced supervisor explained: What I really appreciated was that diversity and just how different things are for different people and how somebody approaches things that might work for them are just not going to be possible for me or the other way around. And, you know, I've been doing this for a long time, but that doesn't mean that I didn't learn a lot. Another experienced supervisor also described "... it was definitely looking at and learning about other perspectives, not just strategies, but other perspectives and philosophies that other graduate supervisors had with regards to what their role was in supporting students." They also added that: There was tremendous diversity in an approach which is great. But it became very clear to me that I think whoever we are, we may have a particular approach, but not all of our students are going to walk in through that same door…And so I think it is the experienced supervisor and the supervisor who aspires to excellence recognizes that and doesn't hold students to a rigid kind of expectation.
Another experienced supervisor emphasized the importance of learning from various opinions and explained: It was very interesting to me to listen to some of my good colleagues whose approach was very methodical. And I don't judge that. I think, wow, good for you. And then to see the varied approaches of others that focus a little more heavily on the relational in order to navigate those milestones. And yet, at the end, your student reaches successful completion.

Flexibility and Accessibility
All survey and interview participants emphasized flexibility and accessibility as benefits of the course and appreciated the opportunity to learn online at their own pace. One indicated that, "I probably did 99% of it on the weekends or in the evenings, which is something that I wouldn't have been willing to do if it had been on campus. And I would have found it a lot more difficult to carve out an hour or two, three times a week, during normal working hours. If this had been scheduled during the day, I probably wouldn't have done it." Another explained how they appreciated the flexibility of the course and the fact that it was offered online: It worked really well for me as I am a lab bench-based, basic scientist. I can catch up on the work in the evening and weekends. On the scheduled webinars, I can log in with my devices and participate whilst doing the more monotonous tasks in the lab -this is a very good use of my time. It is also time-saving as I don't need to travel for the classes.
Other supervisors highlighted the advantage of being able to access the material at one's own pace. "I could start and stop and do a few minutes here and a few minutes there. And that worked for me." While the flexibility of the MOOC helped many of the participants to complete the MOOC, one participant explained it was a challenge to self-pace in the online environment: The online format is excellent… not everybody is on campus all of the time. So it provides that equity of access…it's more inclusive in that way, which is great. Now one of the downsides… when there is so much flexibility, it sort of challenges one to be accountable, to set up their time to then engage in it.
The COVID19 pandemic was a disruption for many MOOC participants. In the middle of the MOOC, all courses and supervision at the University pivoted to remote delivery as physical distancing protocols were put in place. Several participants requested an extension to finish the MOOC and a few dropped out from the impact of the pandemic. One participant explained, "Given the crazy of COVID and everything I had to manage this past two weeks I appreciated the flexibility and ability to complete things at my own pace." Others reported they had more flexibility and time for the MOOC with campus restrictions and the requirement to work from home. For example, one participant mentioned, "So I think the flexibility, the style of the MOOC allowed me to successfully finish it despite some of these other unforeseen challenges that sort of popped up. Like I didn't know that I was going to be quarantined."

Evidence of Impact on Practice: Key Themes
The research team was interested in how peers interacted with peers in the interdisciplinary online learning community and whether the learning experience in the QGS MOOC V2 had any impact on supervisors' work with students. For example, one participant described how they interact differently with their students now given what they learned in the course. I'm trying to differentiate between different students because different students would need different things based on what I've learned here, right? So, some students need more hands-on supervision. Some students don't like that. And I'm actually trying to ask students what they would like based on what I've learned in this course. Of course, still with my goals for them in mind as well. Yeah. I think that is something that has changed in my approach to supervision.
The analysis and integration of survey and interview data on reported impacts on practice and the types of actions taken to improve supervision practice yielded four themes: 1) increased awareness of supervision strategies, 2) establishing good communication, 3) the value of reflection on supervision practice, and 4) increased attention to wellness and self-care.

Increased Awareness of Supervision Strategies
The most prominent theme to emerge regarding the impact on practice was that engaging in the MOOC increased participants' awareness of diverse supervision strategies, tools, and ideas about supervision from across disciplines. The majority of survey and interview comments highlighted either a specific new concept that participants had gained or an increased awareness of effective supervision strategies that are common across disciplines, such as the need to support students with academic writing. Several supervisors indicated that the MOOC learning experience reinforced ideas they already had about effective supervision, such as the need to be aware of students' experiences and listening for how best to provide support. New supervisors commented on their changed ideas about supervision as a result of what they learned in the MOOC. For example, "I have learned that my experience as a supervised graduate student is not necessarily the norm and that I can be a better supervisor based on the knowledge I now have." And another, "It has modified my initial perspective of graduate supervision and I hope to go forward with a renewed approach to training students to become the best of themselves." Supervisors indicated they appreciated becoming more aware of diverse images of effective supervision and claimed that by finishing the MOOC they had more strategies and ideas to draw upon in working well with their students. For example, participants valued opportunities to discuss and exchange strategies for supporting graduate students with academic writing. "Some of the writing ideas were gold and I will be applying them." Supervisors described their commitment to encouraging students to enhance their academic writing skills, to employ strategies such as peer editing so students can help one another, and resisting the urge to edit their students' work in favor of discussing the types of edits needed for continual improvement. Supervisors often described ideas that colleagues had shared in a discussion or webinar that they would like to adopt in their practice. For example, one supervisor indicated "I'd like to try the walking meetings. Just right now with the self-isolation, that's not possible but I think it'd be nice to try." Another supervisor described several ideas they were taking away from their experience in the QGS MOOC, such as "I will start to make changes in my lab, including writing as a daily task, one minute research project pitch, and social sessions for wellness and support."

Establishing Good Communication
The second most prominent theme to emerge about impacts on supervision practice was the need to establish strong communication with students from the start of a relationship. Survey and interview comments associated with this theme included the need for supervisors to engage in active listening to learn more about their students and their needs, to offer person-centered support along with program advice, to establish a regular meeting schedule, to discuss expectations on both sides, and to become more intentional about building relationships with students. Supervisors identified the need to be more deliberate in how they communicated with students, such as learning more about student interests outside of graduate school, by focusing on managing wellness in graduate school, specifics about program and progress, and initiating conversations much earlier (i.e., with new students) about career interests and non-traditional career options. Supervisors indicated a renewed commitment to being more curious and assuming less about their students, initiating or establishing regular meeting patterns and networking opportunities, and sharing resources more directly with their students.
Several supervisors described their plans to establish strong communication with students by formalizing a project management plan for following up with each of their students, to communicate about time management and balance, plan activities for students to engage with each other, and also plan for discussing expectations upfront to promote collaborative work. Supervisors also indicated a clear understanding that relationships need input and commitment from both sides, such as setting clear expectations, using shared documents to ensure understanding goes both ways, and increasing the number of times they check in on students as some of the cogent lessons they learned in the MOOC and plan to incorporate into their style of graduate supervision.

Reflection on Supervision Practice
A third theme that emerged was the value of reflecting on supervision practice and supervisors setting goals for their continual improvement. Several supervisors commented that the Taking Action on Supervision (TAS) project gave them a sustained opportunity to reflect both critically and positively on their supervision practice. A common thread among responses in this theme was supervisors' renewed confidence that while they had much to learn, the MOOC reinforced that they were doing a good job at supervision. Experienced supervisors reflected a realization that they are supervising in changing contexts, and that the MOOC confirmed that they had a great deal of experience to share, while at the same time they still have a great deal to learn from interdisciplinary peers. Several supervisors identified the reflections and goal setting in the TAS project were valuable, along with the actionable items they created for enhancing their practice. One participant described the impact of the MOOC learning experience and their reflections on their current supervision practice in an interview: I am mentoring three students and working with them on a scoping review. I think I would have handled it a little differently if I hadn't taken this course… it really had an impact and challenged my thinking on the role of the graduate student and the role of the graduate supervisor. I went from my own supervisors, right? I had very different experiences with different people. And so that is where I learned how these things should be done… And this course challenged that… I was different with these students and I'm pretty sure it's because of the course.
On the survey, several supervisors indicated a commitment to revisiting their action plan on a regular basis to ensure they are supervising students in ways that align with their goals. Several others described how they plan to keep updating their knowledge and skills about graduate student supervision by building upon what they learned and by connecting with peers across campus.

Wellness and Self-Care
The fourth theme that emerged with regards to impacts on practice was greater awareness of the need to consider both academic excellence and student and supervisor wellness. "I really enjoyed seeing that balancing excellence and wellness as part of the modules." Several supervisors indicated that the MOOC introduced new ideas about balancing excellence and wellness, and also increased their awareness of the need for supervisors to become more aware of student experiences and challenges. One supervisor shared that a key insight and realization was they needed to become more aware of the specific difficulties that each student may be facing to better tailor their supervision strategies and approaches. Many supervisors identified closely with the discussion on promoting wellness and self-care along with excellence by checking in with students and talking about the personal first in meetings before transitioning to projects, tasks, and deadlines. Several supervisors made an explicit commitment to finding balance in both excellence and wellness as part of their revised supervision philosophy. "Communication and self-care are the cogent lessons I have learnt and will be incorporating into my own style of graduate supervision." For some supervisors, the intentional focus on scheduling social events or meetings was a strategy they planned to add to their practice.

Discussion
This evaluation of a faculty development MOOC in graduate supervision offers several key findings about what factors enhanced the online learning experience and the impact it had on practice. We found evidence that each of the six design components enhanced learner engagement in some way. Participants indicated the expert supervisor videos, webinars, discussion forums, and the Taking Action on Supervision project, were highlights of their online learning experience. During interviews, MOOC participants advised that every supervisor, new and experienced, could benefit from engaging in this online course on graduate supervision. The MOOC learning community offered supervisors across the career span an opportunity to share their disciplinary experiences and perspectives on supervision, and to be heard in ways that were not readily available in more formal learning settings and contexts. Participants reported that what they valued most was the opportunity to engage thoughtfully with interdisciplinary colleagues who brought different perspectives, approaches, and strategies for supervising and mentoring graduate students to the learning community. One supervisor shared how they led a major overhaul in the way leaders in their department designed the supervision program and how they worked to reconstruct a more human-centered design that incorporated multiple perspectives on supervision based on what they learned in the QGS MOOC.
Supervisors emphasized that the flexibility and accessibility of the QGS MOOC worked well for their learning even with their busy schedules, and especially when their work and home lives were disrupted by the pandemic and the pivot to fully online engagements. Participants appreciated the opportunity to learn online at their own pace and on their own schedule. They valued engagement in the online learning community through webinars, discussion forums, and access to expert videos. An unexpected benefit from engaging in the QGS MOOC during the pandemic pivot was that several academic faculty and postdoctoral scholars indicated they were able to adapt quickly and more smoothly to remote teaching because of their first-hand experiences as an online learner in the MOOC. Participants' experiences learning online with the instructors and academic peers in the modules, during Zoom webinars, and in the discussion forums, gave them timely strategies and ideas for engaging effectively online with their students. The pandemic pivot which required supervisors to work remotely with their graduate students, and the flurry of discussion that this shift immediately cultivated in the MOOC's discussion forums, has prompted us to gather resources and research on online mentoring and supervision (Kumar & Johnson, 2019;Kumar & Coe, 2017;Maor et al., 2016) to inform the design of a new module for the next iteration.
Our results demonstrate that the QGS MOOC offered graduate supervisors an opportunity to grow and develop their supervision practice through interdisciplinary engagements and connected learning outside of their regular experiences (Salmon et al., 2015). Our findings indicate that faculty learning in the MOOC has an impact on supervision practice and the types of actions supervisors have taken or plan to take to improve their practice. Four themes emerged about the impact on practice, such as increased awareness of common supervision strategies across campus along with new ideas and approaches. Second, increased understanding of the value and importance of establishing good communication and active listening to discern student needs, and to offer both person-centered support and sound program advice in their work with students. Third, supervisors gained from reflecting on their learning and supervision practice and by setting goals for their continual improvement. Fourth, supervisors set in motion strategies for their own and their students' wellness and self-care to balance and accompany their focus on excellence.
We experienced a completion rate of 47%, which is higher than average. With expanded temporal and geospatial flexibility and learner autonomy comes the opportunity for dropping in and leaving the MOOC. Low completion rates tend to be a characteristic of many MOOCs (Reich & Ruiperez-Valiente, 2019). However, early leavers can still benefit from a MOOC. Following a 100% attrition rate, Jacobsen (2017) found some participants reported gaining insights to improve their practice and using the course as a prelude to self-directed online learning. Similarly, MOOCs provide opportunities to expand upon self-determined professional learning goals (Milligan et al. 2013). Thus, depending on the goals or learner perspective, not completing the QGS MOOC does not automatically equate to not learning something of value about supervision.
In this paper, we have reported on phase two of the design, development, implementation, and evaluation of the QGS MOOC in Winter 2020 at the University of Calgary. Our research contributes insights on design components that help create the essential mentorship and learning conditions for graduate supervisors' collective knowledge building and ongoing improvements in supervision and mentoring practices. By extension, although we did not evaluate this directly, we contend that improvements in supervisor awareness and commitment to taking action on diverse strategies in their practice will contribute to improving graduate student experience and wellness, as well as improving students' academic success. Through active knowledge exchange and discussion with faculty peers across disciplines, supervisors expanded their understanding and strategies for supporting students in self-care and wellness, along with timely completion of program, knowledge mobilization and grant writing, and diverse career preparation, through an intentional focus on their own and others' learning and development as effective supervisors.

Conclusions and Recommendations
In the present study, we combined the compelling need for ongoing faculty support in supervision with what is known about the features of MOOCs to leverage accessible and flexible online learning during a pandemic. The current study captures the perspectives and voices of graduate supervisors immediately after they participated in the QGS MOOC. While we are aware of the impact of the course on some participants' practices, we do not know the extent to which goals and actions articulated by supervisors in the TAS project have impacted their practice in the medium or long term; more research is needed to follow up with participants who completed the course to examine how any changes in their practice have been sustained. Implications of our research for further design, development, and practice in using this MOOC for faculty development in graduate supervision include the in-depth analysis and findings of the effectiveness of various design components, as well as findings of levels and types of learner engagement. In subsequent offerings of this MOOC, we plan to expand our inclusion of graduate students' perspectives (McAlpine & McKinnon, 2013) on supervision. One idea for continuing the community of practice beyond the conclusion of the course and extending that energy and learning into future offerings of the MOOC is to invite supervisors who earned the QGS MOOC Badge to engage as mentors with new cohorts of supervisors in the future learning communities. Some MOOC graduates may be satisfied with what they learned and want to move on. For others, returning to the MOOC learning community as a mentor would enable them to have a role in extending the network and community of practice.
Academic faculty on the tenure track balance diverse roles and many responsibilities. The Quality Graduate Supervision MOOC offered flexible and accessible online faculty development for both new and experienced graduate supervisors, especially during a global pivot to online teaching and learning during a pandemic. Results indicate the MOOC is an engaging and impactful learning experience that transcends disciplinary and faculty silos. The MOOC provides an innovative and responsive, transdisciplinary learning community that enables interdisciplinary faculty to engage in and contribute to a collaborative learning experience and to collectively strengthen the quality of graduate supervision available for graduate students. We argue the QGS MOOC offers supervisors an opportunity to share their expertise and experiences and engage in a transdisciplinary community of practice in which supervisors gain much from colleagues who share their commitment to effective graduate supervision. For universities, our research provides evidence of the effectiveness of MOOCs for online faculty development in higher education We recommend greater institutional investment in providing widespread access to flexible online learning experiences for all faculty.