When I took this job as the Academic Director of LTIC I thought about whether it would be a stressful job. And ultimately I came to the conclusion that it would have its moments, but (literally said aloud) it’s not like Canvas is ever going to go down. Of course, as a computer scientist, I know that software fails. And so there was a possibility that Canvas would too. But how bad would it be? How long could it last? A couple of hours of outage at a difficult time of term? A day?
Time has shown that I was wrong to be so cavalier. Naive even.
For many Canvas users, May 7th might become known as the day that shall live in infamy. When a ShinyHunters ransom note appeared on the screen of every user, freaking all of us out, and making us feel like we were in an action movie that nobody ever signed up for.
The Canadian situation was different from our colleagues to the south, who were in final exams. What a terrible time for them. But our semester was about to start. The ransom note appeared on a Thursday, with our Summer term starting on Monday. And we had no Learning Management System to take us through. We had a weekend to get faculty into some kind of safe operating mode. We couldn’t build a consumer grade LMS, but we could do something, so that they could go to class, and concentrate on teaching, as opposed to the failures of technology.
At this point we realised how many aspects of running a university, beyond course delivery proper, actually sit within the LMS.
- Where do students go to find their course delivery platform? LMS
- Where do students go to communicate with their instructors? LMS
- Where do instructors go to communicate with their students? LMS
- Where does the administration go to communicate with all faculty and students at once? LMS
We needed to replace all these functions, fast.
With blinding efficiency and a spirit of solidarity that makes me misty eyed when I think about it, we arranged ourselves into a loose ad hoc institution-wide recovery team, that spanned well beyond LTIC, including (but not limited to!) Cybersecurity, the Office of University Counsel, teams across IT, PrISM, Enrolment Services, Communications, all of the Instructional support units across the university, and our sister-service the CTLT. So when I say “we” and “our” for the rest of this post, I mean that team of around 70 people.
We worked, without stopping, all weekend, and then continued to keep up that fever pitch through the rest of the disruption. A task that would normally take 18 months, was achieved in under two weeks. The dreaded action: migrating platforms. I knew from the tens of emails from faculty that our pace was matched and exceeded by each individual instructor.
The broad team’s first big move was to stop-gap all those fundamental functions: navigation, and tools. We made, from scratch, findmycourse.ubc.ca to help students find their way to their courses. We published an alternative course hosting page, redirected the forbidden URL to a student facing website, and issued daily digests to keep the community up to date. We started a discussion forum (almost a support group) for the thousand instructors and instructional staff, so we could flow information, provide technical help, poll for strategy advice, and recruit beta testers. Through that medium, the broad team was able to experience, first hand, how much belief and trust faculty have in us. We had initially created the forum to house and address panic and confusion, but instead it became a place of relationships and togetherness.
In parallel, we spun up two options for hosting course materials, and many options for the satellite functions such as media and discussions. We were very lucky to have in our back pocket a very mature Blogs site ready and waiting for automated integrations like enrollments and media. We had an open source Moodle install that we rapidly expanded. Vitally, we got all the materials that were in Canvas out of Canvas, and made those directly available to each individual instructor.
Even at this eye watering pace, every act of innovation was underpinned by privacy impact assessments, data access scrutiny, security threat risk assessments, and solid software testing and release practices. We were clear: we would not introduce vulnerability. Trust in the technology was as important as its function.
What struck me during this time was the many conversations I had with faculty friends and colleagues who shared, almost with a sense of shock, that they never realised how much, pedagogically, they had sunk into this one tool. Their materials were their own – but they were inside the LMS. They were written in the structures provided by this one platform. Once extracted, they fell apart. The quiz logic got lost. The modules flattened. The intent and the flow was all encoded in a language that was interpreted by a tool that could just … go away.
After a week, the whole team began the process of making Canvas available again. No easy feat. Integrations were now broken, enrollments (30k of them) were now out of sync, and everybody was fatigued to the point of real concern. In a final push of adrenaline, we did what would normally take yet another four months, in a day.
There’s still a lot of work ahead, and a lot of hard won lessons to be anchored. We now know what it takes to unplug, and plug back in. We know what frantic questions instructors and students will have, and where the fires will start. We know what gaps we were not able to fill, and we can work on prepping for those for the next time, because there will, inevitably, somewhere in our ecosystem, be a next time.
Far from feeling a sigh of relief that everything is now restored, I think instead many feel disillusionment. I, for one, have lost my cavalier sense of naivete, for worse or for better. But much of this experience has, paradoxically, given me a sense of hope and renewed resolve: I have learned that in a crisis, the vast technical core of UBC will come together to move as one, with strength and commitment at a scale that is just unimaginable. I learned that my own LTIC team is all in for faculty, for students, and for UBC thriving, to an extent that I don’t think anyone would ever suspect from the outside. I can see that UBC is in a position to make the necessary moves to secure and elevate our learning technology to unlock creativity, pedagogical expression, and self-determination. And I believe strongly that this job as Academic Director can return to being the inspiring, but maybe not so hair-raising, role that I signed up for.
Share your thoughts about, and experience of, the outage – this will give us insight to help us plan future strategy.
Read Darcy Norman’s take on innovation for institutional resilience and highlights of UBC’s response to the outage.