Source: University of Waikato
While Alert Level 4 saw University of Waikato staff and students obliged to do everything differently, for the University’s Information Technology Services team the lockdown posed a set of challenges nobody could have foreseen.
What began in early February as exploring access to online learning for University of Waikato students at partner university Zhejiang University City College (ZUCC) as Covid-19 took hold in China, became a unique set of circumstances that led the team to deliver a series of solutions with extraordinary turnaround times.
Getting everyone online
Chief Information Officer Eion Hall says the ITS team’s energy and effort were initially centred on delivering content online to ZUCC students through China’s well-known firewalls. As the implications of the virus spread started to become clearer, the focus shifted to planning for the entire University to move its operations online.
Over a period of four weeks, ITS staff oversaw and facilitated the migration of more than 12,000 students and staff to a totally digital environment, all while the team itself was also contending with moving into lockdown as well.
Eion says the team’s work to rapidly expand existing infrastructure to cope with huge new loads on University systems was mindblowing. As an example, February saw Zoom usage across the University of 473 meetings, 1,478 participants and 74,000 minutes. By April, this had ballooned to 14,738 meetings, 124,726 participants and a staggering 4,835,357 minutes.
Scaling up like this had to happen across the board, from ensuring staff had sufficient hardware to work at home (for two days before lockdown staff in the ITS store handed out hundreds of keyboards, webcams, blue network cables and other equipment), to people suddenly becoming heavy users of other online tools like Panopto (a lecture recording platform) and Hangouts.
A series of unfortunate events
Amongst these logistical challenges, came a sudden major incident in the University’s main data centre just as New Zealand went into lockdown. A faulty fire suppression system in the data centre spontaneously discharged its fire retardant gas, creating a pressure wave that caused network storage equipment to fail. This meant staff newly set up at home were unable to access fileshare drives and Panopto storage.
Dougal Mair, Associate Director Applications and Infrastructure, says the data centre equipment has some fault tolerance, but not enough to cope with the amount of damage done in this incident. As a result of the outage, it was then discovered that data replication to backup Panopto recordings and fileshares had not been functioning correctly and wasn’t up to date. This led to an effort from two teams of ITS staff to get the primary storage up and running, and set up an alternative storage solution for Panopto recordings, with both goals achieved following significant hours put in by each team.
No sooner had this incident been wrapped up, a power surge in the ITS building meant ITS staff, working continually to provide support across the University, lost VPN access to their desktops. This was followed a few weeks later by another false fire activation in the comms room of another building on campus. In both instances an electrician was required on site to restore power and get things back up and running quickly.
Months of work done in days
The entire ITS team mobilised during this period, either picking up other work for those assigned to resolving issues, or assisting Service Desk staff to continue supporting staff and students in their new environments. The Service Desk team alone grew from six people to 30.
Alongside the efforts made to re-establish systems when they failed, work continued on other lockdown-related tasks with short turnaround times – sourcing Chromebooks and internet connections for students without the tools to learn online, rolling out a PC-based outbound phone system to assist the Student Services team and teaching staff with contacting every student, launching remote access to computer labs, and implementing a contact tracing system to comply with Alert Level 2 requirements.
The contact tracing system was developed in less than a week. The project team worked around the clock with the Property Services team and Datacom to populate around 5,000 spaces with QR codes and ensure the system was functional in time for campus to reopen.
Many people, one team
Eion Hall points out the team, just like everyone else, had their own bubble challenges to contend with, including the competing demands of children, their partners’ work situations and lack of space to work. A constant conversation around families and personal lives taking first priority was weaved into situations where staff were working intensive hours, including weekends, for several weeks, dedicated to ensuring the University community could continue its operations online.
Both Eion and Dougal credit good relationships with vendors including Dell, Datacom, Gravitas, PBTech and 2degrees with enabling quick recovery from unforeseen issues and swift activation of new initiatives and solutions. But they agree the team’s willingness to go the extra mile and continue providing excellent personal service is really what made an unbelievably challenging time something to be incredibly proud of.
