Computer Bug Locks King’s Login For Three Days
IT desk at The King’s College I Photo Coutesy of Melinda Huspen
A technological issue with The King’s College’s single sign-on services prevented various students, faculty and staff from logging into online college systems for three days between Wednesday, Nov. 9 and Saturday, Nov. 12.
The login errors caused disruptions in regular proceedings, specifically when any student or faculty/staff member logged out of their school account and could not log back in. The outage mainly affected email services, but other third-party applications that required a TKC login such as the student portal and Schoology were impacted as well.
“Whoever logged out of their email Wednesday and tried to log back in kept getting this error message,” said Bracey Fuenzalida, Director of Information Technology (IT) at The King’s College. “They kept getting a message that said, ‘The service is currently unavailable. Please check with your administrator.’”
“I got here a few minutes before 9 a.m. on Wednesday morning,” said Cole Smith, Educational Ventures Administrator for The King’s College. “I logged into my computer as usual and went into Google Chrome, and it usually has my saved profile there with all my usual tabs. I clicked on a couple of those and nothing would load for me. It was asking me to put in my information like my username and password again. And then the screen would essentially not change… So I went and talked to Bracey about it for a little bit and we reset the password quite a few times, changed quite a few things, but couldn't get anything to fix it.”
The source of the problem ended up being a switched setting in the school’s Active Directory and Federated Services single sign-on system.
“Something was flipped to a completely different property,” Fuenzalida said. “It was in a place that we never touch, so we definitely know that it wasn’t a human error.”
When asked how the error happened, Fuenzalida said that the IT team “still doesn’t know how it was switched… When a setting changes the way it did, we check our created processes to see if we may have inadvertently created a rule that changed a property of something. Nothing we had in place changed this setting or its properties.We are still running all sorts of tests right now to see what could have caused that change.” His goal is to proactively patch the system for this error to not occur again.
Fuenzalida also stated that the issue was definitely not caused by a hacker. “We have six different layers to detect hacking,” he said. “If there is an intrusion, one or more of those layers informs us of it, and we didn’t have any of that. We also did not have any data loss, security breach or information theft.”
Fuenzalida and the other two members of the school’s IT department worked around the clock during the shutdown, reaching out to both Google and Microsoft customer support for assistance.
“Google told us right away, ‘Not our problem,’” Fuenzalida said. “I called Microsoft Office 365 three times, and each time we went for two hours checking through everything.”
Fuenzalida sent out three school-wide emails with updates regarding the issue throughout the week. The first email, sent on Wednesday at 10:30 a.m., notified the school of a “Google Workspace disruption.” The second email, sent the next day at 5 p.m., instructed all students, staff and faculty to avoid signing out of any school systems until the problem was resolved.
“Honestly, I’m realizing now that this didn’t affect me that much because I don’t do all of my homework on Google Docs,” said Paige Hagy, City Editor for the Empire State Tribune. “I keep it on my computer with Microsoft Word, but I know that for people who do store all of their homework there, it was a huge inconvenience for them.”
“The evening before Interregnum I heard that other students were having issues and tried to login myself,” said Evan Louey-Dacus, a senior in the House of C.S. Lewis. “I use my email for job applications, and all my academic work is on my King’s Google account. One of my essays was actually late due to the issue.”
“At a certain point I was pulling my hair,” Fuenzalida said. ““We're looking, we're looking, we're looking. Finally we just said, ‘Let's start, one by one, looking at the account credentials that allow the passing of information from our server here to the server in the cloud and let's see what happens.’ We kept trying and trying until finally we found the problem. It was the properties of a particular account that had been switched.”
After flipping the setting back, the team used test accounts to ensure that the login system was properly working again. At 2:12 a.m. on Saturday morning, Fuenzalida sent out a third email to notify the school of the news that the login interruption had been fixed.
“I woke up Saturday morning and I was relieved,” Fuenzalida said. “All the credit goes to the two guys that I have on my team. These guys were relentless in their pursuit of what they were looking for… One of the guys is Marat. I affectionately call him Drago from Rocky, because he's from Russia and he's a genius. He does things with systems and computers that I wish I could do in my dreams. I also have Richard Christensen who runs the help desk, and he is as good as anything. He has an acumen for cybersecurity and for being able to understand how things work. When the three of us are putting our heads together, it bodes really well.”
An email released on Sunday, Nov. 13 announced that due to the email access interruptions, the start of class registration for the Spring 2023 semester would be delayed from Nov. 14 to Nov. 16.
“Before registration opened, we wanted students to be able to get answers to any registration-related questions and staff to be able to remove any registration holds that could otherwise have been lifted if our systems had been fully operational,” Dr. Matthew Parks, Interim Provost of The King’s College, said in a statement. “We made the decision at the end of the day on Friday.”
Because of regular communication from the IT team and anticipation of potential issues, the login interruption did not significantly impact the execution of Interregnum over Nov. 11 to 12.
“It wasn’t too much of a struggle,” said Kaylee Vroon, co-chair of the Interregnum Committee. “We switched over to everyone’s personal emails for the day until it was fixed. The Interregnum email stayed logged in the whole time so we had access to what we needed. We did have plans in place for if houses couldn’t submit because of it, but we didn’t end up needing them.”
“We had a guy on site here for Interregnum,” Fuenzalida explained, referring to Christensen. “He was a judge and he was also watching our system. We didn’t have a hiccup.” (The connection issues during the mandatory film screening were completely unrelated to the server login interruption.)
Although the back-end operations of Interregnum ran smoothly, some students still had to work around the login issues to participate in the competition. “I lost access to my email right before the start of Interregnum,” said Rachel Wells, Chamberlain for the House of Queen Elizabeth I. “Completing academic writing in collaboration with other students was a challenge during Interregnum. I actually had to borrow a friend’s computer and her email to complete our submission.”
Classes were also able to run as normal over Wednesday and Thursday. “One or two of the professors said that they had saved their presentation for class on USB sticks,” Fuenzalida said. “I think what happened is because I communicated clearly and effectively with everybody ahead of time… people were able to plan ahead.”
“We almost take it for granted,” Fuenzalida said, referring to the IT systems the school operates on such as phone, email and internet service. “When everything runs as normal nobody says a word. It's only when there's a disruption that people notice. But such is the nature of the business we have chosen, as they say in that famous movie.” He gestured to a picture of The Godfather on the wall behind his desk. “This is the business we have chosen.”
Melinda Huspen is the Campus Editor of the Empire State Tribune. She is a junior at The King’s College studying Journalism, Culture and Society.