Public kept in the dark on data centre outage details
Documents obtained under Right to Information reveal efforts to contain crisis details
Government employees kept the public in the dark over details of the Marysville Data Centre outage in June, a CBC News investigation shows.
The data centre’s computer systems hard-crashed twice the morning of June 9
The crashes occurred when the Marysville Data Centre’s (MDC) backup power system experienced a series of failures after an unplanned power outage.
Computer programs used in the day-to-day functioning of government in the departments of Public Safety, Health, Justice, Social Development, Finance, and others, took hours, and in some cases days, to be fixed and restored, at an undisclosed cost to taxpayers.
Internal correspondence among communications staff shows deliberate efforts to contain the greater details of the breakdown, while releasing basic information only and deciding what information was “relevant” to the public.
In emails to colleagues, then-Department of Government Services communications representative Craig Chouinard commented:
"Have given the general lines on the outage and work to restore to CBC but am avoiding getting into the weeds on the generator, etc., with any of them. CBC wants someone on Info Morning tomorrow, but I doubt we want to do that," he wrote.
Records later obtained through the Right to Information Act revealed the generator was one of three critical pieces of equipment which failed, forcing technicians to scramble for a portable one to replace it.
'They should just be communicating with the public'
“I think their desire from the beginning was to treat this as if it was not a significant story and to throw journalists off any scent that this was, something serious had gone wrong,” he said.
“I don't think communications officers should be in the business of deciding what the political impact might be or what people might think of it. Their job is simply to tell us what is going on behind the scenes,” he continued.
“They should just be communicating with the public.”
I don't think communications officers should be in the business of deciding what the political impact might be or what people might think of it.- Michael Camp, associate professor of journalism at St. Thomas University
On June 11 when reports were coming in about continuing disruptions to government services, CBC News had requested an interview with Sue Stultz, who was the minister of Government Services, to talk about why the backup failed, and the integrity of the provincial network.
The internal correspondence on the request remarked:
"They want me, a techie or the minister,” wrote Chouinard.
“Minister really shouldn't touch this, given it is technical. Would it be worth it to put a high-level techie like Christian [Couturier, the chief information officer] out to shut down CBC spec once and for all?"
Sarah Ketcheson, a communications officer, replied: "I'm not in favour. I don't think anything good (or frankly informative and helpful to the public) would come from a discussion with CBC f'ton on this."
The next reply from Chouinard stated, "Getting into the weeds is not going to be helpful for sure, and only Frederictonians are grousing about this — I'll tell them we decline that request, but we will need to be able to address the question of where do we go from here and have a basic explanation of the server issue and its resolution."
Stultz received a heads-up before a response was sent: "CBC called wanting to get you/us on CBC Info Morning Fredericton to talk about the network outage we had on Monday and its aftermath. We are going to decline as we don't have the information (or inclination) to drag this out on air with [redacted], since the issue with the network and system is mostly resolved and the backup issue is being dealt with by our technical experts. ECO [Executive Council Office] and senior management and the IT group here at DGS [Department of Government Services] concur.
"Be prepared though, for [redacted] to read the reply I send to [redacted] producers on air and probably be rather snarky and dismissive, as is his way. This is mostly a Fredericton story and issue since GNB.ca and SNB.ca are running.”
A prepared statement followed. Before sending that to CBC News, however, it was run by Executive Council Office employees who shortened it.
It was shortened, then sent. It stated that the outage had impacted backup power system, which affected the MDC servers.
It detailed some of the recovery timelines for the provincial government's network and public websites and said NBISA was working to ensure nothing similar happened in the future.
In another instance, when asked about the outage on June 10 by a Daily Gleaner journalist, the note preceding a written response stated “He didn’t raise it, but we may face the question of whether critical systems like financial and health were affected/ vulnerable due to the outage and why a backup system didn’t work as expected. One hopes this will be the end of it, but never count our friends in the media out,”
'Please keep this information confidential'
Communications staff weren’t the only ones concerned with leaking details.
Were you affected by the power outages at the data centre? Tell us your story. Email us at [email protected]
In one email, Ron Allison with the Executive Council Office, asks an International IT research company for help with a framework for calculating the cost of the data centre outage and its impacts. He describes the system failures then says:
"I include the above information so that you have an understanding of the gravity of the situation… Please keep this information confidential."
The reply includes a toolkit on calculating the cost of a disaster, and an offer to have an analyst come in and help quantify the cost.
In another email a Bell representative says he or she had been asked to remind colleagues of the "NDAs [non-disclosure agreements] and confidentiality agreements that we have with GNB. If news of this does make it to facebook, twitter, or print, it will not be from us."
'He's looking for smoke'
Ten days after the outage, when a court decision was delayed in the case of two anti-fracking protesters because the judge could not access computer files, CBC reporter Jacques Poitras was given an interview with Ketcheson.
The interview occurred just two days after an internal email detailed plans for file restorations going forward, and while the MDC itself was still running on diesel power and in need of repairs to its infrastructure.
The documents show Ketcheson prepares information and does an interview with Poitras, reiterating the basic facts released to CBC News the week before and stating that some system applications were impacted, but that they were online and “everything was operational."
She sends an email afterward saying her interview with Poitras was pretty straight-forward and he didn't have a lot of follow-up questions.
McInerney replied, "Good work Sarah."
Chouinard replied, "That sounds better than I would have expected. He's looking for smoke, but the reality of the outage doesn't even have smouldering embers…"
There is a public interest in knowing that, for example our public institutions aren't functioning because we have a difficulty with computer servers.- Alison Ménard, Moncton lawyer
“There is a public interest in knowing that, for example our public institutions aren't functioning because we have a difficulty with computer servers,” she said.
“That's why the government needs to be accountable in these kinds of circumstances.”
Camp said the public needs open communication.
“I think that people need to feel confident that the records of the government, especially the records of the justice system, are impregnable. That they’re well maintained and that they are not subject to any kind of corruption or system failure," Camp said.
"Or if there is a system failure, I think it’s important for these people in communications to tell people about it because it makes it worse when you try to understate or be indirect about what the effect of this failure really was.”
Government would not answer CBC News’ questions relevant to what it learned in the documents released under the Right to Information and Privacy Act.
Premier Brian Gallant issued a statement to CBC News on Tuesday reiterating the statement,
"My understanding is that the former government initiated an external review. We followed-up and can confirm that review is on-track. It is our intention to release findings and recommendations in as timely a way as possible," Gallant said.