Backup software vendor Veeam deleted forum data after restoration SNAFU
DevOps team did the dirty on a database
Data management software vendor Veeam has admitted to an embarrassing oopsie: messing up a restoration job and erasing data.
The good news is this wasn’t a mission-critical mistake, or the result of problems with the company’s products.
The company’s error was disclosed on its forums in a February 11 thread in which a product management chap reported “we have noticed some topics and comments from the past 24 hours are currently missing.”
As is often the case with vendor forums, Veeam’s mix comments from staffers and customers and the result is a heady brew of product support info, commentary, and company news. It’s therefore possible that useful support info or posts about product roadmaps could have gone missing.
The Veeam rep who posted news of the missing posts therefore added: “We understand how important timely updates are, and we want to assure you that we are actively looking into this matter.”
Once they did so, they discovered the following root cause:
During this morning's forum maintenance window, a backup of the forum database from one day ago (Monday, February 10th) was mistakenly restored by internal infrastructure DevOps team over the production database, which did not experience any issues and did not require restoration.
As a result of significant changes (new topics and comments) made since the time of the restore, we are unable to integrate the topics and comments from yesterday and todays [sic] morning into the current version.
Oops.
“We sincerely apologize for any inconvenience this may have caused and appreciate your understanding,” the Veeam staffer wrote.
- I was told to make backups, not test them. Why does that make you look so worried?
- You had a year to patch this Veeam flaw – and now it's going to hurt some more
- Memories fade. Archives burn. All signal eventually becomes noise
- Veeam tests support for another VMware alternative: XCP-NG
This looks like the kind of honest mistake that IT shops everywhere shouldn’t make but do. The Register is sympathizes (see what we did there?).
We learned of the error because Veeam listed it as “Best Post Of The Week” in its weekly forums digest, which we subscribe to as the company uses it to post news such as its plans to support VMware alternatives Proxmox and XCP-Ng.
The forum and email were a great read for years because Veeam’s senior veep of product management Anton Gostev would use it to comment on the company’s activities, as well as goings-on in the wider data management industry. Sadly, Gostev put down his pen in 2022 as he is Ukrainian and felt unable to continue writing his articles amid Russia’s illegal invasion of his homeland. ®