Banana will write something about this better explaining what kind of data we lost and our process of fixing things today.
Most importantly, I want to thank everyone who assisted in getting the server back into a working state today. I also want to thank our players for their understanding and patience over the past 12 hours. Thanks to a combined effort,
no player builds or data was lost. Everything players have built on the server, their balances, and all of their related data is safe and sound. Ultimately, the only damage was to server buildings such as Fedguardia spawn or Pern spawn. Staff have been working tirelessly to repair the remaining damage or missing features.
What happened?As Migas explained in the initial post, our server software Paper was updated to the latest version at the time. This build contained a severe bug. Every chunk loaded with this build is irreversibly corrupted. I cannot emphasize enough that this is an extraordinary occurrence, and we were not the only server affected by this issue.
Sentinel was fortunate enough that no players were online when this version was loaded, so the damage was only to the "spawn chunks" (chunks that are always loaded around the center of each world). We are also fortunate that none of our "spawn chunks" contain player builds. Plugin data, player data, and chunks that were NOT loaded during this time were not affected.
How did we fix it?First and foremost, we avoided the worst case scenario because the update was performed without any players online. We then tried to ascertain the damage, and to what extent the chunks had been corrupted. After testing separately, we confirmed that only the "spawn chunks" were corrupted.
Initially, the server failed to start but we managed to get it successfully up and running with full functionality. We repaired as much of the Fedgardia spawn island as we could, copying from a creative copy of the island. From there, the server was opened again, and staff set about repairing miscellaneous things (NPC's, shop items, treasure hunt, etc).
At the time of this writing, the server is very nearly back to where it was before all this happened. The remaining issues are all fixable, and will be fixed in the coming days.
What happens now?Our first priority is completing the remaining tasks to repair the damage. I and many other staff spent the entire day doing so, and I'm sure we will be continuing to do so. We also will be making a proper copy of the spawn island once it is fully repaired, should the need arise for it.
Lastly, we want to make sure that something like this never, ever happens again. We were fortunate that the impact was small enough to be repairable this time. Moving forward, we as staff will be discussing and implementing new systems to make sure the server is running safely and smoothly.
We ask that players be patient with staff, as we are working overtime at the moment. Thank you <3
