Wither Skull Incident

"Oh Oh..........lol...help...danger...problems...no no no no no...not good"

— Docm77's first tweet about the incident

Wither Skull Incident
A graph from Server Monitoring Tool showing the number of entities on the server over time from the day of the first incident.
Season	Season 9
Location	The Perimeter
Date	October 20, 2023 (first incident) Between November 4 and November 11, 2023 (second incident)
Participants	Both incidents: Docm77 , Maescool Only second incident: Rendog

The Wither Skull Incident was an accident that occurred in Season 9 and led to the crash of whole Hermitcraft server. This happened while Docm77 was building pixel art on one of The Perimeter walls. He was working on it near his Overreactor for a long time, close enough for it to continuously produce Wither Skull projectiles, but too far away for them to be destroyed. This resulted in the accumulation of thousands of entities and ultimately caused a server crash. Docm77 managed to fix it by reducing the render distance and removing all skulls via commands.

Shortly thereafter, the Incident occurred a second time, caused by Rendog, while building the Scab Castle in Scabland. The situation was again quickly fixed by Docm77.

Event summary

Background

Overreactor

The Overreactor.

During Season 9, Docm77 built The Perimeter as his main project to place his farms there. One of them was an Overreactor (also known as Wither Reactor), located in the southwest corner of Perimeter, in which he trapped several Withers to constantly produce Wither Skull projectiles whenever the contraption is loaded. Docm77 used it to produce wither roses and make hardened concrete (or adaptable to his other needs), but when the farm is turned off, all projectiles were destroyed in special structures (Wither Skull Destroyers) at the end of their trajectory, so it doesn't cause any damage.

Due to the high risk involved in spawning multiple Withers, Docm77 was given the role of server operator to be able to control the situation in case of mobs escaping from the farm.

Walls of The Perimeter

Another part of Perimeter's development was decorating the walls. It took some time for Docm77 to start this project and finally decided that the walls should be covered with selected pixel art created by his viewers. Building each was a time-consuming initiative. He created the first pixel art on the north wall and when he finished it he decided to build another one in the middle of the south wall, near the Overreactor.

The Incident

At some point while building the second pixel art, Docm77 came close enough to the reactor that the Withers were in loaded chunks, which allowed them to produce new Wither Skull projectiles. However, he was too far away for these projectiles to be destroyed and instead they got stuck in the lazy chunks. Due to the complicated process of building pixel art, Docm77 spent an hours working in one place, which allowed more and more Wither Skulls to accumulate.

In the meantime, Docm77 was moving around while working, causing changes in the loaded chunks. This probably resulted in some of the projectiles being able to move in groups all the way to the Wither Skull Destroyers, causing multiplied explosions, not strong enough to damage the Wither Skull Destroyers, but enough to leave few holes in outer wall of the Overreactor.

Unfinished pixel art.

Over time, however, the amount of Wither Skulls increased to the point where Docm77 began to experience a lot of lag while working and wasn't able to fly with Elytra. When the number of loaded entities reached almost 25,000 the server crashed.

Resolving of the crisis

After the crash, Docm77 contacted Maescool, who verified that there were over 18,000 loaded Wither Skulls on the server. This explained where the cause of the problem was.

To solve this crisis Docm77 returned to the server with a reduced render distance to about 6 chunks. In this way, he could slowly approach the Overreactor and, using the command (he had access to it because he still had the role of operator), gradually remove the excess Wither Skulls until the number of entities normalized.

Aftermath

Once the situation was under control, Docm77 could return to building the pixel art. This time he controlled the number of loaded entities, so he managed to finish without similar problems. However, he no longer used commands, and instead simply approached the Overreactor from time to time to fully load it, so the Wither Skulls were able to reach the destroyers.

The whole situation forced him to consider introducing safety measures to avoid another Incident. He was particularly concerned about Scabland, under construction at the time, due to its location near the Overreactor.

Second Incident

About a week or two later the incident occurred again. Rendog had been building his Scab Castle at the Scabland for a long time and it turned out that the construction place was another location causing the Overreactor to be partially loaded. Docm77 was at the time AFK at his moss farm, when he was alerted about another accumulation of Wither Skulls. He took care of loading and checking the reactor, which saved the situation.

According to Docm77, there were up to 38,000 loaded entities that time. This amount of Wither Skull projectiles penetrated outside of the Overreactor and drilled a long tunnel in the Perimeter wall, but didn't cause any important damage.

Details and explanations

Wither Skulls amount

It is unclear exactly how many Wither Skulls accumulated during the incidents. Various values were mentioned, but it is not known whether they referred to the sum of all Wither Skulls or the number of skulls at a specific moment.

Known amounts during the first incident:

1. 14,636 - the only precise number of Wither Skulls indicated in one of the graphs from the Server Monitoring Tool. However, other graphs indicated that there were many more entities, so it most likely only indicated the number at a specific moment.
2. 14,000 or 15,000 - the largest number of skulls Docm77 has removed at once.
3. 18,000 (likely a rounded value) - according to Docm77, Maescool pointed out this amount of Wither Skulls after checking what caused the server crash. He also pointed out this number in one of his posts.
4. 20,000 (likely a rounded value) - Docm77 mentioned this value as the total number of Wither Skulls he had removed.
5. 22,000 (likely a rounded value) - Maescool pointed out this amount of skulls in his post about the Incident. This may refer to the largest number of Wither Skulls, as one of the graphs showed that the number of entities during the incident reached almost 25,000 (total number of loaded Wither Skulls and other entities) while usually the number of entities was around 2,000.
6. 25,000 (likely a rounded value) - according to Docm77, Maescool pointed out this amount of Wither Skulls as "peak". However, he wasn't sure of this value, and the graphs also show that this amount wasn't achieved.

Known amounts during the second incident:

1. 10,000 (likely a rounded value) - mentioned by Docm77 as the amount of Wither Skulls, that he loaded, so it could be destroched in Wither Skull Destroyers.
2. 38,000 (likely a rounded value) - mentioned by Docm77 as the full reached number.

Command

"/kill @e[type=wither_skull,distance=..500 ]"

— Command used by Docm77 to remove excess Wither Skulls.

The function of the command Docm77 used was to delete Wither Skull projectiles within a 500-block radius from the player. Each time it was used, he received a response stating the number of entities removed or that no entities were detected. Docm77 had access to this feature because he still had the operator role on the server.

This command made it possible to solve the problem that occurred during the forst incident. Docm77 also used it to check what areas were causing the problem (staying in which location leads to the accumulation of Wither Skulls), monitor the situation later and demonstrate what happened on his episode.

The command range chosen ensured that it would only affect Wither Skulls in the Perimeter and would not remove those in the Hermitsonian Museum of History and Culture or Scarland.

Graphs explanation

Along with explanations of the first incident, there were also published two screenshots from the Server Monitoring Tool, used by Docm77 and Maescool to check details of the situation. These screenshots include graphs showing the number of entities on the server at a specific time period.

The hours are visible on the graphs, but the time zone is unknown. This could likely be CET, which would match Docm77's first post about the Incident posted at 3:26 pm CET (2:26 pm GMT), it would be about an hour after the event.

Original screenshots

First screenshot posted on X (Twitter) by Docm77 and later also by Maescool.

Second screenshot showed by Docm77 in his episode.

Graphs	Explanation
Part of second screenshot (modified).	Two line graphs showing changes in the number of loaded entities on the server in specific time periods. The first one shows almost 24 hours starting at 16:00 the previous day. The second focuses on a much narrower time frame, during the Incident itself, from c. 11:50 to c. 14:50. Together, both graphs allow to read some of the course of events chronologically. 16:00 - c. 11:00: regular number of entities on the server, ranging from 0 to about 5,000, but mostly staying between 2,000 and 3,000. c. 11:00 - c. 15:00: the time period in which the Incident occurred. c. 11:00 - c. 11:50: initial, rapid increase of entities to around 10,000, due to the accumulation of Wither Skulls. At that time, Docm77 was definitely in a place that partially loaded the Overreactor and did not move much from that position. c. 11:50 - 13:20: much slower increase, interrupted by very sudden declines, with a quick return to the previous level, that happened 37 times. This probably happened because Docm77 was moving around more and it led to changes in the loaded chunks that contained Wither Skulls. Usually the value dropped by around 2,000 - 03,000 entities. However, at one point the value dropped by about 10,000 and when it suddenly increased baca, there were about 1,000 fewer entities. Probably the Overreactor was completely out of render distance, and when it was loaded again, some of the Wither Skulls were destroyed. 13:20 - c. 14:40: another faster increase to a critical value of almost 25,000. c. 14:40: sudden drop in value to 0, most likely the moment of server crash. c. 14:40 - c. 14:50: three, sudden increases in value to almost 25,000 with an immediate drop to 0 after a few minutes. Probably attempts to return to the server. c. 14:50: fourth, sudden increase, but this time only to around 15,000, a gradual decline as Docm77 was removing the Wither Skulls and a quick descent to a regular level (probably the moment he mentioned when he removed between 14,000 and 15,000 Wither Skulls at once). c. 15:00 - 16:00: the number of entities back at a regular level.
Part of first screenshot (modified).
Part of first screenshot.	A bar graph showing the number of specific entities loaded on the server at an unknown time during the incident. Only the bottom part of the list is visible, where Wither Skulls are mentioned with amount of 14,636.

Trivia

• There were probably no other Hermits on the server when the first incident occurred (partly due to Twitchcon). Docm77 joked that he cannot be left alone on Hermitcraft because of this. He also laughed about the fact that what broke the server was the building of pixel art.
• The incident was first mentioned by Docm77 in his posts on X (Twitter) from October 20, 2023. At first (2:26 pm GMT) he vaguely suggested that something bad had happened and attached a screenshot of the graphs from the Server Monitoring Tool. Shortly afterwards he mentioned that "server died" and briefly explained how it happened. A few hours later (10:40 pm GMT) he announced that the crisis had been resolved.
• Because of the first incident Docm77 titled his episode I BROKE the SERVER!. However, the event itself was not shown on video because it was not recorded. Docm77 only narrated the course of events.
• Although there were many more Wither Skulls during the second incident than the first, it did't lead to a server crash.
• Much less has been explained about the second incident. Everything that is known was shortly narrated by Docm77 in his episode. Rendog didn't even mention it, despite making a timelapse of Scab Castle being built.