Preface by Silke Brand | When terms and services are not taken seriously, spooky things can happen. We just had survived Halloween when Hetzner, a large Germany data center took more steps forward to fight against the misuse of their services. They just went ahead and shut down all accounts that provided validation nodes for the proof-of-stake (PoS) algorithm of the Solana blockchain.Was this an unexpected ghostly move?
Well, not really – at least not for informed crypto insiders as Sander Lutz from Decrypto explains below.
Hetzner Online GmbH, a German cloud service provider, moved to block all Solana activity on its servers on November 2nd 2022, instantly taking over 1,000 Solana validators offline. While the affected nodes are largely back online, the action triggered a broader discussion in the crypto community about the increasing influence of centralized service providers in controlling the fate of supposedly-decentralized blockchain networks.
Approximately 40% of the Solana network’s validators—and 20% of the network’s total stake—were housed on Hetzner as recently as August 2022, according to independent analysis by Google Cloud engineer Sam Padilla on Twitter.
Is it dangerous that Solana depends heavily on individual service providers?
While the event did not crash Solana—that would have required taking 39% of the network’s total stake offline, halting its consensus mechanism and cutting off all network activity—it did highlight the increasing dependence of networks like Solana on individual service providers like Hetzner.
In the hours following Wednesday’s ban, Solana validators—who stake large quantities of SOL with the network to earn more SOL, and keep Solana’s proof of stake consensus mechanism operating by processing blocks of the network’s transactions—raced to find new service providers and get their stake back online.
A representative of the Solana Foundation told Decrypt that Hetzner’s ban would not meaningfully impact Solana, as validators have many alternative options for service providers.
Was the deactivation of the nodes very much unexpected?
“This decision by Hetzner was certainly unfortunate, but not entirely unexpected,” the representative said. “There are hundreds of data centers that Solana operates out of, so those impacted are migrating to new locations.”
Operators either managed to move or the stake was deactivated,” a separate source told Decrypt on Thursday.
The representative also pointed to the Solana Foundation’s Server Program initiative, which pairs validators with vetted and approved independent service providers in an effort to further decentralize the Solana network.
Despite such efforts, though, a majority of Solana validators still depended on centralized companies like Hetzner, Equinix, and AWS for their hosting needs.
“It was super easy to get set up on Hetzner, and as a result, many low-quality validators just defaulted to using Hetzner as it was quick and simple for them, even if it wasn’t the best choice,” the anonymous founder of Solana staking platform SolBlaze told Decrypt.
Hetzner, on its part, has long signaled its hostility towards crypto. In August 2022, the company began banning Ethereum nodes operating on its servers, and told their audience about the ban as bluntly as possible. Maggie Love writes on August 25th 2022 on Twitter that she would like to see a discussion about Ethereum not being decentralized if the stack is not decentralized. She outlines that 69% of the nodes on the Ethereum Mainnet are hosted and that over 50% of them coming from Amazon Web Services (AWS), over 15% from Hetzner and 4.1% from OVH. So Hetzner fact it hosted roughly 10% of all Ethereum nodes around that date.
Katie, a representative of Hetzner Online replies to Maggie Love that she recommends to read their statement on Reddit from August 22nd 2022, which quotes as follows:
Using our products for any application related to [crypto] mining, even remotely related, is not permitted. This includes Ethereum. It includes proof of stake and proof of work and related applications. It includes trading. It is true for all of our products, except colocation. Even if you just run one node, we consider it a violation of our ToS.
Hetzner_OL on Reddit
And the wary user of Hetzner services could read the warning in their Terms and Conditions in paragraph 8.3:
The operation of applications for mining cryptocurrencies remains prohibited. These include, but are not limited to, mining, farming and plotting of cryptocurrencies. We are entitled to lock the Customer’s access to their Hetzner services or account in the event of non-compliance.
Hetzner Terms and Conditions, 8.3 (current version 7 November 2022)
So this would have applied to Solana proof of stake validating. And yet, a substantial percentage of Solana validators continued to utilize Hetzner servers. “This [action] is not a surprise at all. Hetzner has been advising about censoring crypto nodes for a very long time and this finally happened,” Guilherme Soubihe, CEO of internet service provider latitude.sh, told Decrypt.
Why the heck hates Hetzner cryptocurrencies on its servers?
“It seems Hetzner doesn’t want Solana servers on its infrastructure because [validating] is too intensive for its server disks, which tend to fail after a few months of use,” said Soubihe. “There are bare metal providers, like us, that are crypto friendly and would never censor crypto.” So-called bare metal service providers rent entire physical servers to customers, unlike cloud servers, which host multiple tenants.
In the aftermath of Wednesday’s ban, validators of other blockchain networks that rely heavily on Hetzner, including Cardano and Cosmos, began searching for new service providers, in anticipation of similar bans on their nodes that could seriously impact network functionality.
The move by Hetzner comes during a period when users are re-evaluating the dependence of blockchain networks – particularly proof of stake networks, which depend on ample server space – to function on centralized service providers.
When Ethereum merged to a proof of stake mechanism in September, the network faced criticism over the fact that centralized entities now control over 60% of all ETH staked on the network.
The events on similarly brought about another wave of criticism directed at networks like Solana and Cardano, and their dependence on centralized service providers as Holochain Explained states on Twitter on November 3rd 2022.
#AWS, #hetzner & #DigitalOcean all host & control just over 51% (ironic number) of all #Cardano #ADA $ADA
— Holochain Explained (@holo_of) November 3, 2022
validator staking nodes.
No Finney attacks in PoS but validator downtime is possible if they do the same as they did to Solana. pic.twitter.com/qKy1hkJLU1
