Network Security Proposal: Tombstone AllNodes-Compromised LUNC Validators That Refuse To Remake Their Validators & Validator Wallets

[RabbiJebediah]

Written by: Rabbi Jebediah (https://twitter.com/RabbiJebediah)
Co-Author: Bilbo Baggins (https://twitter.com/NovaValidator)

SUMMARY
Recently, the LUNC community has been made aware of massive security risks to the entire chain which tie into AllNodes and the hosting service they provide to other validators. Certain validators on the LUNC blockchain have been completely compromised via having their validator wallet seed phrase retained by AllNodes. This creates an open and unacceptable risk vector that endangers the security of the entire chain and the funds of everyone on it! Our proposal seeks to protect the chain (and all its investors) by removing contaminated validators from the active set through unbonding and tombstoning them at the protocol level if they refuse to remake their compromised validators and validator wallets within 1 week of the passing of this proposal.

DEFINITIONS
Unbonding Validator
https://classic-docs.terra.money/docs/develop/module-specifications/spec-staking.html?highlight=unbonding#unbonding-validators

Unbonding of compromised validators will allow their delegators to redistribute their delegations to non-compromised validators, such as the compromised validator’s new validator (that’s been made with a fresh validator wallet)

Tombstone
https://classic-docs.terra.money/docs/learn/glossary.html?highlight=tombstone#tombstone

https://classic-docs.terra.money/docs/develop/module-specifications/spec-slashing.html?highlight=tombstone

Tombstoning of compromised validators will prevent that validator from ever being able to enter the active set again, which will ensure the future security of our chain by ensuring that the compromised validator can never sneak back into the active set.

MOTIVATION
Through the tireless efforts of Jacob Gadikian (CEO of Notional Labs), we’ve learned that AllNodes lied to their customers and held onto validator wallet seed phrases which can be used to take control of “independent” validators that are hosted by Allnodes. For a validator, having the seed phrase of their validator wallet compromised means that AllNodes can take control of their validator and act as them at any time. If enough Voting Power is concentrated under the AllNodes + Compromised Validators umbrella, it could allow the company or a malicious hacker to control over ~34% of the network and execute a brute-force takeover of the entire blockchain! This includes but is not limited to: halting the chain, signing validator transactions and voting, printing false blocks, manipulating Oracle price feeds, etc. In short, it would give them complete and uncontested control over Terra Classic! Whoever controls consensus controls the chain!

And to be clear – AllNodes doesn’t even have to be the ones doing malicious behaviour for this to still be a risk – given that they’ve had access to their customers’ validator wallet seed phrases, that sensitive info is now compromised and could’ve come into the possession of other parties which could seek to harm the chain in the future (hackers, disgruntled employees, random people via leaked e-mails, etc.). Despite AllNodes claiming they’ve erased the critical information on their end, we have no way of knowing whether it was leaked to other parties during the many months this info rested within AllNodes posssession. Even if none of the above were to occur, AllNodes as a company could be sold to a third party in the future, which would give said party a way to exploit Terra Classic if it were so inclined!

All in all, this is a massive, open, unacceptable risk to our chain! Given the value of the LUNC chain (and everyone’s investments), the only sane resolution to this problem is to have the compromised validators to remake their nodes and validator wallets… which is something we’ve tried to encourage, but they’ve refused to accept. Hence, this tombstoning proposal is now the final line of defense for our chain short of delegators redelegating away from these compromised validators.

Here is professor Edward Kim supporting the initiative:

Currently Known Compromised Validators:

• Classy’s Sphere

  • Current Voting Power: 3.83%
  • Compromised Validator Operator Address: terravaloper14xjkj5rv72fgqz3h78l883rw0njwhmzce45006
  • Compromised Validator Wallet: terra14xjkj5rv72fgqz3h78l883rw0njwhmzce6cjlf
  • Twitter: https://twitter.com/ClassyCrypto_
  • Status: To date they have refused to remake their compromised validator and validator wallet.

• HappyCattyCrypto

  • Current Voting Power: 2.52%
  • Compromised Validator Operator Address: terravaloper15ahd0dg9qwkg5tjmkn7fm6sdrpwa47m50l4zrg
  • Compromised Validator Wallet: terra15ahd0dg9qwkg5tjmkn7fm6sdrpwa47m50selnm
  • Twitter: https://twitter.com/HappyCatKripto
  • Status: They have claimed they will look into remaking their validator in 3-4 months, which is too long of a timeline and can easily be forgotten by the community. As far as we are aware, they also have given no public reason for why they are seeking to wait for such a long period of time before remaking their validator.

• CryptoKing Burn And Build

  • Current Voting Power: 1.66%
  • Compromised Validator Operator Address: terravaloper1h7eetq4atvnxsaamx9q5jmhu7jzdkx7f34rkl0
  • Compromised Validator Wallet: terra1h7eetq4atvnxsaamx9q5jmhu7jzdkx7f360t0u
  • Twitter: https://twitter.com/CryptoKing_NFT
  • Status: It is currently unknown whether they have plans to remake their compromised validator or not.

• LUNCLIVE

  • Current Voting Power: 0.42%
  • Compromised Validator Operator Address: terravaloper1c7dhgf6lx6fy829tf30tvawj86u0gprda2eg4m
  • Compromised Validator Wallet: terra1c7dhgf6lx6fy829tf30tvawj86u0gprda9449g
  • Twitter: https://twitter.com/Luncliveorg
  • Status: To date they have refused to remake their compromised validator and validator wallet.

• ToxicLabs DAO

  • Current Voting Power: 0.37%
  • Compromised Validator Operator Address: terravaloper1syxnkjzmvwy4lm8saq4qf5354we38p6m8hfzvm
  • Compromised Validator Wallet: terra1syxnkjzmvwy4lm8saq4qf5354we38p6m8c9lug
  • Twitter: https://twitter.com/ToxicLabsDAO
  • Status: They are remaking their compromised validator and validator wallet and are EXEMPT from this proposal.

• The Millenial Market

  • Current Voting Power: 0.35%
  • Compromised Validator Operator Address: terravaloper1z056yhed5xr9yfc9vnpl23hmy97rqqhvfxuc8k
  • Compromised Validator Wallet: terra1z056yhed5xr9yfc9vnpl23hmy97rqqhvffs9h9
  • Twitter: https://twitter.com/TMMcryptos
  • Status: It is currently unknown whether they have plans to remake their compromised validator or not.

• lunc_nymph

  • Current Voting Power: 0.29%
  • Compromised Validator Operator Address: terravaloper1qrgwphlf9c64m5ys6pzsvy055ud03e4kr7sq8u
  • Compromised Validator Wallet: terra1qrgwphlf9c64m5ys6pzsvy055ud03e4kr3uah0
  • Twitter: https://twitter.com/lunc_nymph
  • Status: Initially committed to remaking their validator when talking to Bilbo in DMs, but their current status is now unknown and requires further confirmation.

• BetterLunc

  • Current Voting Power: 0.02%
  • Compromised Validator Operator Address: terravaloper1hlwfelx6s05a43tszudj4w02zz500xhupsrnxm
  • Compromised Validator Wallet: terra1hlwfelx6s05a43tszudj4w02zz500xhupl0wkg
  • Twitter: https://twitter.com/BetterLunc
  • Status: It is currently unknown whether they have plans to remake their compromised validator or not.

• David Goebelt

  • Current Voting Power: 0.02%
  • Compromised Validator Operator Address: terravaloper1pe3cvzlx6yqrd666qngpmfsllhemk9yqyyn58m
  • Compromised Validator Wallet: terra1pe3cvzlx6yqrd666qngpmfsllhemk9yqytlfhg
  • Twitter: https://twitter.com/davidagoebelt
  • Status: They are remaking their compromised validator and validator wallet and are EXEMPT from this proposal.

• LUNC808

  • Current Voting Power: 0.01%
  • Compromised Validator Operator Address: terravaloper1rr53sjy3dmn7n4xeh4gu8nvrupd3n3wsa0n600
  • Compromised Validator Wallet: terra1rr53sjy3dmn7n4xeh4gu8nvrupd3n3wsaql8lu
  • Twitter: https://twitter.com/LUNC808
  • Status: They are remaking their compromised validator and validator wallet and are EXEMPT from this proposal.

In total, the current total VP of known compromised validators is 9.49%. Of that, only three compromised validators (0.40% of known total compromised VP) so far have publicly demonstrated willingness to remake their compromised validators and validator wallets (ToxicLabs, David Goebelt, and LUNC808). If there are any compromised validators out there that are now remaking their validators that we haven’t been made aware of yet, please reach out and let us known and we will update this proposal accordingly.

ALLNODES CLIENT LIST
While the list above is comprehensive, we have no way to know for sure that it makes up the entirety of compromised validators on the chain, and with the large amount of noise around this crisis it’s incredibly likely there are compromised validators on the chain right now who are choosing to stay quiet and hoping to lay low until the situation resolves. We would ask Allnodes to provide the community with a detailed list of their current and historical customers that were validating on the Terra Classic chain in a manner that can be verified as being accurate and factual, so we can also encourage those validators to remake their compromised validators and validator wallets as well.

DENIED OUTREACH
Multiple efforts have been made to reach out to compromised validators and try and help them to remake their compromised validator and validator wallets. Unfortunately, most compromised validators who were contacted refused help and chose to try and turn a blind eye to the situation (no doubt spurred on by Tobias’ comments downplaying the magnitude of the situation as well), and seemed to put their profit and voting power they’ve accumulated before the safety of the chain and their delegators. Their seemingly continued avoidance of this issue has left us with no choice but to make this proposal.

The ongoing existance of these compromised validators who are refusing to remake their compromised validators and validator wallets represents an unacceptable exploitable risk vector for the entire chain, and as such they must either remake their compromised validators and validator wallets, or have their validators unbonded and tombstoned via changes in the code in order to protect the future security of the chain and investors funds on it!

Below are a few examples with how much disregard some compromised validator-owners treated this ongoing risk to the chain…

Classy Crypto

HappyCattyCrypto

LUNCLIVE

It is for all these reasons that we propose the following changes…

PROPOSAL
We propose that all compromised validators remake their validators and validator wallets. If they refuse to do so within 1 week of this proposal passing, we will then seek to tombstone their validators and remove them from the active set.

ADDENDUM/REMINDER
To any validators that have claimed they’ll be remaking their nodes within 3 months or so, this is simply an unacceptable timeline. If this proposal passes the compromised validators will have 1 week to remake their validators and validator wallets! If they do not comply within that time-frame then the tombstoning will go into effect. Given the enormous risk to the chain’s safety, we feel 1 week is more than generous enough to allow the transition from contaminated to fresh nodes.

Thank you for reading! And many thanks to Jacob Gadikian for spearheading this initiative and championing the chain’s safety! The LUNC community would not have known about this issue were it not for his tireless efforts and continual warnings.

Signed by: Rabbi Jebediah, Bilbo Baggins
Thanks to: Jacob Gadikian (Notional Labs)

[Ryan_Spunt]

as someone who was shunned by the community after questioning them on this i am glad you have the hutzvah to speak up

[Leroy]

This will be a painful but necessary step in the right direction for the chain. I appreciate the frustration that may be felt by the validators affected by this, but with Terra’s history we cannot build our house on sand again!

It’s a ‘yes’ from me.

Shalom,

Leroy

[Grizzly]

Classy and all these compromised influencers needs to be humbled down. They tried (and failed) to gaslight their community. @ClassyCrypto What are your arguments now that Professor Ed himself acknowledges the existence of a security issue?

[Exoneratelarouche]

I’ll be voting YES WITH ENTHUSIASM. We cant let MALICIOUS ENTITIES control a BILLION DOLLAR BLOCKCHAIN even if some of the BONEHEADS in this community refuse to acknowledge the problem.

Queue the brainless “NO WITH VETO” comments any second now…

[Gut_Daddy]

Hey-o finally someone’s doing something about all the idiots that don’t want to secure the chain but want to use it to peddle other scam coins to us. Gut_Daddy fully supports this. There must be consequences for not securing the chain and playing with everyone’s money here.
Gut_Daddy say YES!

[Rummy]

Too far. Just a NO with a VETO.

[Exoneratelarouche]

Geta brain moran

[Wraith]

How do you justify anything you’re saying here when Ed himself has said these validators are a security risk and they refuse to not only acknowledge the risk because they obviously don’t want to lose their voting power but put everyone else at risk? No one is exiling them? They would have to remake their validators that aren’t tied to Allnodes isn’t that right so how is this “centralized tyranny”? I don’t want my investment to get vanish into thin air because people refuse to secure the blockchain. Not sure why you seem to want something like that?

[BasedTorontonian]

Great proposal. The LUNC community should not be compromised because of laziness of a few Validators. Yes from me.

[StrathCole]

@RabbiJebediah

We propose that all compromised validators remake their validators and validator wallets. If they refuse to do so within 1 week of this proposal passing, we will then seek to tombstone their validators and remove them from the active set.

You know that this definition would not only affect some Allnodes validators, but also e.g. LUNC DAO and others that partnered with or bought nodes from others before new validators were allowed, right?

Also this proposal is highly unbalanced overall as it disadvantages all those that publicly stated to use non-self-managed (or non-bare-metal) nodes vs. all those quiet.

However it is a strong no from me for many reasons, including completely overreacting.

[RabbiJebediah]

@StrathCole

You know that this definition would not only affect some Allnodes validators, but also e.g. LUNC DAO and others that partnered with or bought nodes from others before new validators were allowed, right?

Yes, and? The scope of this proposal covers only the listed validators in the OP; LUNCDAO and others similar to him who have compromised nodes via buying keys may be the subject of another similar prop in the future. Regardless, any compromised node is a security risk to the chain.

@StrathCole

Also this proposal is highly unbalanced overall as it disadvantages all those that publicly stated to use non-self-managed (or non-bare-metal) nodes vs. all those quiet.

True. Which is why we’re asking AllNodes to release their client list (as much was stated in the prop itself - I’m getting the feeling you only skimmed it, instead of reading it thoroughly). Also, this is only the first such proposal; if/when information about other compromised validators is revealed, steps will be taken to encourage them to remake their nodes and validator wallets.

@StrathCole

However it is a strong no from me for many reasons, including completely overreacting.

LUNC is a $1B+ asset, if anything this prop is a severe underreaction, given the gravity of the situation and the ongoing, open-ended security risk to the entire network. Also, you’re partnered with HCC – who runs his server on AllNodes infra, and is one of the stated validators marked for tombstoning – so you’ll excuse me if I don’t consider your opinion as objective, since you have a personal incentive to see this proposal fail.

I do appreciate the feedback, though.

Shalom!

[StrathCole]

@RabbiJebediah

Yes, and? The scope of this proposal covers only the listed validators in the OP; LUNCDAO and others similar to him who have compromised nodes via buying keys may be the subject of another similar prop in the future. Regardless, any compromised node is a security risk to the chain.

True. Which is why we’re asking AllNodes to release their client list (as much was stated in the prop itself - I’m getting the feeling you only skimmed it, instead of reading it thoroughly). Also, this is only the first such proposal; if/when information about other compromised validators is revealed, steps will be taken to encourage them to remake their nodes and validator wallets.

You should know that no company in their right mind will simply release a client list. Of course I read the proposal.
The whole discussion about the compromised seed phrases is completely besides the point. It does not really impose significantly more risk to the chain than the physical/remote access to the server itself.
It increases risk to the validator itself due to the ability of impersonation, but e.g. bring down the validator is possible without the seed.

@RabbiJebediah

LUNC is a $1B+ asset, if anything this prop is a severe underreaction, given the gravity of the situation and the ongoing, open-ended security risk to the entire network. Also, you’re partnered with HCC – who runs his server on AllNodes infra, and is one of the stated validators marked for tombstoning – so you’ll excuse me if I don’t consider your opinion as objective, since you have a personal incentive to see this proposal fail.

Just to make clear as you brought it up, the node is HCC. And I can well separate personal things from professional things. But that said, if I was to bring up my own validator, I would most likely choose a service like Allnodes, just because although I am able to do all those things on my own in terms of experience and knowledge, it doesn’t mean that it is always better to do it. While I would do it without them knowing seeds (as they only offer that service now anyway), as I said above it doesn’t really make much difference in terms of chain security. As long as they are not above 33% VP combined (which would still need to be proven they are), it is more secure and reliable than having 80% of validators hosting in their basement with unreliable network or power source. And if they would be above 33% VP combined, you really wanna say not having a single key would make it significantly safer for the chain?
And if you answer this with “no”, is the next proposal then to ban all VaaS from the chain completely?

[Bilbo]

Compromised validators just need to remake their validators and validator wallets. Not a big deal.

@StrathCole

You should know that no company in their right mind will simply release a client list.

No company in their right mind would also hold onto the seed phrases of their customers because of the security risk to the chain that creates, but alas, here we are. The simple fact is Allnodes’ chosen business practices has created an unacceptable vector of risk and harm to the community. We didn’t do this. They did. It’s on them to make it right and to help provide us with the tools to secure the chain and clean up the mess their business practices made. Giving a client list is the smallest of asks considering the situation.

@StrathCole

The whole discussion about the compromised seed phrases is completely besides the point. It does not really impose significantly more risk to the chain than the physical/remote access to the server itself.
It increases risk to the validator itself due to the ability of impersonation, but e.g. bring down the validator is possible without the seed.

Both are issues. One can be solved by compromised validators remaking their validators and validator wallets. The other cannot. Secure the compromised validator and validator wallets first, then we worry about the impact of VaaS on the ecosystem. One step at a time.

@StrathCole

As long as they are not above 33% VP combined (which would still need to be proven they are), it is more secure and reliable than having 80% of validators hosting in their basement with unreliable network or power source.

It is easier to prevent a fire then it is to put a fire out. Waiting until we’re over 33% VP to try and solve things isn’t wise by any measure. If we wait till we’re over 33% VP to solve the issue it’ll be too late by then. We’re already uncomfortably close to 33% VP, you can do the math yourself. We’ve already identified another compromised validator with 0.57% VP that is hosted on Allnodes that isn’t currently included in this proposal, but will be.

@StrathCole

it is more secure and reliable than having 80% of validators hosting in their basement with unreliable network or power source.

Who said anything about hosting their validator in a basement? We certainly didn’t. Host it on another cloud provider if you want. This “hosting in a basement isn’t secure” argument that compromised validators (like the one you’re managing for HCC) have been using is just a complete and utter strawman, because we aren’t pushing for compromised validators to host their validator in their basement. We’re pushing for them to remake their compromised validator and validator wallets.

On top of that, the “basements aren’t secure” argument is also misleading as an argument because it ignores the fact that if a validator does choose to host in their basement (as is their right) and their validator goes down due to unreliable network or power source, the blast radius is only contained to that one validator instead of the large amount of validators that are currently hosted by Allnodes.

@StrathCole

is the next proposal then to ban all VaaS from the chain completely?

I don’t think we’ll the ones who decide that. That’s a bigger question the entire Cosmos ecosystem is going to have to ask and come up with a solution for. As I said before, for now we can really only focus on mitigating the risk that has been exposed to our chain as a result of Allnodes’ business practices by encouraging compromised validators to remake their validators and validator wallets. Looking at solutions for how to properly handle VaaS as a concept can come after. One step at a time.

[RabbiJebediah]

UPDATE/ADDEDNUM: February 8th, 2023

We’ve added another AllNodes-comrpomised validator to the master list…

Note that we cannot edit the OP and update the list after submitting the thread (Agora doesn’t allow it), so short of remaking and reposting this entire thread we’re forced to include addenda like this one when new information comes to light. Regardless, any compromised validators included in updates such as this one are to be treated no differently than those outlined in the OP - if this proposal passes, they will either have to remake their validators + validator wallets, or face tombstoning.

The most recent addition to the list:

• JESUSisLORD
  • Current Voting Power: 0.61%
  • Compromised Validator Operator Address: terravaloper16e0s5t7q69elnlchrupryw3h7vu8zk23pe5wh8
  • Compromised Validator Wallet: terra16e0s5t7q69elnlchrupryw3h7vu8zk23pkcn85
  • Twitter: https://twitter.com/ForTheCross_CH
  • Status: It is currently unknown whether they have plans to remake their compromised validator or not.

Due to the urgency of the overall situation and open security risk posed to the chain, this thread will remain on the Agora only for 7 days before we send the proposal for voting on the Station. If anyone reading this has additional information pertaining to compromised validators then please let us know via DMs either here on Agora, or Twitter (handles are at the top of the OP).

Shalom!

[Grizzly]

.It appears that some inbreed animals have flagged certain arguments that they disagree with. However, it’s important to keep in mind that certain node providers, validators, and influencers are compromised. You cannot deny facts. I’m not letting your BS pass all nodes fanbois!