Terra Rebels Infrastructure Operations and Maintenance

Terra Rebels is pleased to have recently announced the successful deployment of our Rebel Station Infrastructure Cluster into full production, fully completing the work that was outlined in our previous spend proposal. In addition to the work outlined in that spend proposal we have also included the deployment of the Rebel Finder and Rebel Agora services.

The purpose of this spend proposal is to ensure that the infrastructure can now be operated, maintained and supported in a sustainable manner. This spend proposal covers the following items:

  • Ongoing running costs (hosting, power, and internet) for the cluster
  • Technical support and maintenance of the cluster hardware and software

The cluster technical support team consists of the following people who will operate in a support rotation to ensure 24/7 coverage of support and maintenance for the cluster:

  • Echel0n (team lead)
  • BeeJee
  • Raider70

The cluster technical support team monthly duties will be as follows but not limited to:

  • Maintenance of our Kubernetes GitOps repo
  • Monitoring of our Kubernetes cluster resources
  • Monitoring and maintenance of Terra Classic LCD nodes and resource usage
  • Monitoring and maintenance of Terra Classic FCD nodes and resource usage
  • Monitoring and maintenance of Terra Classic RPC nodes and resource usage
  • Monitoring and maintenance of Terra Classic API nodes and resource usage
  • Monitoring and maintenance of Terra Classic seed nodes and resource usage
  • Monitoring and maintenance of server hardware
  • Monitoring and maintenance of firewall hardware
  • Monitoring and maintenance of network hardware
  • Monitoring and maintenance of Rebel Station
  • Monitoring and maintenance of Rebel Station Extension
  • Monitoring and maintenance of Rebel Finder
  • Monitoring of Kubernetes cluster backups
  • Monitoring of S3 cluster storage
  • Monitoring of DNS services
  • Monitoring of Prometheus alerts
  • Monitoring of email alerts
  • Backup maintenance of FCD PostgreSQL database and configs
  • Backup maintenance of API PostgreSQL database and configs
  • Backup maintenance of LCD configs and data
  • Backup maintenance of RPC configs and data
  • Backup maintenance of S3 storage

The cost of 3 months running costs and ongoing support is USD 37,593 (to be paid in LUNC from the community pool at the prevailing exchange rate at the time of the submission of the spend proposal). The costs are broken down as follows:

  • Running Costs: USD 7,593 (USD 2,531 per month)
  • Wages: USD 30,000 (USD 10,000 per month)

The cluster support team monthly wages of USD 10,000 are broken down as follows:

  • Echel0n: USD 4,000 ($16.60 per hour, 240 hours per month, 30 days a month)
  • BeeJee: USD 3,000 ($12.50 per hour, 240 hours per month, 30 days a month)
  • Raider70: USD 3,000 ($12.50 per hour, 240 hours per month, 30 days a month)

We will submit similar future running cost spend proposals (prices may be adjusted) every three months.

We urge the community to vote for this proposal to ensure the ongoing reliable operation of the TR infrastructure, and the Rebel Station wallet (and associated applications).

Signed by:
Raider70, echel0n, BeeJee

12 Likes

Are you currently maintaing the LCD and FCD endpoints that all applications on the platform are using?

Are these endpoints separate from TFL endpoints?

Are these the only endpoints we have on Terra Luna Classic or are there more FCDs and LCDs running apart from Terra Rebels owned infrastructure?

4 Likes

The LCD and FCD endpoints we are maintaining are completely separate from TFL, they are used by Rebel Station and all other Terra Rebel related applications, and are publicly available for use as well.

There is other endpoints out there but those are maintained by some other 3rd party and support TFL’s Terra Station app that is maintained by TFL.

6 Likes

I believe that this is a good proposal at the beginning TFL did not want to support luna classic and only luna 2 having a separate station we are independent of them but I would also support because tfl may have problems with Do kwon and we will be left with nothing

4 Likes

It’s a no from me, after the last TR proposal I couldn’t justify any more money for TR.
Use some of the 150k or find a way to fund it yourselfs, sorry but you trashed TR’s reputation you can’t come back for more

10 Likes

You do realize that we used those funds to also build this infrastructure, hardware is not cheap, nor is our time involved to build it, the work was completed as we said, but again, that is your choice and you are entitled to it.

2 Likes
  1. Does this proposal cover minor enhancements and bug fixes to the Rebel Station software?
  2. Does this proposal cover 24x7 customer help desk support for Rebel Station end users?

I am a fan of own infrastructure. It seems out of date but the big guys never stop building (apple, google, tesla, aws…). It makes you independent. Perhaps you can highlight a bit more the benefit for the terra Network…

For Example:
Security Benefit
Scaling Benefit
Performance…
New Features…
…

Also you need a plan to scale. I don´t know exactly your cluster structure, but it seems a lot of team monthly wages ist just for monitoring. The expenses are the same for 1 or xx servers.

In my opinion it is important that we start building and not only burning :). People invest because they believe in the future development. And we must burn for the next 20 years to be back on track - is unsexy. More Building, more Volume, more Burning, better Price… more value for the investors.

7 Likes

This is a shining example of commitment and not taking the piss when it comes down to funding out of the community pool.

The only issue I see is whether we are going to have any money in the CP to use for funding purposes soon… a shame the tax was not increased to help fill up the CP faster.

6 Likes

#1. Yes, minor enhancements and bug fixes can be a part of this
#2. We are working on building support forums to help provide this type of support for Rebel Station

2 Likes

3 months ago,ppl said:”We don’t need infrastructure,Terra station is working good.”
And now,we see a unfunctional terra starion.
Time is always the sharpest knife in the world.

8 Likes

To be fair the issue about the last proposal was about the distribution of pay. Not only did TR hit all the deliverables promised. They actually provided more.

The issue of distribution has been addressed in this prop. And what they are offering looks very competitive if you look at the hourly rate and the amount of hours they will be putting in.

6 Likes

Waiting to see Rabbi’s alternate proposal on this …

6 Likes

36/48k annual for devops/sysadmin is very cheap; I assume the 8 hours per day means break/fix on-call for 8 hours per day?

The overall price seems reasonable, although its unclear what service level you are offering. Can you clarify what you service level you are able to offer, and would you be willing to offer financially backed SLA’s? i.e. lunc service credits for the next quarter should your service availability drop below x nines.

Heh, I like that. Such a good quote! :+1:

:joy: :joy: :joy:

Slash this in half as a gesture of good will to regain the community’s trust.
Then after 3 months have gone by, possibly increase it next time you submit this prop.
That’s my advice. TR’s reputation is in shambles, if you want to repair it it’ll come at a cost/sacrifice.

Personally, I’ll be ABSTAINING on this prop vote.

Shalom! :pray:

10 Likes

Im just not impressed with communication and direction. This will be a NO.

2 Likes

This is the Public Node provided and developed by Allnodes on TerraCVita nodes.
Ref: PublicNode

As you can see we get nearly 60m requests per day and actually support the network through open and accessible endpoints, including TFL station. NB: TFL Station can have upto 650k users per day (TFL) could confirm / clarify this.
These nodes are scattered around the globe to ensure security and stability. Scalability we have 100 nodes available in the rig. Also this was available before the initially infrastructure proposal by Terra Rebels back in 2022 which provided you with $150k funding and has been updated and refined since.

What makes your infrastructure more worthy of funding over lets say the Allnodes / TerraCVita provision? and how does the performance of the TerraRebels offering compare to this?

Can you provide information:

  1. on volume / traffic / usage of your nodes, and what is the maximum volume you have experienced?
  2. how many nodes do you have?
  3. how many different locations these are stored in (not the actual locations)?
  4. give examples of how the physical infrastructure is protected from natural disasters, energy black outs, theft etc?
  5. if it is kept in someone’s basement or lock up if that person falls ill how the nodes will continue to be managed and available?
  6. why Echel0n: is worth 33% more than Raider and BeeJee, i.e. what is that person providing that the others are not?
  7. What is the carbon footprint of this and how have you factored in Sustainability?
  8. What scalability has your infrastructure got?
  9. Physical management, how many of the three of you have physical access to these nodes in the event of a physical malfunction / issues, and what is the response time you commit to?
  10. How are you ensuring that the community has access to you to notify you about problems etc (there are countless stories of people being blocked by TR from discords etc)

Funding
Things to consider:-
You already have $150k to cover this, and you have (so I believe) funding from Neblio at $10k per month to pay for this.
Assuming that you paid $25k to fund the physical infrastructure (receipts would be welcome if this is to be supported by community funds) then you have $125k of your existing provided community funds to pay for this (you’ll need to chase your mods for the infrastructure money you gave them I guess) This will cover over a year of the funds you say that you would like. Also it was announced in the last couple of days that TerraRebels is no longer modding and supporting its Telegram group, work the community I assume has paid for by misappropriation of funds. So this should make it easier internally in your group to claw back money paid to these people if they are not delivering a service. So taking this into account i think the community has paid you for a years supply of this infrastructure already.

9 Likes

Why isn’t Terra Station operating correctly for weeks now?

Users incapable of withdrawing rewards.
Validators not showing up on governance.
Failed msg sends across the board.

I would say this is a good indication of the quality of TCVs work.

What makes your infrastructure more worthy of funding over lets say the Allnodes / TerraCVita provision? and how does the performance of the TerraRebels offering compare to this?
Terra Rebels owns the hardware and has physical access to it, we can scale it at will with no waiting of a 3rd party hosting provider, we are not constrained / restricted to what a 3rd party hosting provider has to offer, any changes to the infrastructure or apps that run on it are not restricted by a 3rd party such as TFL, we can add in features or fix existing code without needing to request permission.

Can you provide information:

  1. on volume / traffic / usage of your nodes, and what is the maximum volume you have experienced?
    as we have just started up, its not large amounts of traffic, but it is consistently growing as more people make the switch

  2. how many nodes do you have?
    we have started out with 6, but Kubernetes is more efficient then running bare-metal boxes as standalone servers as it harnesses the resources of all those nodes into a scalable cluster solution

  3. how many different locations these are stored in (not the actual locations)?
    only a single location, its all that is required

  4. give examples of how the physical infrastructure is protected from natural disasters, energy black outs, theft etc?
    backed by multiple ISP connections and generator power

  5. if it is kept in someone’s basement or lock up if that person falls ill how the nodes will continue to be managed and available?
    management of the cluster is completely done remotely and through GitOps, 3 people have access

  6. why Echel0n: is worth 33% more than Raider and BeeJee, i.e. what is that person providing that the others are not?
    I have an extensive background and knowledge of Kubernetes, I also have physical access to this cluster, so I will be on point for hardware failures as well.

  7. What is the carbon footprint of this and how have you factored in Sustainability?
    not exactly sure what you are asking here, are you asking for power consumption?

  8. What scalability has your infrastructure got?
    this Kubernetes cluster was built with scalability in mind from the start, scaling compute resources is as simple as adding in a new node, scaling storage is as simple as adding in more drives since our storage is clustered as well

  9. Physical management, how many of the three of you have physical access to these nodes in the event of a physical malfunction / issues, and what is the response time you commit to?
    I will be the only one with physical access to the systems, I live close by to the location.

  10. How are you ensuring that the community has access to you to notify you about problems etc (there are countless stories of people being blocked by TR from discords etc)
    we will be setting up forums like agora that will be used to provide support to end-users in regards to Rebel station and related apps

You already have $150k to cover this, and you have (so I believe) funding from Neblio at $10k per month to pay for this.
The $150k was for infrastructure, not running costs, and we no longer receive funding from Neblio.

7 Likes

I would not like to influence anyone’s decision as there are many things to consider in this proposal, but I will comment on the wallet first:

  1. In the amount of time and money allotted to you before this, you haven’t improved the wallet from the perspective of functionality apart from improving the UI and a few things here and there like the validators list. The amount of money invested for the work is enough to finish the work and innovate further. You have taken this work like normal work, not like community work. I have said this for the L1 Task Force as well that they have taken the work too formally while this should have been done with passion and interest. You should have been the ones who should have suggested to us what extra features can be added to the wallet and whether it can become an Interstation Wallet like the Luna Wallet. You should have worked with the L1 team to make that happen, but instead what we are looking at is another spend proposal for 3 more months of maintenance and bug fixes

  2. For the time the wallet was funded till now, there is no proper support system setup for the wallet users. If I am in good terms with you, I can join the Discord group and beg to TR for help. But the proactive response of the team to provide a non-biased portal for support has not been done and you have replied saying that it is something that you are thinking about. With this kind of support, we will not be able to make this wallet globally acceptable for payments because payment providers like Gpay and Paytm have an active support service because of which people use that payment provider. You need to understand that we wish to see Terra Luna Classic grow to more users, not create a system which restricts the number of users using an application. We will never grow if our service isn’t world class

  3. You have bought hardware servers instead of acquiring AWS or GCP servers like we had previously. I do not know how the community can even get access to hardware servers and who will retain them if by chance you do not continue maintaining the wallet. Also, I do not know why powerful hardware servers are required for the LCD and FCD also. If it was running from AWS, it must be possible to run it from AWS. I don’t know how much your AWS costs are, but at the cost of 1 server in my currency, I could have run a powerful server on AWS for 1-2 years depending upon which server we need. We could have even got that subsidized since Google and Amazon both offer free servers to non-profits. You did not do this till now even though I had mentioned this during the time you were asking for the funding initially. You have Nebilio’s support also. I see more costs, rather than less costs, which should have been the logical scenario in this case

As mentioned here, there is a public node which is being operated by TerraCVita and AllNodes. In fact, it is possible for other validators to operate nodes as well. You did not reveal this information when I asked if there are more nodes which are running currently. This makes me wonder if this proposal has an artificial sense of urgency embedded into it for the wrong reasons.

Coming to the budget that you have outlined in this proposal:

This is high for an AWS server taken from my country at least. I do not think it would cost so much even if we added S3, mails, notifications, etc to it.

This is extremely high for just maintenance. You are not doing active development in this proposal. Whatever has been outlined is clearly just maintenance work.

This can’t be a regular thing unless you have an actual roadmap to this development. I do not see any. Roadmap for a wallet would include how you would get investment from VCs also and add value to Terra Luna Classic. There was enough time and money to think about all these things but nothing of that sort was done.

I would gladly take the risk of setting up a node myself for the community or using one which is available already. I am fine with dealing with a broken wallet till the time we have an actual replacement.

5 Likes