Fast Validator Discussions!

[GabrielePicco]

🚀 Fast Validator MVP

Initial discussion for components and architecture needed for the Fast Validator MVP.

1: Delegation Module

The delegation module/program locks components/accounts on the mainnet and delegates the authority to the identity (a.k.a. private key) of the validator/sequencer.

2: Validator

The fast validator is SVM compatible and can execute any transaction that would work on the mainnet.

Websocket RPC Subscriptions

name	must have	done
accountSubscribe	✅	[ ]
signatureSubscribe	✅	[ ]
programSubscribe	✅	[ ]
logsSubscribe		[ ]

HTTP RPC Methods

name	must have	done
getLatestBlockhash	✅	✅
sendTransaction	✅	[ ]
simulateTransaction		[ ]
getAccountInfo	✅	✅
getProgramAccounts	✅	[ ]

Targets:

1. Fast: We don't need crazy performance at this stage, and more optimization could be done after the MVP. We are doing quick optimization for speed-ups and we can reduce the target block time.
   Considerations: Running the solana-test-validator as it is locally is already quite fast. It's not a priority to be 10x faster at this stage.
   Targeting 100ms end-to-end

2. Minimal Footprint: We decided to start from a bottom-up approach and gradually add the crates we need. This allows us to package and containerize a minimal SVM runtime.
   Considerations/Questions:

   • How many crates can we effectively remove, assuming consensus/voting/staking/etc... is not needed? Can we estimate percentage of the reduced footprint?
   • How much custom features and code do we have to develop/maintain? How much are we deviating from the standard implementation?

3. Ability to Clone on Demand Accounts: The validator needs to expose an RPC method (or have a built-in program) that allows it to be instructed from the outside to clone accounts on-demand.
   Consideration: This is perhaps one of the most important features of the validator, which allows it to execute transactions that rely on accounts living on the mainnet.

4. Easy Startup Configuration:

   • Expose a startup configuration for variables that are not part of the standard solana-test-validator flag.
     Consideration:
     For the MVP, the most likely useful configuration will be:
     • Reference cluster. From where the validator will clone accounts from e.g. mainnet
     • block/slot time?

3: Sequencer

The sequencer module writes the state-diff of delegated accounts on the mainnet and runs the undelegation process.

Considerations: It's key to understand the frequency, policy, and mechanism of sequencing (e.g., periodic vs. end of the session).

Notes:

• Mainnet could be any reference SVM cluster, including devnet or others (e.g., Eclipse SVM).

• Rpc router can be faked for the demo if not enough time. A quick/easy implementation is probably doable with Cloudflare functions

• (Not needed for the MVP) For fast reads, we could use a database on the edge, something like:

  • https://upstash.com/docs/redis/features/globaldatabase
  • https://developers.cloudflare.com/workers/databases/native-integrations/upstash/
    from where clients can easily get < 100ms updates. We still need to carefully optimize for fast writes with another approach.

4: Luzid

5: RPC Router

The RPC router determines where to route the transaction, whether on the mainnet or the auxiliary validator, depending on the account needed in the transaction. See section 2.2.2 of https://arxiv.org/pdf/2311.02650.pdf.

• Potentially interesting for an implementation: https://github.com/cloudflare/pingora?tab=readme-ov-file

6: Gasless API

7: Ticking

Add sysvar to the validator
BOLT System that provides Delta time

8: Systems/Components Registry

[thlorenz]

The sequencer is not exactly needed for the MVP as all transactions can be processed on our router for now and we can leave out the rollback for now.
Related to the sequencer we still need to figure out how/who pays for the rollup transaction on mainnet.

Related to that I already am building cost calculation into the current implementation, but we have no way of actually debiting those accounts on mainnet unless we have some accounts for each user that we control and that we debit from. Users would have to fund those before being able to play using our validator.
Also that is not needed for the MVP even though I am including cost calculations as I'm building the validator.

Mainnet could be any reference SVM cluster, including devnet or others (e.g., Eclipse SVM).

Could even be a locally running validator while we are testing this and possibly even for the demo.

Rpc router can be faked for the demo if not enough time

I think the quickest solution would be to figure this out locally first, but I agree that we don't need it at all and just send all txs to our validator for the demo

we can increase reduce the target block time

That will not affect latency at all as we declare txs to be finalized as soon as we committed account state changes to the bank.

It's not a priority to be 10x faster

Agreed, but we should get the latency to <100ms even for the demo so there is a visible difference to the main validator

How many crates can we effectively remove, assuming consensus is not needed?

I'm not even including them to begin with and am removing vote/stake related code as I port things over. However WRT crates that will be a longer endeavor as they aren't very clearly structured and sometimes just to get access to a specific type a whole crate is imported for now.
I'm planning to wrap these crates for the real implementation so we can then slowly replace the wrapped code with our own as we go.

most likely useful configuration will be block time

As I'm understanding things more it'd be easier to just stick with the 400ms slot time. It's less confusing to users. We can include more transactions in a single block though, i.e. we don't even produce blocks, such it is simply a matter of increasing the slot and pretending that a specific transaction was executed as part of it. We can do that since we only have one bank. I can elaborate on that if needed.

For fast reads, we could use a database on the edge

I'm not sure what reads we are referring to here. I'm using the accounts-db for account states on the validator itself.
Reaching <100ms will not be enough in the long run. For specific games we need to be faster, so I would try to avoid having data access be a limiting factor here.

[GabrielePicco]

• For the demo, let's target showcasing delegating accounts from mainnet to a local cluster, executing fast transactions, and writing back the state on mainnet.
  • Without writing to mainnet, it would be equivalent to using a local cluster (which would already be way faster than mainnet). I believe we would lose a lot for the demo if the changes does not surface to mainnet.
  • I think building an MVP for the sequencer is fairly easy to implement; the sequencer just overrides the account (assuming security is not a concern at this stage).
  • For the MVP, we don't need to design how/who pays for the rollup transaction on mainnet; I suggest to keep this for a later stage. An easy solution to keep track is to airdrop (automatically) "validator sol" to each participant of the validator and keep the standard fee mechanism of Solana. I would avoid doing anything custom at this stage.
• In my mind, block time and latency are connected, somewhat similar to the account-db update frequency, but I may be wrong. My assumption is that in our validator what is in the previous block will be finalized. Is this correct? (Perhaps slot is the correct terminology.) Question: Can we update the account-db with frequency < block time? This will help us understand the configuration we need for the validator.
  Also, order of transactions within a block is arbitrary, i.e. users don't have a guarantee that transactions will be executed sequencially in the block, e.g.: user 1 want to execute a -> b -> c with 50ms latency (hard to achieve if block time is 400ms). In my mind the easier solution is reducing the block time (I don't see how this would be confusing for users)
• On a local cluster, it should be easy to achieve <100ms latency. I believe the solana-test-validator is close to 400ms (the block time) using the processed commitment (I haven't measured, but it's almost instant running a game with the local validator + processed commitment).
• Let's try to get a better understanding of how much we can reduce the validator footprint with the bottom-up approach. It makes total sense that for now, the percentage of removed crates is not high, but we should have an idea of how much code we can remove.
• To clarify the idea for fast reads: mirror/propagate the validator accounts-db to a db on the edge, using some existing distributed network for low latency, e.g., Cloudflare or Upstash. Just an idea at this stage, I thought it was worth sharing.
  • For fast write we have a slightly more complicated problem that a centralised db/state on the edge, as the transactions are only an intent to change state (even without consensus we still need to lock write accounts and manage concurrency)

[GabrielePicco]

Follow-up question: how to be faster then solana-test-validator with processed commitment?

Speedups I see:

• Remove consensus/staking/voting overhead (with the bottom-up approach we are taking)
• Shorter target block/slot time

@thlorenz this is in principle the quick speedups I see we can get. Am I missing some important points? Is this accurate?

[thlorenz]

in our validator what is in the previous block will be finalized.

We don't produce blocks (at least for now) until we find a need to do so
We just increase the slot (mainly for show) since in our case it doesn't mean anything, we just process transactions as they come in and write account updates to the accounts-db.
We obviously need a way to query transactions, but how we do this is up to us and not tied to producing a block or even writing to a ledger.

It's ephemeral so all state/ledger, etc. that we maintain while it's running will disappear anyways once we roll back the changes.

Basically we're taking a single bank block less approach and just fill in things like slots to be compatible. So since there are no blocks then slot time (400ms) is only meaningful to explorers, in our case it makes no difference at all since as soon as a transaction is persisted to the accounts-db it is considered finalized, i.e. it is never finalized into a block.

Also, order of transactions within a block is arbitrary, i.e. users don't have a guarantee that transactions will be executed sequencially in the block

Forgetting about the block then yes we can batch such that each thread can take on a max of 128 (or more if we want to) transactions for each batch. However we could make that batch smaller in order to get more frequent account states persisted - again not at all related to a block, but just to however large we allow the batches to be.

• larger batches: less updates to accounts-db
• smaller batches: more frequent updates + smaller latency

WRT ordering I'm using the central scheduler which considers compute units (we don't support prioritization fees to avoid cheating/arbitrage for now). So as in mainnet there is no guarantee of the order also since its hard to determine which thread a transaction will run on (in parallel with non-conflicting other transactions).

However if we keep batches small enough then max latency should be kept small

Let's try to get a better understanding of how much we can reduce the validator footprint with the bottom-up approach

Agreed! That's why we do this in a quicker dirty prototype to then have a look and decide how to re-implement it better

To clarify the idea for fast reads: mirror/propagate the validator accounts-db to a db on the edge

Do we need that in the MVP? I was under the impression that for now the RPC runs on the same box as the validator and will provide the data by querying it?
Mirroring would only be necessary if the overhead of queries affects validator perf.
I'm keeping the mechanism to propagate tx results via channels while I port this, so we could provide this to an RPC for now.
Later we could have a service that propagates those results to a distributed network to offload load due to RPC on the validator process.

even without consensus we still need to lock write accounts and manage concurrency

That part I don't understand as we go for a single node approach for now. Why do we need to manage concurrency in a distributed manner? Unless you're referring to the accounts we lock on mainnet, and in that case whoever locks them first wins, i.e. the solana cluster already is the system managing locks for us.

[thlorenz]

NOTE: that if we want to align batch completion with slot increase then we can do that (may make more sense for explorers and such).

In that case we can make slots go as fast as we need, i.e. a slot time shouldn't be larger than the max latency we want to allow.
But again we aren't much doing with the slot aside from providing it for book keeping to RPCs, etc., thus we just will tune the batch size to get a decent latency.

[GabrielePicco]

Questions/notes for discussion:

• Do we currently update the slot after each transaction, or after a batch/block of transactions?
• Is there any difference between batch, block and slot? Good point that we can control the batch size.

Do we need that in the MVP?

We don't; let's forget about this for the MVP.

Even without consensus, we still need to lock write accounts and manage concurrency.

To clarify with one example the situation I'm thinking of (with both write and read issues):

• There is one account holding a value, which can be updated by multiple users. Clients build the transaction with tx: update(prev_value) to update it, where prev_value is the previous value and update is an instruction of a program.
• X users concurrently send transactions to update the value.

Even with a single node, there is going to be concurrency/locks as multiple users are going to hit the RPC with transactions, trying to change the same state. Also, users have to read the prev account state.

Assuming case 1), where we have some sort of batches/blocks:

• Each user will read the state in the previous block, creating the tx with the blockhash of the observed state.
• They will concurrently build the transaction, but only one of them will go through within a block, as all the others are going to be invalidated because of prev_value changing.
  In this scenario, batch/block "production time" limits the number of possible edits of the account holding the value. Also, each batch/block would be the concurrency window.

Assuming case 2) we don't have batches/blocks (streaming?):

• How do we handle concurrent incoming transactions which want to modify the account holding the value?
• How would the blockhash/commitment work in this scenario?

Questions: 1 or 2 ? Pros, Cons? Up to discussion

[GabrielePicco]

Terminology:

• Batch: a group of transactions that will be executed within a slot (e.g. 128 txs)

• Slot: A set of batches, executed in parallel in a multithreaded env, within a slot

  The period of time for which each leader ingests transactions and produces a block.
  Collectively, slots create a logical clock. Slots are ordered sequentially and non-overlapping, comprising roughly equal real-world time as per PoH.

• Block: A contiguous set of entries on the ledger covered by a vote. A leader produces at most one block per slot.

• Blockhash:
  A unique value (hash) that identifies a record (block). Solana computes a blockhash from the last entry id of the block.