Fuzz Network Test Framework by samliok · Pull Request #326 · ava-labs/Simplex

samliok · 2026-01-14T13:12:03Z

No description provided.

testutil/random_network/network.go

.gitignore

testutil/random_network/config.go

testutil/random_network/node.go

testutil/random_network/block.go

testutil/random_network/config.go

testutil/random_network/block_test.go

testutil/random_network/logging.go

yacovm · 2026-02-02T18:09:56Z

testutil/random_network/network.go

+}
+
+func (n *Network) StartInstances() {
+	panic("Call Run() Instead")


This method isn't used. Why is it needed?

because BasicNetwork is embedded in Network i don't want the caller accidentally calling StartInstances thinking its a valid method

testutil/random_network/network.go

testutil/random_network/mempool.go

testutil/random_network/node.go

yacovm · 2026-02-02T19:44:53Z

Can we rebase on top of main?

yacovm · 2026-02-25T19:31:48Z

This is still somewhat non-deterministic to some degree:

When I create a digest of the crashed and recovered node IDs in the schedule I get two possible outcomes: 9b9336fc010a3ccf10d41bb4344e36f2b2ec5c1c2f3940c76a4994005c1560c8 and 753a40743c43bf352998aad8d5a12df4df01fcd001d0baff1d513f3e85741a20

Patch is below:

diff --git a/testutil/random_network/network.go b/testutil/random_network/network.go
index e381fc0..188c7e9 100644
--- a/testutil/random_network/network.go
+++ b/testutil/random_network/network.go
@@ -4,6 +4,8 @@
 package random_network
 
 import (
+       "crypto/sha256"
+       "encoding/hex"
        "math/rand"
        "sync"
        "testing"
@@ -27,6 +29,9 @@ type Network struct {
 
        // tx stats
        allIssuedTxs int
+
+       recoveredNodes []simplex.NodeID
+       crashedNodes   []simplex.NodeID
 }
 
 func NewNetwork(config *FuzzConfig, t *testing.T) *Network {
@@ -224,6 +229,7 @@ func (n *Network) crashAndRecoverNodes() {
                        // randomly decide to recover based on NodeRecoverPercentage
                        if n.randomness.Float64() < n.config.NodeRecoverPercentage {
                                n.startNode(i)
+                               n.recoveredNodes = append(n.recoveredNodes, node.E.ID)
                                recoveredNodes = append(recoveredNodes, node.E.ID)
                                maxLeftToCrash++
                        }
@@ -294,6 +300,18 @@ func (n *Network) PrintStatus() {
                node.PrintMessageTypesSent()
        }
 
+       digest := sha256.New()
+
+       for _, node := range n.crashedNodes {
+               digest.Write([]byte(node.String()))
+       }
+
+       for _, node := range n.recoveredNodes {
+               digest.Write([]byte(node.String()))
+       }
+
+       n.logger.Info("Crashed and Recovered Nodes Digest", zap.String("digest", hex.EncodeToString(digest.Sum(nil))))
+
        // prints total issued txs and failed txs
        n.logger.Info("Transaction Stats", zap.Int("total issued txs", n.allIssuedTxs))
 }
@@ -302,6 +320,7 @@ func (n *Network) crashNode(idx int) {
        n.logger.Debug("Crashing node", zap.Stringer("nodeID", n.nodes[idx].E.ID))
        n.nodes[idx].isCrashed.Store(true)
        n.nodes[idx].Stop()
+       n.crashedNodes = append(n.crashedNodes, n.nodes[idx].E.ID)
 }
 
 func (n *Network) startNode(idx int) {

When we have 9b9336fc010a3ccf10d41bb4344e36f2b2ec5c1c2f3940c76a4994005c1560c8 we have 50 iterations of the loop or 51.

When we have 753a40743c43bf352998aad8d5a12df4df01fcd001d0baff1d513f3e85741a20 we have 49 iterations of the loop.

Can we make this fully deterministic? 🙏

It looks like we're very close to have it deterministic.

yacovm · 2026-02-24T16:19:28Z

testutil/random_network/mempool.go

+
+}
+
+func (m *Mempool) moveTxsToUnaccepted(block *Block) {


Why do we need to do this? we only delete from unacceptedTxs in AcceptBlock so if a transaction is in unacceptedTxs in a child block or in a grandchild block, it won't be removed from it anyway.

fuzz_network_test.go

yacovm · 2026-02-24T18:03:46Z

testutil/random_network/network.go

+				node.mempool.lock.Lock()
+				n.logger.Debug("Not all txs accepted yet by node", zap.Stringer("nodeID", node.E.ID), zap.Int("unaccepted txs in mempool", len(node.mempool.unacceptedTxs)))
+				node.mempool.lock.Unlock()
+				allAccepted = false


we can just break here

why can we break here? this is when allAccepted is false and we want to break when all are accepted

yacovm · 2026-02-25T21:39:55Z

testutil/random_network/network.go

+	return minHeightNodeID
+}
+
+func (n *Network) recoverToHeight(height uint64) {


The below way we are more deterministic (written by me, not thanks to CC):

func (n *Network) recoverToHeight(height uint64) { // Decide which crashed nodes to recover and in what order. // This keeps all randomness calls deterministic (fixed count per invocation), // independent of the non-deterministic AdvanceTime loop below. var nodesToRecover []int for i, node := range n.nodes { if node.isCrashed.Load() { nodesToRecover = append(nodesToRecover, i) } } n.randomness.Shuffle(len(nodesToRecover), func(i, j int) { nodesToRecover[i], nodesToRecover[j] = nodesToRecover[j], nodesToRecover[i] }) // Recover nodes one at a time in the chosen order, advancing time until // all nodes have caught up before recovering the next one. for _, idx := range nodesToRecover { n.logger.Debug("Recovering node", zap.Stringer("nodeID", n.nodes[idx].E.ID)) n.startNode(idx) for n.nodes[idx].storage.NumBlocks() < height { n.logger.Debug("Advancing network time", zap.Uint64("num crashed nodes", n.numCrashedNodes()), zap.Uint64("min height", n.getMinHeight()), zap.Uint64("max height", n.getMaxHeight()), zap.Stringer("Smallest node ID", n.getMinHeightNodeID()), zap.Uint64("target height", height), ) for _, node := range n.nodes { n.lock.Lock() node.AdvanceTime(n.config.AdvanceTimeTickAmount) n.lock.Unlock() } } } // Advance time until all remaining nodes reach the target height. for n.getMinHeight() < height { n.logger.Debug("Advancing network time", zap.Uint64("num crashed nodes", n.numCrashedNodes()), zap.Uint64("min height", n.getMinHeight()), zap.Uint64("max height", n.getMaxHeight()), zap.Stringer("Smallest node ID", n.getMinHeightNodeID()), zap.Uint64("target height", height), ) n.lock.Lock() n.BasicInMemoryNetwork.AdvanceTime(n.config.AdvanceTimeTickAmount) n.lock.Unlock() } }

for _, node := range n.nodes {
n.lock.Lock()
node.AdvanceTime(n.config.AdvanceTimeTickAmount)
n.lock.Unlock()
}

why not just advance the time using

n.BasicInMemoryNetwork.AdvanceTime(n.config.AdvanceTimeTickAmount)

also i think this change is good, can we add it as a follow up though?

yacovm · 2026-02-25T21:40:18Z

will continue the review tomorrow

yacovm · 2026-03-09T17:58:54Z

testutil/random_network/tx.go

+	return &TX{ID: idArr, shouldFailVerification: aTX.ShouldFailVerification}
+}
+
+func CreateNewTX() *TX {


Does the randomness here have any influence on the outcome? It sure has influence on the block hashes.

Should we perhaps inject randomness from our PRG?

eh, i think it's fine to not include our PRG here. I've found block hashes aren't that helpful to keep deterministic imo, at least when i'm debugging.

testutil/random_network/mempool.go

testutil/random_network/logging.go

yacovm · 2026-03-09T19:37:56Z

testutil/random_network/mempool.go

+}
+
+// go through any blocks that build off of this one and move their txs
+func (m *Mempool) purgeChildren(block *Block) {


aren't we missing a call to m.moveTxsToUnaccepted(block) ?

Because what about the transactions of the sibling block itself?

yep I think this commit fixes that issue
b2f99b1

testutil/random_network/block.go

yacovm · 2026-03-09T21:01:28Z

testutil/random_network/mempool.go

+	m.lock.Lock()
+	defer m.lock.Unlock()
+
+	// Ensure the block has not already been verified or accepted


I'm wondering if we should fail the test if we encounter this and the below... not just return an error. Thoughts?

added error logs which should fail the test

testutil/random_network/mempool.go

yacovm · 2026-03-09T21:04:24Z

Will make another pass tomorrow, I think that's enough comments for now.

testutil/random_network/utils.go

testutil/logger.go

testutil/random_network/network.go

yacovm · 2026-03-11T15:19:57Z

testutil/random_network/node.go

+	return n.BasicNode.HandleMessage(&msgCopy, from)
+}
+
+// copyMessage creates a copy of the message and its relevant fields to avoid mutating shared state in the in-memory network


so basically we just need to assign our own mempool, is that it?

testutil/random_network/node.go

yacovm · 2026-03-11T15:23:33Z

I think that's all comments I have for now.

samliok changed the base branch from main to refactor-testutil January 20, 2026 22:21

samliok marked this pull request as ready for review January 21, 2026 20:05

samliok linked an issue Jan 22, 2026 that may be closed by this pull request

Create a testing framework that generates random test cases #320

Closed

yacovm mentioned this pull request Feb 2, 2026

Refactor Test Utils #329

Merged