Network stability and security remain a core facet of Hemi’s architecture. To achieve these, Hemi engineers uphold rigorous standards and practices in terms of protocol development and deployment.
This multi-part series will examine how Hemi developers test the network to ensure its ongoing integrity. As the first installment in this series, this article will introduce and elaborate on how the Hemi Network is tested in isolation, or without external dependencies to run these test cases. Once built, these test cases can be run, and should be run, on an isolated machine connected to no other network or machine.
The goal is to run automated actions, or test cases, against the network, which will reveal bugs in a reproducible, deterministic manner. Should the test cases pass, then there is a high-degree of confidence that the code subjected to these test conditions is ready for production and will not knowingly break.
It should be further noted:
Tests are automated. In this case, manual testing is tedious, time-consuming, and error-prone.
Tests are reproducible and deterministic.
Tests cannot prove the non-existence of bugs. The goal of testing is to find the presence of bugs, should they exist; a test can confirm a specific bug, but not that no bugs exist.
Given these core ideas, Hemi engineers strive to create a development environment where they can iterate quickly on stable software on an isolated network, “Localnet”.
Introducing Localnet: Network-Wide Automated Tests in Isolation
The Hemi Network is split across many daemons. This presents the challenge of testing all of the daemons when interacting with each other to ensure that they interact correctly, making Hemi run. To do this, we create a local, isolated network called “Localnet” and test against that.
As previously stated, Localnet tests are designed to be automated, deterministic, reproducible, and effective at surfacing bugs.
With the more-exhaustive testing ideally handled within each daemon’s repository, engineers gather critical actions where bugs could reside and test against them to attempt to find bugs. If one of these fails, then it’s very likely that there is a larger issue at hand.
These network-wide tests can be found in this single file, summarized below: https://github.com/hemilabs/heminetwork/blob/13f58dbf44c58658a2a64d3ed53044920ed3b42f/e2e/monitor/main_test.go.
Network-Wide Test Cases
As a PoP miner, are L2 Keystones from the BFG daemon being received, being written to Bitcoin via bitcoind, and then, are the PoP payout rewards being received in turn as expected from Hemi?
As a user of Hemi:
Can $ETH be bridged from Ethereum to Hemi? (i.e. bridge L1 → L2)
Can ERC20 tokens be created and bridged by a user from Ethereum to Hemi?
Can that same ERC20 token be bridged from Hemi to Ethereum? (i.e. bridge L2 → L1)
Can $ETH be bridged from Hemi to Ethereum?
Is the HVM bitcoin tip near the real (regtest) bitcoin tip?
Can a $BTC balance be queried for a BTC address in an HVM smart contract?
Can all of this be done on both a sequencing and a non-sequencing node?
Are the output roots the same between the sequencing and non-sequencing nodes?
The above test cases are summarized greatly; additional detail becomes apparent upon closer examination. For them to work, three things must be accomplished:
First, they will have to go through smart contracts (including L1StandardBridge, L2StandardBridge, OptimismMintableERC20) on both Ethereum and Hemi, then the network will have to progress, and finally, the mechanics that allow bridging (for example, state root publishing) will have to work as well.
How Is Testing Automated?
Engineers capture this network’s definition in a docker-compose.yml file, found here: https://github.com/hemilabs/heminetwork/blob/13f58dbf44c58658a2a64d3ed53044920ed3b42f/e2e/docker-compose.yml
That file declares how to run Localnet. When used with Docker Compose, a tool that spins up the file on a local test network, it can be run on a machine for testing against.
There are a few more steps that can be found in this GitHub workflow file: https://github.com/hemilabs/heminetwork/blob/13f58dbf44c58658a2a64d3ed53044920ed3b42f/.github/workflows/localnet-test.yml
The easiest way to run this is the push-button GitHub action that the workflow file defines. For demonstration purposes, refer to this screenshot of the workflow:
Simply selecting a branch and clicking “Run Workflow” spins up Localnet and runs the tests against the selected branch, all in GitHub Actions CI. There is no manual intervention needed.
How Is Testing Reproducible And Deterministic?
Localnet tests need not depend on anything external. As such, this includes but is not limited to the following constraints:
Do not connect to anything that is outside of the VM.
No live networks. For example, do not connect to Ethereum, Sepolia, Bitcoin, etc.
No external APIs
Generate all data within the test.
Do not pull in data from an external source. For example, do not pull part of a production blockchain into a local data directory.
All data that is needed should be generated by the test case, and our code should adapt.
For most of the daemons, this is straightforward. All of the Hemi daemons, when traced down their dependencies, will eventually depend on two L1 Blockchains to function: Ethereum and Bitcoin. Thus, Ethereum and Bitcoin must be completely local.
This is achieved by configuring bitcoind and geth to run in regtest mode and dev mode, respectively.
Geth has a flag that allows for block production at a given rate, so that rate is configured to be high, currently 3 seconds. https://github.com/hemilabs/heminetwork/blob/13f58dbf44c58658a2a64d3ed53044920ed3b42f/e2e/docker-compose.yml#L157).
For bitcoind, a daemon is restarted that creates 1 bitcoin block, then exits successfully. That can be found here: https://github.com/hemilabs/heminetwork/blob/13f58dbf44c58658a2a64d3ed53044920ed3b42f/e2e/docker-compose.yml#L51
Note: a consensus client is not used in Localnet testing; this is on the roadmap https://github.com/hemilabs/heminetwork/issues/785
When those are all running, a genesis block is generated by enginners for our Localnet network, and the daemons are run. This will create an isolated, fast network for testing. The test cases are run against this.
Writing Effective Tests
The test cases were chosen for a reason; engineers want to find bugs in those common, critical workflows. They are not simply (and impossibly) proving that the network works. If a test case passes, it gives a high-degree of confidence that the tested functionality is working.
Everyone must understand this: there will always be bugs, and when they’re found downstream in a higher environment, there must be a strategy to address them.
This is easier said than done, but entirely possible with discipline: when a bug is found, a test case for that bug–in which the test case reproduces the bug itself–must be written in an automated test case. Then the bug must be fixed. Going forward, engineers will have a test case that looks for that bug.
It must be emphasized that test cases need to be effective. If they do not actively search for bugs or meaningfully exercise the code, they are not useful. In an era where vibe coders use AI tools to generate tens of thousands of test cases, ineffective tests do not meaningfully validate the code. Effectiveness matters more than volume or coverage.
Automated Tests Within A Repository
As many software projects do, each of Hemi's repositories has its own set of tests that validate the code in semi-isolation from other software projects. The term “semi-isolation” applies as there are cases where developers may want to test interacting with a single dependency that’s not worth mocking out. One example of this would be bitcoind in regtest mode.
To illustrate, here is a link to a test helper function in which bitcoind is run in regtest mode via testcontainers:
Some repositories are better tested than others. Regardless, these tests are necessary to find bugs within a codebase itself. This is where the most exhaustive testing occurs.
Engineers at Hemi automate the testing of the network in CI. They use bitcoind and geth in regtest and dev mode, respectively, to ensure that there are no external dependencies during the tests. Given this, all tests are isolated, reproducible, and deterministic. Testing developers chose, and wrote, effective test cases for critical paths to attempt to find bugs within them.