Testing Hemi Part 2: Hemitrap

The prior article in this series discussed how software changes are tested across a live, isolated network. Whilst an effective strategy for testing changes to the Hemi daemons, other “network-wide” changes also require tests, namely, any changes to the smart contracts that power The Hemi Network.

Hemi consists of smart contracts deployed on both the Hemi L2 and the Ethereum L1 blockchain, as well as Sepolia for testnet purposes. These smart contracts enable the network to progress and facilitate communication between the two layers. There are times when these contracts require updates. Tests are run to verify that the network behaves as expected under known conditions (test cases).

Hemi’s smart contracts play a critical role in the network’s progression and functionality. A bug or incorrect deployment in these smart contracts risks causing reliability issues, a network outage, or potentially loss of funds. As such, it is extremely important that all changes are well-tested before deployment.

How To Test Smart Contract Updates In A Localnet Environment

Localnet is a local network that runs all of the Hemi L2 and required L1 daemons in an isolated environment. This environment allows developers to safely run test cases, which test network-wide functionality without the risk of breaking a production network, and contains additional features that can be leveraged for easier and clearer smart contract testing.

To do this, Localnet creates an isolated “fork” of the Hemi network on the machine using the Localnet orchestration configuration. This fork is created by stopping the live node at a specific block (trapping it at a certain point in time), before starting a separate network that begins at that block. This allows tests to be run on the fork with an expected state, which only affects the local isolated network. Since the state of the network is then “trapped” and a “fork” starts with that state, this feature is named Hemitrap.

These are the steps for Hemitrap setup:

First, it’s necessary to fork a live Ethereum network. Anvil’s forking feature will start an Ethereum L1 fork. This can be found here: https://github.com/hemilabs/hemi-node/blob/ad0c2cc77925a477c32f43cacaeef931d97633b8/testnet/docker-compose.yml#L316

Since there is no sequencer on the fork, a proxy must be established. This involves a few changes. Primarily, an op-node in sequencing mode, an op-batcher, and an op-proposer need to be set up and run. These have a “-forked” suffix added to them in the Docker Compose file. 

Here is op-node-forked, for example: https://github.com/hemilabs/hemi-node/blob/ad0c2cc77925a477c32f43cacaeef931d97633b8/testnet/docker-compose.yml#L329.  Inspecting this file reveals “op-proposer-forked” and “op-batcher-forked”.

Running a local sequencer will use a different private key than the one in production. Since this is the case, checks that will error if the sequencer is a different account are skipped. The “--hemitrap.enabled” flag performs these skips:
https://github.com/hemilabs/hemi-node/blob/ad0c2cc77925a477c32f43cacaeef931d97633b8/testnet/docker-compose.yml#L367. 

Additionally, searching “hemitrap” in the hemilabs/optimism repo shows exactly how this is used.

To speed up sequencing on the fork, modify these values in the rollup.json file:

{
    "max_sequencer_drift": 200,
    "seq_window_size": 20
}

With these details, a synced, non-sequencing hemi node is required to correctly fork the hemi network. Snap sync a node as documented (in the hemi-node directory):

./scripts/gen.sh testnet snap hemi-min
docker compose -f ./testnet/docker-compose.yml --profile hemi-min up

After this is synced, stop the daemons (via ctrl+c).

Then, run the hemitrap profile, which will fork our synced node into the isolated private network:

docker compose -f ./testnet/docker-compose.yml --profile hemitrap up

Hemi currently uses the L2OutputOracle smart contract, which requires a proposer address in its storage that matches the network’s proposer. Running a fork separate from production requires updating this address so the proposer can propose output roots. Use hardhat_setStorageAt to do:

curl http://localhost:9988/ -X POST -H 'Content-Type: application/json' --data '{"method":"hardhat_setStorageAt","params":["0x032d1e1dd960A4B027a9a35FF8B2b672E333Bc27", "0x7", "0x000000000000000000000000f39Fd6e51aad88F6F4ce6aB8827279cffFb92266"],"id":1,"jsonrpc":"2.0"}'

At this point, unsafe, safe, and finalized blocks should be progressing when checking the L2. If additional smart contracts must be deployed, now is the time to do so. Often, smart contract deployments are specific to the version in use.

For example, these are the steps used to update the SystemConfig smart contract from 1.12.0 to 2.3.0:

1. Deploy the new implementation via forge using “forge create”

   1. This smart contract version can be found here on this upstream tag https://github.com/ethereum-optimism/optimism/tree/op-contracts/v1.8.0-rc.4 

   2. Run the following

   DEPLOY_CONFIG_PATH=... forge script --sig deploySystemConfigImplementation  --rpc-url http://localhost:9988 --private-key=... ./scripts/upgrades/holocene/DeployUpgrade.s.sol:DeployUpgrade  --broadcast

2. Create a transaction that points the SystemConfigProxy to the new implementation 

3. Verify that the proxy’s version matches the newly deployed implementation (i.e. verify that the proxy points at the implementation) via eth_call

4. Run the test suite

When deploying smart contracts for testing, use one of Anvil’s prefunded private keys (available in the logs) to broadcast the transactions.

Once the smart contracts are updated and deployed, then reuse localnet test cases for the fork.  Note that since there is only a sequencing node, the sequencing node test cases are the only ones that matter.

TESTING_FORK=true go test -count=1 -timeout 120m -run TestL1L2Comms/testing_sequencing_client -v

When passing should return: 

--- PASS: TestL1L2Comms (0.00s)                                                                                                                                                                                                                                               
    --- PASS: TestL1L2Comms/testing_sequencing_client (2388.99s)                                                                                                                                                                                                              
PASS                                                                                                                                                                                                                                                                          
ok      github.com/hemilabs/heminetwork/e2e/monitor     2389.165s

Any test failures prompt a notification. Now there is a high degree of confidence that any updates to smart contracts won’t negatively affect the network.

Deploying These Changes To Production

Since testing changes against a forked live network, the same smart contract deployment steps used in testing are usable for live/production deployments. When doing so, generally it is only necessary to repeat the steps and use a production L1 RPC URL (opposed to the forked Anvil L1) and a funded production key for deployments.